From 51432dcb19f0534663bc4da181949b018dc3915a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:14:14 -0800 Subject: [PATCH 01/13] docs(plan): ITL-906 async system-tag names plan and measurements Co-Authored-By: Claude Opus 5.5 --- ...26-11-12-itl-906-async-system-tag-names.md | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md diff --git a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md new file mode 100644 index 000000000..b52ca6fc7 --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md @@ -0,0 +1,95 @@ +# ITL-906: async system-tag names after a non-source input — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan +> task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** The async path of `MergeJoin`, `Batch` and `GroupBy` names system-tag columns (and so +derives record ids) exactly as the sync path does when an input is not a root source. + +**Architecture:** The barrier rebuild (`StaticOutputOperatorPod._materialize_to_stream`) returns a +producer-less `ArrowTableStream`, whose pipeline hash is the schema-only root hash. The fix gives +the rebuilt stream the pipeline hash the caller already passes through the existing +`input_pipeline_hashes` keyword of `async_execute` (the operator node passes +`[s.pipeline_hash() for s in self._input_streams]`). No public signature changes. + +**Tech stack:** Python, PyArrow, pytest via `uv run`. + +**Status:** measurement done, ask sent to the coordinator (no code before approval). +Linear: ITL-906 In Progress, project "Orcapod Python: v0.3 release". +Branch: `eywalker/itl-906-mergejoin-async-path-names-system-tag` (from origin/main 25650c42). + +--- + +## Measurements (main 25650c42) + +Scratch scripts (not committed): `measure.py` (operator level, every `CASES` entry of +`test_async_sync_equivalence.py`, over a root source and after a no-op `DropNulls("k")`), +`measure_job.py` (job level, Delta), `itl885.py` (ITL-885's own graph), `proto.py` +(monkeypatched prototype of the fix), `old_store.py` (pre-fix async store re-run post-fix). + +Operator level, after a non-source input (system-tag names / values / tag content hash): + +| Operator | async path | names | tag hash | +|---|---|---|---| +| Join (2, 3, rotated) | own, uses hashes | same | same | +| MergeJoin | `BinaryOperator` barrier | **differ** | same | +| SemiJoin | own (build/probe) | same | same | +| Batch (size 2, size 0) | own, `_materialize_to_stream` per batch | **differ** | same | +| GroupBy | `UnaryOperator` barrier | **differ** | same | +| PolarsFilter | `UnaryOperator` barrier | same (name-preserving) | same | +| MapTags, MapData, Select/Drop Tag/Data, Index, Pick, DropNulls | per row | same | same | + +Job level (Delta store; function node downstream of the operator): sync and async write the +SAME `pdb_v2` path, with different system-tag column names and different `__record_id`s. Over a +root source the two agree. With the prototype fix: identical columns and `__record_id`s. + +ITL-885 item 1 (its own graph `source -> double -> GroupBy -> first`, int control): sync then +async and async then sync FAIL on Delta (non-nullable `_tag_source_id::…` violation) and on the +in-memory store (schema mismatch). With the prototype fix all four orders pass on both stores with +no recomputation on the second run. Item 2 (GroupBy reads back empty after async) no longer +reproduces (ITL-896). + +A store written by a PRE-fix async run of an affected graph, re-run POST-fix (async or sync): +the downstream function node's write FAILS (Delta: non-nullable violation; in-memory: +`ArrowInvalid` schema mismatch). Same failure as ITL-885 today. + +Golden: `tests/test_hashing/golden/cases.py` runs MergeJoin (after `SelectDataColumns`), Batch +and GroupBy through `op(stream)` only (sync). No golden entry covers the async path; none moves. + +## Tasks (after approval) + +### Task 1: failing tests (public interface) + +- `tests/test_core/operators/test_async_sync_equivalence.py`: add a `NON_SOURCE_CASES` matrix: + every `CASES` entry with each input wrapped by a no-op operator (`DropNulls` on a non-nullable + tag) and by a function pod; run through `assert_equivalent` (names, values, hashes). Covers + every operator with `async_execute`. Expected to fail for merge_join, batch_2, batch_all, + group_by only. +- `tests/test_pipeline/...`: un-xfail + `test_merge_join_after_an_operator_reads_back_same_system_tags_after_async_run`; add Batch and + GroupBy after an operator; add a mixed-orchestrator test (sync then async, async then sync, on + in-memory and Delta) for ITL-885's graph, asserting no recomputation on the second run. +- Run, show failures, commit (WIP). + +### Task 2: fix + +- `static_output_pod.py`: `_materialize_to_stream(rows, pipeline_hash=None)` returns a private + `ArrowTableStream` subclass whose `pipeline_hash()` returns the given hash (falls back to the + schema-only hash when `None`). Default `StaticOutputOperatorPod.async_execute` passes + `input_pipeline_hashes[i]`. +- `base.py`: `UnaryOperator.async_execute`, `BinaryOperator.async_execute` pass their hashes. +- `batch.py`: `Batch.async_execute` takes the same keyword the base classes have and passes it. +- Length mismatch between `inputs` and `input_pipeline_hashes` raises `ValueError` (as `Join`). +- Run tests, ruff, basedpyright, commit. + +### Task 3: docs and records + +- `changelog.d/ITL-906.md` under `### Breaking changes and migration`. +- `design_issues/ITL-906-.md`, status resolved with **Fix:**. +- Update test docstrings/xfail reasons; remove the stale "Ignored by single-input operators" + docstring line. + +### Task 4: finish + +- Full `uv run pytest tests/`, ruff check/format, basedpyright exit code. +- Merge origin/main, push, PR "Fixes ITL-906" (+ ITL-885 per coordinator), CodeRabbit rounds. From 72b174ec43dc610536c15943de13729250fdfa65 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:18:21 -0800 Subject: [PATCH 02/13] test(operators): async system-tag names after a non-source input (ITL-906, ITL-885) Failing regression tests: every operator with an async path, with its inputs behind an operator or a function pod, in the sync/async equivalence matrix; the operator node read-back after an async run; one store run by both orchestrators in both orders. Co-Authored-By: Claude Opus 5.5 --- .../operators/test_async_sync_equivalence.py | 143 +++++++++++++++ .../test_pipeline/test_mixed_orchestrators.py | 165 ++++++++++++++++++ .../test_operator_node_async_readback.py | 49 ++++-- 3 files changed, 346 insertions(+), 11 deletions(-) create mode 100644 tests/test_pipeline/test_mixed_orchestrators.py diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 49afdbb0b..98ce37370 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -335,6 +335,149 @@ def test_async_rows_equal_sync_rows(case: Case) -> None: ) +# ── inputs that are not root sources (ITL-906) ────────────────────────────── + + +def after_operator(source: StreamProtocol) -> StreamProtocol: + """``source`` behind an operator that keeps every row: ``DropNulls`` on tag ``k``. + + Every source of the matrix has the non-nullable tag ``k``, so the rows and columns + are those of ``source``; only the pipeline hash differs, as it does after any + operator or function pod. + """ + stream = DropNulls("k")(source) + assert stream.pipeline_hash() != source.pipeline_hash() + return stream + + +def _after_operator_case(case: Case) -> Case: + """``case`` with every input behind ``after_operator``.""" + + def make_sources() -> tuple[StreamProtocol, ...]: + return tuple(after_operator(s) for s in case.make_sources()) + + return Case(case.label, case.make_op, make_sources) + + +def _first_after_operator_case(case: Case) -> Case: + """``case`` with only its first input behind ``after_operator``.""" + + def make_sources() -> tuple[StreamProtocol, ...]: + first, *rest = case.make_sources() + return (after_operator(first), *rest) + + return Case(case.label, case.make_op, make_sources) + + +AFTER_OPERATOR_CASES = [ + pytest.param(_after_operator_case(c), id=f"{c.label}-all") for c in CASES +] + [ + pytest.param(_first_after_operator_case(c), id=f"{c.label}-first") + for c in CASES + if len(c.make_sources()) > 1 +] + + +@pytest.mark.parametrize("case", AFTER_OPERATOR_CASES) +def test_async_rows_equal_sync_rows_after_an_operator(case: Case) -> None: + """Every operator's async path equals its sync path when an input is not a source. + + System-tag column names of ``MergeJoin`` (name-extending), ``Batch`` and ``GroupBy`` + (reducing) end in the input's pipeline hash; the async path must use the hash of the + stream the rows came from, not that of the rows rebuilt into a table. + """ + sources = case.make_sources() + assert_equivalent( + sync_rows(case.make_op(), *sources), async_rows(case.make_op(), *sources) + ) + + +def function_pod_stream(source_id: str, offset: int) -> StreamProtocol: + """A function pod's output on tag ``k: int16``: ``x = a + offset`` as ``int``.""" + source = _source( + {"k": pa.array([1, 2, 3], pa.int16()), "a": pa.array([1, 2, 3], pa.int8())}, + tags=("k",), + source_id=source_id, + ) + + def shift(a: int) -> int: + return a + offset + + pod = FunctionPod( + PythonDataFunction(shift, output_keys="x", function_name=f"shift_{source_id}") # pyright: ignore[reportArgumentType] # PythonDataFunction is not yet typed as matching DataFunctionProtocol (baselined in other tests); it works at runtime + ) + return pod(source) + + +FUNCTION_POD_CASES = [ + Case( + "merge_join", + MergeJoin, + lambda: (function_pod_stream("p", 0), function_pod_stream("q", 10)), + ), + Case( + "merge_join_with_source", + MergeJoin, + lambda: ( + function_pod_stream("p", 0), + _source( + { + "k": pa.array([1, 2, 3], pa.int16()), + "x": pa.array([7, 8, 9], pa.int64()), + }, + tags=("k",), + source_id="r", + ), + ), + ), + Case("join", Join, lambda: (function_pod_stream("p", 0), source_b())), + Case("semijoin", SemiJoin, lambda: (function_pod_stream("p", 0), source_b())), + Case("batch_2", lambda: Batch(batch_size=2), lambda: (function_pod_stream("p", 0),)), + Case("batch_all", Batch, lambda: (function_pod_stream("p", 0),)), + Case("group_by", lambda: GroupBy(["k"]), lambda: (function_pod_stream("p", 0),)), + Case( + "polars_filter", + lambda: PolarsFilter(constraints={"k": 1}), + lambda: (function_pod_stream("p", 0),), + ), +] + + +@pytest.mark.parametrize( + "case", FUNCTION_POD_CASES, ids=[c.label for c in FUNCTION_POD_CASES] +) +def test_async_rows_equal_sync_rows_after_a_function_pod(case: Case) -> None: + """The async path equals the sync path when an input is a function pod's output.""" + sources = case.make_sources() + assert_equivalent( + sync_rows(case.make_op(), *sources), async_rows(case.make_op(), *sources) + ) + + +@pytest.mark.parametrize( + "make_op", + [lambda: Batch(batch_size=2), Batch, lambda: GroupBy(["ta"])], + ids=["batch_2", "batch_all", "group_by"], +) +def test_reducing_operator_names_system_tags_after_its_input( + make_op: Callable[[], StaticOutputOperatorPod], +) -> None: + """A folded system-tag column name ends in the input's pipeline hash on both paths. + + System tag rule 3: ``Batch`` and ``GroupBy`` append ``::{pipeline_hash}`` of their + input. The input here is an operator's output, whose pipeline hash is not the + schema-only hash of its rows. + """ + op = make_op() + stream = after_operator(source_a()) + suffix = stream.pipeline_hash().to_hex(op.orcapod_config.hashing.system_tag_n_char) + for rows in (sync_rows(make_op(), stream), async_rows(make_op(), stream)): + for tag, _ in rows: + names = list(tag.system_tags()) + assert names + assert all(name.endswith(f"::{suffix}") for name in names), names + + def test_drop_nulls_without_tag_columns() -> None: """A stream without tag columns: the async path drops the same rows, in order. diff --git a/tests/test_pipeline/test_mixed_orchestrators.py b/tests/test_pipeline/test_mixed_orchestrators.py new file mode 100644 index 000000000..722e63dc8 --- /dev/null +++ b/tests/test_pipeline/test_mixed_orchestrators.py @@ -0,0 +1,165 @@ +"""One store, two orchestrators: a run on one extends a store written by the other (ITL-885). + +A job run by ``SyncPipelineOrchestrator`` and then by ``AsyncPipelineOrchestrator`` on the same +store, or the other way round, must find every stored result: the second run computes +nothing, and every node (source, operator and function nodes) reads back the rows of the +first run. Before ITL-906 the async path named the system-tag columns of ``GroupBy``, +``Batch`` and ``MergeJoin`` after a non-source input differently from sync, so the function +node downstream wrote a second set of system-tag columns into the same pipeline table and +the second run failed. +""" + +from __future__ import annotations + +from collections.abc import Callable +from pathlib import Path + +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.operators import Batch, GroupBy, MergeJoin, SelectDataColumns +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator +from orcapod.pipeline.job import PipelineJob +from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator +from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol + +ORCHESTRATORS = { + "sync": SyncPipelineOrchestrator, + "async": AsyncPipelineOrchestrator, +} + +CALLS: list[str] = [] + + +def _source(name: str, x: list[int]) -> ArrowTableSource: + """Tags ``id: int8`` and ``grp: large_string``; non-nullable ``x: int8``.""" + table = pa.table( + { + "id": pa.array([1, 2, 3, 4], type=pa.int8()), + "grp": pa.array(["g1", "g1", "g2", "g2"], type=pa.large_string()), + "x": pa.array(x, type=pa.int8()), + "s": pa.array(["p", "q", "r", "t"], type=pa.large_string()), + }, + schema=pa.schema( + [ + pa.field("id", pa.int8(), nullable=False), + pa.field("grp", pa.large_string(), nullable=False), + pa.field("x", pa.int8(), nullable=False), + pa.field("s", pa.large_string(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id", "grp"], source_id=name) + + +@function_pod(output_keys="y") +def double(x: int) -> int: + """The function node upstream of the operator.""" + CALLS.append("double") + return x * 2 + + +@function_pod(output_keys="first") +def first(y: list[int]) -> int: + """A function node after a reducing operator.""" + CALLS.append("first") + return y[0] + + +@function_pod(output_keys="total") +def total(x: list[int]) -> int: + """A function node after ``MergeJoin`` or ``Batch``.""" + CALLS.append("total") + return sum(x) + + +def _group_by_after_function_pod(a: StreamProtocol, b: StreamProtocol) -> None: + """ITL-885's graph: source, function node, ``GroupBy``, function node.""" + grouped = GroupBy(by=["grp"])(double.pod(a, label="double"), label="grouped") + first.pod(grouped, label="first") + + +def _merge_join_after_operator(a: StreamProtocol, b: StreamProtocol) -> None: + """``MergeJoin`` of an operator's output and a source, then a function node.""" + merged = MergeJoin()( + SelectDataColumns(["x"])(a, label="selected"), + SelectDataColumns(["x"])(b, label="selected_b"), + label="merged", + ) + total.pod(merged, label="total") + + +def _batch_after_operator(a: StreamProtocol, b: StreamProtocol) -> None: + """``Batch`` of an operator's output, then a function node.""" + batched = Batch(batch_size=2)(SelectDataColumns(["x"])(a, label="selected")) + total.pod(batched, label="total") + + +def _function_pod_over_source(a: StreamProtocol, b: StreamProtocol) -> None: + """A function node reading a source directly: both orchestrators always agreed.""" + double.pod(a, label="double") + + +GRAPHS: dict[str, Callable[[StreamProtocol, StreamProtocol], None]] = { + "function_pod_over_source": _function_pod_over_source, + "group_by_after_function_pod": _group_by_after_function_pod, + "merge_join_after_operator": _merge_join_after_operator, + "batch_after_operator": _batch_after_operator, +} + + +def _run( + graph: str, orchestrator: str, store: ArrowDatabaseProtocol +) -> tuple[PipelineJob, list[str]]: + """Build ``graph`` in a new job on ``store``, run it; return the job and the pod calls.""" + CALLS.clear() + job = PipelineJob(name="mixed", store=store) + with job: + GRAPHS[graph](_source("a", [1, 2, 3, 4]), _source("b", [5, 6, 7, 8])) + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + return job, list(CALLS) + + +def _read_back(job: PipelineJob) -> dict[str, list[str]]: + """Every node's rows: tag and data content hashes and system tags, sorted.""" + return { + label: sorted( + repr( + ( + tag.content_hash().to_string(), + data.content_hash().to_string(), + sorted(tag.system_tags().items()), + ) + ) + for tag, data in node.iter_data() + ) + for label, node in job.nodes.items() + } + + +def _store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + if kind == "in_memory": + return InMemoryArrowDatabase() + return DeltaTableDatabase(base_path=tmp_path / "store") + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("orders", [("sync", "async"), ("async", "sync")]) +@pytest.mark.parametrize("graph", list(GRAPHS)) +def test_second_orchestrator_finds_every_stored_result( + graph: str, orders: tuple[str, str], store_kind: str, tmp_path: Path +) -> None: + """The second run on the other orchestrator computes nothing and reads back the same rows.""" + store = _store(store_kind, tmp_path) + first_job, first_calls = _run(graph, orders[0], store) + second_job, second_calls = _run(graph, orders[1], store) + + assert first_calls + assert second_calls == [] + first_rows = _read_back(first_job) + assert all(first_rows.values()) + assert _read_back(second_job) == first_rows diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index cee87f120..8c303829b 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -447,21 +447,48 @@ def test_operator_node_with_no_rows_keeps_columns_and_types_after_async_run( ) -@pytest.mark.xfail( - strict=True, - reason=( - "ITL-906: the async MergeJoin names system-tag columns from the schema-only " - "hash of its materialized inputs, not from its upstreams' pipeline hashes" +AFTER_OPERATOR_CASES = [ + MERGE_JOIN_AFTER_OPERATOR, + Case( + "batch_after_operator", + lambda a, b, c: Batch(batch_size=2)(SelectDataColumns(["x"])(a), label="op"), + 2, + ), + Case( + "group_by_after_operator", + lambda a, b, c: GroupBy(by=["grp"])(SelectDataColumns(["x"])(a), label="op"), + 2, + ), + Case( + "group_by_after_function_pod", + lambda a, b, c: GroupBy(by=["grp"])(double_x.pod(a), label="op"), + 2, ), +] + + +@pytest.mark.parametrize( + "case", AFTER_OPERATOR_CASES, ids=[case.name for case in AFTER_OPERATOR_CASES] ) -def test_merge_join_after_an_operator_reads_back_same_system_tags_after_async_run( - tmp_path: Path, +def test_operator_after_an_operator_reads_back_same_system_tags_after_async_run( + case: Case, tmp_path: Path ) -> None: - """Async ``MergeJoin`` after an operator should name system tags as sync does.""" - sync_node, async_node = _sync_and_async_nodes(MERGE_JOIN_AFTER_OPERATOR, tmp_path) + """An async run names system tags as sync does when the input is not a source (ITL-906). - assert async_node.as_table(all_info=True).column_names == ( - sync_node.as_table(all_info=True).column_names + ``MergeJoin``, ``Batch`` and ``GroupBy`` put their inputs' pipeline hashes into + system-tag column names; the async path must use the upstream node's hash. + """ + sync_node, async_node = _sync_and_async_nodes(case, tmp_path) + + sync_table = sync_node.as_table(all_info=True) + async_table = async_node.as_table(all_info=True) + assert async_table.column_names == sync_table.column_names + assert async_table.num_rows == sync_table.num_rows == case.expected_rows + assert _row_hashes(async_node) == _row_hashes(sync_node) + assert sorted( + repr(sorted(tag.system_tags().items())) for tag, _ in async_node.iter_data() + ) == sorted( + repr(sorted(tag.system_tags().items())) for tag, _ in sync_node.iter_data() ) From 0f2a22c58b839f3e347b2763f964bf6c31891065 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:18:43 -0800 Subject: [PATCH 03/13] test(pipeline): mixed-orchestrator sources keep only the columns the pods read (ITL-885) Co-Authored-By: Claude Opus 5.5 --- tests/test_pipeline/test_mixed_orchestrators.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/test_pipeline/test_mixed_orchestrators.py b/tests/test_pipeline/test_mixed_orchestrators.py index 722e63dc8..0f8616c42 100644 --- a/tests/test_pipeline/test_mixed_orchestrators.py +++ b/tests/test_pipeline/test_mixed_orchestrators.py @@ -42,14 +42,12 @@ def _source(name: str, x: list[int]) -> ArrowTableSource: "id": pa.array([1, 2, 3, 4], type=pa.int8()), "grp": pa.array(["g1", "g1", "g2", "g2"], type=pa.large_string()), "x": pa.array(x, type=pa.int8()), - "s": pa.array(["p", "q", "r", "t"], type=pa.large_string()), }, schema=pa.schema( [ pa.field("id", pa.int8(), nullable=False), pa.field("grp", pa.large_string(), nullable=False), pa.field("x", pa.int8(), nullable=False), - pa.field("s", pa.large_string(), nullable=False), ] ), ) From bd450dd56ac64b762020ba19a9222a7063defdd5 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:22:57 -0800 Subject: [PATCH 04/13] fix(operators)!: async system-tag names use the input's pipeline hash (ITL-906, ITL-885) The barrier-mode async_execute rebuilt each input from its rows as a producer-less ArrowTableStream, whose pipeline hash is the schema-only root hash. MergeJoin, Batch and GroupBy name system-tag columns after their inputs' pipeline hashes, so after an operator or function pod the async path gave different names and record ids than sync, and a function node downstream wrote two column sets into one pipeline table (ITL-885). _materialize_to_stream now takes the input's pipeline hash, passed through the existing input_pipeline_hashes keyword, and the rebuilt stream reports it. Async names and record ids move to what sync already produces; sync output and golden values do not move. Fixes ITL-906 Fixes ITL-885 Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-906.md | 23 +++++ design_issues/ITL-906-O7.md | 27 ++++++ src/orcapod/core/operators/base.py | 38 +++++++- src/orcapod/core/operators/batch.py | 24 ++++- .../core/operators/static_output_pod.py | 92 +++++++++++++++++-- .../operators/test_async_sync_equivalence.py | 38 +++++++- .../test_operator_node_async_readback.py | 2 +- 7 files changed, 223 insertions(+), 21 deletions(-) create mode 100644 changelog.d/ITL-906.md create mode 100644 design_issues/ITL-906-O7.md diff --git a/changelog.d/ITL-906.md b/changelog.d/ITL-906.md new file mode 100644 index 000000000..fdfc7f8c9 --- /dev/null +++ b/changelog.d/ITL-906.md @@ -0,0 +1,23 @@ +### Breaking changes and migration + +#### The async orchestrator names system-tag columns as the sync orchestrator does (ITL-906, ITL-885) + +`MergeJoin`, `Batch` and `GroupBy` put their inputs' pipeline hashes into the names of the +system-tag columns they write. When one of these operators read an input that is not a +source (an operator's or a function pod's output), `AsyncPipelineOrchestrator` used a +different hash than `SyncPipelineOrchestrator`, so the same graph got different system-tag +column names, and different record ids, depending on which orchestrator ran it. The async +orchestrator now uses the input's pipeline hash, and its names and record ids are the ones +the sync orchestrator has always produced. Nothing a sync run produces changes, and graphs +where these operators read a source directly are unaffected. + +This also fixes ITL-885: a store written by one orchestrator can now be extended by the +other. Before, the function node downstream of such an operator wrote two sets of +system-tag columns into one pipeline table, and the second run failed while storing its +results. + +Migration: stores written by v0.2 are not affected, because v0.3 never reads a table +written by v0.2. Only a store written by an async run of an unreleased v0.3 build, for a +graph with `MergeJoin`, `Batch` or `GroupBy` after an operator or a function pod, holds the +old names: a later run on that store fails when it writes the pipeline table of the function +node downstream. Delete that table, or use a fresh store, and run again. diff --git a/design_issues/ITL-906-O7.md b/design_issues/ITL-906-O7.md new file mode 100644 index 000000000..95778d0c5 --- /dev/null +++ b/design_issues/ITL-906-O7.md @@ -0,0 +1,27 @@ +## `src/orcapod/core/operators/` — Async execution + +### O7 — Async system-tag names after a non-source input differed from sync +**Status:** resolved +**Severity:** high +**Issue:** ITL-906 + +The barrier-mode `async_execute` (`UnaryOperator`, `BinaryOperator`, the default +`StaticOutputOperatorPod.async_execute`, and `Batch`) rebuilt each input from its rows with +`_materialize_to_stream`, an `ArrowTableStream` without a producer whose pipeline hash is the +schema-only hash of a root. `MergeJoin` (system tag rule 2), `Batch` and `GroupBy` (rule 3) +name system-tag columns after their inputs' pipeline hashes, so when an input was an +operator's or a function pod's output the async names, `MergeJoin`'s canonical input order +and every record id derived from them differed from the sync path. The `input_pipeline_hashes` +keyword the operator node passes was ignored. A function node downstream then wrote two sets +of system-tag columns into one pipeline table, and a run on one orchestrator could not extend +a store written by the other (ITL-885). + +**Fix:** `_materialize_to_stream` takes the input's pipeline hash and returns a stream that +reports it (`_RowsOfStream`); every barrier path passes `input_pipeline_hashes`, and a hash +count that does not match the inputs raises `ValueError`. `Batch.async_execute` declares the +keyword instead of `**kwargs`. Async names and record ids now equal the sync ones; nothing a +sync run produces moves, and no golden value moves (the golden cases run the sync path). +`tests/test_core/operators/test_async_sync_equivalence.py` runs every operator with an async +path over inputs behind an operator and behind a function pod; +`tests/test_pipeline/test_mixed_orchestrators.py` runs one store with both orchestrators in +both orders. diff --git a/src/orcapod/core/operators/base.py b/src/orcapod/core/operators/base.py index 000cf346f..1fb0e4a93 100644 --- a/src/orcapod/core/operators/base.py +++ b/src/orcapod/core/operators/base.py @@ -117,10 +117,24 @@ async def async_execute( *, input_pipeline_hashes: Sequence[ContentHash] | None = None, ) -> None: - """Barrier-mode: collect single input, run unary_static_process, emit.""" + """Barrier-mode: collect single input, run unary_static_process, emit. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: The input stream's pipeline hash, as a + one-element sequence. The rebuilt input reports it, so + system-tag names equal the sync path's (ITL-906). + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. + """ try: + (pipeline_hash,) = self._pair_input_pipeline_hashes( + len(inputs), input_pipeline_hashes + ) rows = await inputs[0].collect() - stream = self._materialize_to_stream(rows) + stream = self._materialize_to_stream(rows, pipeline_hash) result = self.static_process(stream) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -201,13 +215,27 @@ async def async_execute( *, input_pipeline_hashes: Sequence[ContentHash] | None = None, ) -> None: - """Barrier-mode: collect both inputs concurrently, run binary_static_process, emit.""" + """Barrier-mode: collect both inputs concurrently, run binary_static_process, emit. + + Args: + inputs: The left and right readable input channels. + output: Writable channel for downstream consumption. + input_pipeline_hashes: The left and right input streams' pipeline + hashes. Each rebuilt input reports its own, so system-tag names + and canonical input order equal the sync path's (ITL-906). + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. + """ try: + left_hash, right_hash = self._pair_input_pipeline_hashes( + len(inputs), input_pipeline_hashes + ) left_rows, right_rows = await asyncio.gather( inputs[0].collect(), inputs[1].collect() ) - left_stream = self._materialize_to_stream(left_rows) - right_stream = self._materialize_to_stream(right_rows) + left_stream = self._materialize_to_stream(left_rows, left_hash) + right_stream = self._materialize_to_stream(right_rows, right_hash) result = self.static_process(left_stream, right_stream) for tag, data in result.iter_data(): await output.send((tag, data)) diff --git a/src/orcapod/core/operators/batch.py b/src/orcapod/core/operators/batch.py index d3419599d..255d97c04 100644 --- a/src/orcapod/core/operators/batch.py +++ b/src/orcapod/core/operators/batch.py @@ -8,7 +8,7 @@ from orcapod.core.streams import ArrowTableStream from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig +from orcapod.types import ColumnConfig, ContentHash from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -152,7 +152,8 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash] | None = None, ) -> None: """Streaming batch: emit full batches as they accumulate. @@ -160,13 +161,26 @@ async def async_execute( materialized and emitted immediately, allowing downstream consumers to start processing before all input is consumed. When ``batch_size == 0`` (batch everything), falls back to barrier mode. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: The input stream's pipeline hash, as a + one-element sequence. Every rebuilt batch reports it, so the + folded system-tag names end in it as on the sync path (ITL-906). + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: + (pipeline_hash,) = self._pair_input_pipeline_hashes( + len(inputs), input_pipeline_hashes + ) if self.batch_size == 0: # Must collect all rows — barrier fallback rows = await inputs[0].collect() if rows: - stream = self._materialize_to_stream(rows) + stream = self._materialize_to_stream(rows, pipeline_hash) result = self.unary_static_process(stream) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -176,7 +190,7 @@ async def async_execute( async for tag, data in inputs[0]: batch.append((tag, data)) if len(batch) >= self.batch_size: - stream = self._materialize_to_stream(batch) + stream = self._materialize_to_stream(batch, pipeline_hash) result = self.unary_static_process(stream) for out_tag, out_data in result.iter_data(): await output.send((out_tag, out_data)) @@ -184,7 +198,7 @@ async def async_execute( # Flush partial batch if batch and not self.drop_partial_batch: - stream = self._materialize_to_stream(batch) + stream = self._materialize_to_stream(batch, pipeline_hash) result = self.unary_static_process(stream) for out_tag, out_data in result.iter_data(): await output.send((out_tag, out_data)) diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 5ad11c5ad..a1b9cdb7d 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -11,6 +11,7 @@ from orcapod.config import OrcapodConfig from orcapod.contexts import DataContext from orcapod.core.base import TraceableBase +from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.protocols.core_protocols import ( @@ -28,6 +29,8 @@ if TYPE_CHECKING: import pyarrow as pa + + from orcapod.protocols.hashing_protocols import SemanticHasherProtocol else: pa = LazyModule("pyarrow") @@ -242,6 +245,7 @@ def _rows_to_table( @staticmethod def _materialize_to_stream( rows: Sequence[tuple[TagProtocol, DataProtocol]], + pipeline_hash: ContentHash | None = None, ) -> StreamProtocol: """Materialize a list of (Tag, Data) pairs into an ArrowTableStream. @@ -251,6 +255,13 @@ def _materialize_to_stream( Args: rows: Non-empty sequence of (Tag, Data) pairs. + pipeline_hash: The pipeline hash of the stream the rows came from. + The rebuilt stream reports it as its own, so an operator that + names system-tag columns after its input's pipeline hash + (``MergeJoin``, ``Batch``, ``GroupBy``) names them as the sync + path does (ITL-906). ``None`` leaves the rebuilt stream's own + hash, which is schema-only: equal to the input's only when the + input is a root source. Returns: A stream over the rows, in order. @@ -258,12 +269,37 @@ def _materialize_to_stream( Raises: ValueError: If ``rows`` is empty. """ - from orcapod.core.streams.arrow_table_stream import ArrowTableStream - table, tag_keys = StaticOutputOperatorPod._rows_to_table(rows) # No ``source_info`` argument: the table's own ``_source_*`` columns # pass through, one value per row. - return ArrowTableStream(table, tag_columns=tag_keys) + if pipeline_hash is None: + return ArrowTableStream(table, tag_columns=tag_keys) + return _RowsOfStream(table, tag_columns=tag_keys, pipeline_hash=pipeline_hash) + + @staticmethod + def _pair_input_pipeline_hashes( + n_inputs: int, input_pipeline_hashes: Sequence[ContentHash] | None + ) -> list[ContentHash | None]: + """The pipeline hash of each input, positionally; ``None`` each when none given. + + Args: + n_inputs: The number of input channels. + input_pipeline_hashes: The ``async_execute`` keyword of that name. + + Returns: + One entry per input. + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. + """ + if input_pipeline_hashes is None: + return [None] * n_inputs + if len(input_pipeline_hashes) != n_inputs: + n_hashes = len(input_pipeline_hashes) + raise ValueError( + f"input_pipeline_hashes length ({n_hashes}) must match inputs length ({n_inputs})" + ) + return list(input_pipeline_hashes) async def async_execute( self, @@ -281,13 +317,23 @@ async def async_execute( inputs: Readable channels, one per upstream node. output: Writable channel for downstream consumption. input_pipeline_hashes: Pipeline hash for each input stream, - positionally matching ``inputs``. Multi-input operators - (e.g. Join) use these to compute canonical system-tag - column names. Ignored by single-input operators. + positionally matching ``inputs``. Each rebuilt input stream + reports its hash, so system-tag column names equal the sync + path's (ITL-906). Without it, the rebuilt streams' schema-only + hashes are used, which equal the inputs' only for root sources. + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: + hashes = self._pair_input_pipeline_hashes( + len(inputs), input_pipeline_hashes + ) all_rows = await asyncio.gather(*(ch.collect() for ch in inputs)) - streams = [self._materialize_to_stream(rows) for rows in all_rows] + streams = [ + self._materialize_to_stream(rows, pipeline_hash) + for rows, pipeline_hash in zip(all_rows, hashes) + ] result = self.static_process(*streams) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -295,6 +341,38 @@ async def async_execute( await output.close() +class _RowsOfStream(ArrowTableStream): + """Rows collected from a channel, standing in for the stream they came from. + + The barrier-mode ``async_execute`` rebuilds each input from its rows. A + rebuilt ``ArrowTableStream`` has no producer, so its own pipeline hash is + the schema-only hash of a root; this stream reports the pipeline hash of + the stream the rows came from instead, as the operator node passes it + (``input_pipeline_hashes``). Its content hash is that of its table, as for + any ``ArrowTableStream`` without a producer. + """ + + def __init__( + self, + table: pa.Table, + *, + tag_columns: Collection[str], + pipeline_hash: ContentHash, + ) -> None: + super().__init__(table, tag_columns=tag_columns) + self._input_pipeline_hash: ContentHash = pipeline_hash + + def pipeline_hash( + self, hasher: SemanticHasherProtocol | None = None + ) -> ContentHash: + """The pipeline hash of the stream the rows came from. + + It was computed by the operator node with the hasher of its data + context, so it is returned whatever ``hasher`` is given. + """ + return self._input_pipeline_hash + + class DynamicPodStream(StreamBase): """Recomputable stream wrapping a ``StaticOutputOperatorPod`` invocation.""" diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 98ce37370..edec1d97c 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -432,7 +432,9 @@ def shift(a: int) -> int: ), Case("join", Join, lambda: (function_pod_stream("p", 0), source_b())), Case("semijoin", SemiJoin, lambda: (function_pod_stream("p", 0), source_b())), - Case("batch_2", lambda: Batch(batch_size=2), lambda: (function_pod_stream("p", 0),)), + Case( + "batch_2", lambda: Batch(batch_size=2), lambda: (function_pod_stream("p", 0),) + ), Case("batch_all", Batch, lambda: (function_pod_stream("p", 0),)), Case("group_by", lambda: GroupBy(["k"]), lambda: (function_pod_stream("p", 0),)), Case( @@ -478,6 +480,37 @@ def test_reducing_operator_names_system_tags_after_its_input( assert all(name.endswith(f"::{suffix}") for name in names), names +@pytest.mark.parametrize( + "case", + [ + c + for c in CASES + if c.label + in ("merge_join", "batch_2", "batch_all", "group_by", "polars_filter") + ], + ids=lambda c: c.label, +) +def test_barrier_async_rejects_wrong_number_of_pipeline_hashes(case: Case) -> None: + """A barrier ``async_execute`` needs one input pipeline hash per input channel.""" + sources = case.make_sources() + + async def run() -> None: + channels: list[Channel[Row]] = [Channel(buffer_size=64) for _ in sources] + for stream, channel in zip(sources, channels): + for row in stream.iter_data(): + await channel.writer.send(row) + await channel.writer.close() + output: Channel[Row] = Channel(buffer_size=64) + await case.make_op().async_execute( + [channel.reader for channel in channels], + output.writer, + input_pipeline_hashes=[s.pipeline_hash() for s in sources] * 2, + ) + + with pytest.raises(ValueError, match="must match inputs length"): + asyncio.run(run()) + + def test_drop_nulls_without_tag_columns() -> None: """A stream without tag columns: the async path drops the same rows, in order. @@ -666,8 +699,7 @@ def make_sources() -> tuple[StreamProtocol, ...]: # Join-style fix. Strict, so a fix flags them. _MATERIALIZE_DROPS_META = ( "ITL-846: barrier-mode async_execute rebuilds its input with " - "_materialize_to_stream, which drops Data meta columns; the rebuilt stream's " - "pipeline hash then differs, so system-tag names and canonical order differ too" + "_materialize_to_stream, which drops Data meta columns" ) _SYNC_DROPS_META = ( "ITL-846: the sync path drops Data meta columns that the async path keeps (a raw " diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index 8c303829b..a7e759a55 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -461,7 +461,7 @@ def test_operator_node_with_no_rows_keeps_columns_and_types_after_async_run( ), Case( "group_by_after_function_pod", - lambda a, b, c: GroupBy(by=["grp"])(double_x.pod(a), label="op"), + lambda a, b, c: GroupBy(by=["grp"])(double_x.pod(c), label="op"), 2, ), ] From 0be0a8bf8364e6983d776350d08fca1bca150119 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:24:16 -0800 Subject: [PATCH 05/13] docs(plan): ITL-906 decisions and ITL-865 measurements Co-Authored-By: Claude Opus 5.5 --- ...26-11-12-itl-906-async-system-tag-names.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md index b52ca6fc7..dc1cbfad7 100644 --- a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md +++ b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md @@ -93,3 +93,33 @@ and GroupBy through `op(stream)` only (sync). No golden entry covers the async p - Full `uv run pytest tests/`, ruff check/format, basedpyright exit code. - Merge origin/main, push, PR "Fixes ITL-906" (+ ITL-885 per coordinator), CodeRabbit rounds. + +## Coordinator decisions (2026-11-12) + +- Fix approved as proposed; `Batch.async_execute`'s `**kwargs` becomes the keyword-only + `input_pipeline_hashes`. +- PR says "Fixes ITL-906" and "Fixes ITL-885" (mixed-orchestrator test: source, operator and + function nodes, in-memory and Delta, both orders). MergeJoin source-info tie finding: out of + scope, minimal repro in worker_done. +- Changelog: only stores written by an async run of an unreleased v0.3 build are at risk. +- ITL-865 approved: same PR, its own commit(s), its own fragment `changelog.d/ITL-865.md`. + +## Progress + +- [x] ITL-906 tests (failing first), fix, changelog, design issue (commit bd450dd5). +- [ ] ITL-865: zero rows on every input / one input for every async operator (equivalence + test), async PipelineJob with a no-match filter upstream of a barrier operator and of a + function pod; fix: barrier path emits nothing when an input has no rows; docs + changelog + (`### Fixed`, `### Known limitations`: no typed empty result on async until ITL-858). + +## ITL-865 measurements (after the ITL-906 fix) + +Operator level, zero rows: MergeJoin (either or both inputs empty), GroupBy, PolarsFilter +raise `ValueError: Cannot materialize an empty list of rows`; Batch, Join, the per-row +operators give 0 rows as sync. SemiJoin with an empty RIGHT input gives the left rows when +the operator object was never validated (`_validated_common_keys` is set by +`validate_inputs`, i.e. by `op(...)` / node construction): operator-state hazard, ITL-858. + +Job level: a sync job whose upstream operator node produced zero rows feeds a binary operator +(Join/MergeJoin/SemiJoin) a widened table (ITL-912: int8 -> int64) and raises a type +disagreement; out of scope (ITL-912). From 1ff38edc8bb4498498161015dfcd0269ab5373a1 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:24:52 -0800 Subject: [PATCH 06/13] test(operators): zero input rows on every async operator path (ITL-865) Failing regression tests: every operator with an async path, with all inputs or one input delivering no rows, in the sync/async equivalence matrix; async and sync jobs with a no-match filter upstream of barrier operators and of a function pod. Co-Authored-By: Claude Opus 5.5 --- .../operators/test_async_sync_equivalence.py | 56 +++++++++++++++++ .../test_operator_node_async_readback.py | 60 +++++++++++++++++++ 2 files changed, 116 insertions(+) diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index edec1d97c..2e60c897a 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -511,6 +511,62 @@ async def run() -> None: asyncio.run(run()) +# ── inputs that deliver no rows (ITL-865) ─────────────────────────────────── + + +def no_rows(source: StreamProtocol) -> StreamProtocol: + """``source`` behind a filter that matches no row (every source has ``k`` in 1..3).""" + stream = PolarsFilter(constraints={"k": 0})(source) + assert list(stream.iter_data()) == [] + return stream + + +def _zero_row_cases() -> list[Any]: + """Every case with all inputs empty, and each multi-input case with one input empty.""" + params: list[Any] = [] + for case in CASES: + n_inputs = len(case.make_sources()) + empty_sets = [tuple(range(n_inputs))] + if n_inputs > 1: + empty_sets += [(i,) for i in range(n_inputs)] + for empty in empty_sets: + + def make_sources( + case: Case = case, empty: tuple[int, ...] = empty + ) -> tuple[StreamProtocol, ...]: + return tuple( + no_rows(s) if i in empty else s + for i, s in enumerate(case.make_sources()) + ) + + suffix = "all" if len(empty) == n_inputs else f"input_{empty[0]}" + params.append( + pytest.param( + Case(case.label, case.make_op, make_sources), + id=f"{case.label}-empty_{suffix}", + ) + ) + return params + + +@pytest.mark.parametrize("case", _zero_row_cases()) +def test_async_rows_equal_sync_rows_when_an_input_has_no_rows(case: Case) -> None: + """An input with no rows gives the sync path's rows on the async path, never an error. + + The operator is applied to its inputs first, ``op(...)``, as an operator node does when + the graph is built: ``SemiJoin`` with an empty right input reads the common keys that + application recorded, because a channel with no rows says nothing about its columns + (ITL-858). + """ + sources = case.make_sources() + expected = sync_rows(case.make_op(), *sources) + op = case.make_op() + _ = op(*sources) + actual = async_rows(op, *sources) + assert len(actual) == len(expected) + assert_equivalent(expected, actual) + + def test_drop_nulls_without_tag_columns() -> None: """A stream without tag columns: the async path drops the same rows, in order. diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index a7e759a55..c576d0fda 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -404,6 +404,66 @@ async def run() -> None: ) +def _no_match(stream: StreamProtocol) -> StreamProtocol: + """``stream`` behind a filter that matches no row.""" + return PolarsFilter(constraints={"grp": "none"})(stream) + + +AFTER_NO_ROWS_CASES = [ + Case( + "group_by_after_no_rows", + lambda a, b, c: GroupBy(by=["grp"])(_no_match(a), label="op"), + 0, + ), + Case( + "polars_filter_after_no_rows", + lambda a, b, c: PolarsFilter(constraints={"grp": "g1"})( + _no_match(a), label="op" + ), + 0, + ), + Case( + "batch_after_no_rows", + lambda a, b, c: Batch(batch_size=2)(_no_match(a), label="op"), + 0, + ), + Case( + "batch_all_after_no_rows", lambda a, b, c: Batch()(_no_match(a), label="op"), 0 + ), + # Both inputs empty: a zero-row operator node reads back types built from Python + # types (ITL-912), so one empty and one non-empty input disagree on ``id`` on sync. + Case( + "merge_join_after_no_rows", + lambda a, b, c: MergeJoin()(_no_match(a), _no_match(c), label="op"), + 0, + ), + Case( + "function_pod_after_no_rows", + lambda a, b, c: double_x.pod(_no_match(c), label="op"), + 0, + ), +] + + +@pytest.mark.parametrize( + "case", AFTER_NO_ROWS_CASES, ids=[case.name for case in AFTER_NO_ROWS_CASES] +) +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_node_after_an_input_with_no_rows_completes_the_run( + case: Case, orchestrator: str, tmp_path: Path +) -> None: + """A filter that matches no row upstream: the job completes with zero rows (ITL-865). + + On the async path a barrier operator whose input delivered no rows emits none, as the + sync path returns an empty stream; it does not raise. + """ + job = _run_job(case, orchestrator, "in_memory", tmp_path) + node: object = job.nodes["op"] + assert isinstance(node, StreamProtocol) + assert list(node.iter_data()) == [] + assert node.as_table().num_rows == 0 + + def _sync_and_async_nodes( case: Case, tmp_path: Path ) -> tuple[OperatorJobNode, OperatorJobNode]: From 64909dfc27be4ee810d59c1be13ef0043f84fa13 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:27:04 -0800 Subject: [PATCH 07/13] fix(operators): an async barrier operator with an empty input emits no rows (ITL-865) UnaryOperator, BinaryOperator and the default StaticOutputOperatorPod barrier paths raised 'Cannot materialize an empty list of rows' when an input delivered no rows, where sync returns an empty stream. They now emit nothing and close the output: an operator never synthesizes a row, and MergeJoin is an inner join. A typed empty result on async needs the stream schema (ITL-858). No identity moves. Fixes ITL-865 Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-865.md | 16 +++++++++++++ design_issues/ITL-865-O8.md | 24 +++++++++++++++++++ docs/concepts/operators.md | 7 +++++- src/orcapod/core/operators/base.py | 15 ++++++++++++ .../core/operators/static_output_pod.py | 9 ++++++- .../test_operator_node_async_readback.py | 2 +- 6 files changed, 70 insertions(+), 3 deletions(-) create mode 100644 changelog.d/ITL-865.md create mode 100644 design_issues/ITL-865-O8.md diff --git a/changelog.d/ITL-865.md b/changelog.d/ITL-865.md new file mode 100644 index 000000000..e78f08fa6 --- /dev/null +++ b/changelog.d/ITL-865.md @@ -0,0 +1,16 @@ +### Fixed + +- An async barrier operator (`PolarsFilter`, `GroupBy`, `MergeJoin`, and the default + `async_execute` of `UnaryOperator`, `BinaryOperator` and `StaticOutputOperatorPod`) whose + input delivers no rows emits no rows and finishes, as the sync path returns an empty + stream. It raised `ValueError: Cannot materialize an empty list of rows into a stream`, so + a pipeline with a filter that matches no row upstream of such an operator completed under + `SyncPipelineOrchestrator` and failed under `AsyncPipelineOrchestrator`. (ITL-865) + +### Known limitations + +- An async channel carries rows, not a schema. Where the sync path returns an empty stream + with typed columns, the async path emits nothing, and an operator node that received no + rows reads back columns typed from its Python output schema (`int8` becomes `int64`). + Giving async execution the stream's schema is ITL-858; the read-back types are ITL-912. + (ITL-865) diff --git a/design_issues/ITL-865-O8.md b/design_issues/ITL-865-O8.md new file mode 100644 index 000000000..648dd10c8 --- /dev/null +++ b/design_issues/ITL-865-O8.md @@ -0,0 +1,24 @@ +## `src/orcapod/core/operators/` — Async execution + +### O8 — Async barrier operators raised on an input with no rows +**Status:** resolved (no rows emitted); a typed empty result on async waits for ITL-858 +**Severity:** high +**Issue:** ITL-865 + +The barrier-mode `async_execute` of `UnaryOperator` and `BinaryOperator` (used by +`PolarsFilter`, `GroupBy` and `MergeJoin`) and the default `StaticOutputOperatorPod.async_execute` +rebuilt each input from its rows with `_materialize_to_stream`, which raises +`ValueError: Cannot materialize an empty list of rows into a stream` when a channel delivers +none. The sync path returns an empty stream, so a pipeline with a filter that matches no row +upstream of such an operator completed under the sync orchestrator and failed under the async +one. + +**Fix:** a barrier path whose input (any input, for the binary and default paths) delivers no +rows emits no rows and closes its output. That is the sync result of every barrier operator +that exists: an operator never synthesizes a row, and `MergeJoin` is an inner join. The sync +path's typed empty stream needs a schema the channels do not carry, so on async an operator +node that received no rows still reads back columns typed from its Python output schema +(ITL-858, ITL-912). `tests/test_core/operators/test_async_sync_equivalence.py` runs every +operator with an async path with all inputs and with each one input empty; +`tests/test_pipeline/test_operator_node_async_readback.py` runs sync and async jobs with a +filter matching no row upstream of barrier operators and of a function pod. diff --git a/docs/concepts/operators.md b/docs/concepts/operators.md index 945ba9ad5..cd89f7fe5 100644 --- a/docs/concepts/operators.md +++ b/docs/concepts/operators.md @@ -562,7 +562,12 @@ silent widening; convert explicitly. the join runs, not when the pipeline is built. - **Sync and async execution agree.** An operator run asynchronously (`async_execute`, as in an async pipeline) yields rows with the same Arrow types, column order, system tags and - content hashes as the same operator run synchronously. + content hashes as the same operator run synchronously, whether its inputs are sources or + the outputs of other operators and function pods. An input that delivers no rows gives no + rows on both paths. One limit remains: an async channel carries rows, not a schema, so + where the sync path returns an empty stream with typed columns the async path emits + nothing, and an operator node that received no rows reads back columns typed from its + Python output schema (ITL-858, ITL-912). ```python import pyarrow as pa diff --git a/src/orcapod/core/operators/base.py b/src/orcapod/core/operators/base.py index 1fb0e4a93..77390dd09 100644 --- a/src/orcapod/core/operators/base.py +++ b/src/orcapod/core/operators/base.py @@ -119,6 +119,8 @@ async def async_execute( ) -> None: """Barrier-mode: collect single input, run unary_static_process, emit. + An input with no rows emits no rows (ITL-865). + Args: inputs: The one readable input channel. output: Writable channel for downstream consumption. @@ -134,6 +136,11 @@ async def async_execute( len(inputs), input_pipeline_hashes ) rows = await inputs[0].collect() + if not rows: + # No rows in, no rows out: an operator never synthesizes a row. + # The typed empty stream the sync path returns needs a schema the + # channel does not carry (ITL-858), so nothing is emitted (ITL-865). + return stream = self._materialize_to_stream(rows, pipeline_hash) result = self.static_process(stream) for tag, data in result.iter_data(): @@ -217,6 +224,8 @@ async def async_execute( ) -> None: """Barrier-mode: collect both inputs concurrently, run binary_static_process, emit. + An input with no rows emits no rows, as an inner join does (ITL-865). + Args: inputs: The left and right readable input channels. output: Writable channel for downstream consumption. @@ -234,6 +243,12 @@ async def async_execute( left_rows, right_rows = await asyncio.gather( inputs[0].collect(), inputs[1].collect() ) + if not left_rows or not right_rows: + # The one barrier binary operator, MergeJoin, is an inner join: an + # input with no rows gives no rows, as on the sync path. The typed + # empty stream needs a schema the channels do not carry (ITL-858), + # so nothing is emitted (ITL-865). + return left_stream = self._materialize_to_stream(left_rows, left_hash) right_stream = self._materialize_to_stream(right_rows, right_hash) result = self.static_process(left_stream, right_stream) diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index a1b9cdb7d..2c4444dff 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -311,7 +311,9 @@ async def async_execute( """Default barrier-mode async execution. Collects all inputs, runs ``static_process``, emits results. - Subclasses override for streaming or incremental strategies. + Subclasses override for streaming or incremental strategies. When an + input has no rows, nothing is emitted (ITL-865): an operator whose + result is not empty then must override this method. Args: inputs: Readable channels, one per upstream node. @@ -330,6 +332,11 @@ async def async_execute( len(inputs), input_pipeline_hashes ) all_rows = await asyncio.gather(*(ch.collect() for ch in inputs)) + if not all(all_rows): + # Every barrier operator that exists gives no rows when an input has + # none. The typed empty stream the sync path returns needs a schema + # the channels do not carry (ITL-858), so nothing is emitted (ITL-865). + return streams = [ self._materialize_to_stream(rows, pipeline_hash) for rows, pipeline_hash in zip(all_rows, hashes) diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index c576d0fda..d29f5fd26 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -491,7 +491,7 @@ def test_operator_node_with_no_rows_reads_back_empty_after_async_run( @pytest.mark.xfail( strict=True, reason=( - "ITL-858/ITL-865: no row reaches the async node, so it reads back the table " + "ITL-858/ITL-912: no row reaches the async node, so it reads back the table " "built from the Python output schema: int8 becomes int64, and the system-tag " "and source-info columns are missing" ), From f13a8ce85700d6d4246cdffa07fe08b52a86e9d5 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:56:28 -0800 Subject: [PATCH 08/13] docs(plan): ITL-906 review round, required input_pipeline_hashes Co-Authored-By: Claude Opus 5.5 --- ...26-11-12-itl-906-async-system-tag-names.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md index dc1cbfad7..5f90420a7 100644 --- a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md +++ b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md @@ -123,3 +123,27 @@ the operator object was never validated (`_validated_common_keys` is set by Job level: a sync job whose upstream operator node produced zero rows feeds a binary operator (Join/MergeJoin/SemiJoin) a widened table (ITL-912: int8 -> int64) and raises a type disagreement; out of scope (ITL-912). + +## Review round: developer comment 4557365620 ("what's the point of this helper function??") + +Decision (developer "Go with A", coordinator approval 2026-11-12): REQUIRE the hashes and remove +`_pair_input_pipeline_hashes`. One rule for every operator: + +- Every operator's `async_execute` declares `*, input_pipeline_hashes: Sequence[ContentHash]`, + required, no default (replaces `| None = None` and `**kwargs`). Missing -> `TypeError`. +- The five that use it (UnaryOperator, BinaryOperator, StaticOutputOperatorPod default, Batch, + Join) check `len(input_pipeline_hashes) != len(inputs)` inline and raise `ValueError` naming + both numbers. Join's no-hash fallback (positional suffixes "0", "1") goes too. +- Per-row operators (MapTags, MapData, Select/Drop Tag/Data, DropNulls, Index, Pick) and SemiJoin + take it and say it is not used (they preserve names). +- `_materialize_to_stream(rows)` loses its hash parameter (output holders: operator_node ~1150, + ~1488, Index, Pick); barrier paths call a new `_materialize_input_stream(rows, pipeline_hash)` + (required hash) returning `_RowsOfStream`. +- Protocols unchanged (no protocol declares an operator's `async_execute`). +- Changelog ITL-906: add Join's old fallback. PR body: signature uniform and why. + +### Steps +- [ ] Failing tests (`tests/test_core/operators/test_async_sync_equivalence.py`): each barrier + operator + Join without hashes raises `TypeError`; wrong length raises `ValueError` naming both. +- [ ] Implement; update direct test callers; design issue Fix note; changelog. +- [ ] Full suite 3.12, 3.11, 3.13; ruff; basedpyright; merge origin/main; one push; reply. From d7689e0666b4bc597c4503c87385cf8d443db91a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:56:53 -0800 Subject: [PATCH 09/13] test(operators): async_execute requires input_pipeline_hashes (ITL-906) Co-Authored-By: Claude Opus 5.5 --- .../operators/test_async_sync_equivalence.py | 31 +++++++++++++++++-- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 2e60c897a..3ebb26355 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -480,19 +480,43 @@ def test_reducing_operator_names_system_tags_after_its_input( assert all(name.endswith(f"::{suffix}") for name in names), names +@CASE +def test_async_execute_requires_input_pipeline_hashes(case: Case) -> None: + """Every operator's ``async_execute`` takes the input pipeline hashes, always. + + Without them a barrier path would name system tags after a schema-only hash, + unlike the sync path (ITL-906), so there is no default to fall back on. + """ + channels: list[Channel[Row]] = [Channel() for _ in case.make_sources()] + output: Channel[Row] = Channel() + with pytest.raises(TypeError, match="input_pipeline_hashes"): + case.make_op().async_execute( # pyright: ignore[reportCallIssue, reportUnusedCoroutine] # the missing keyword is the point + [channel.reader for channel in channels], output.writer + ) + + @pytest.mark.parametrize( "case", [ c for c in CASES if c.label - in ("merge_join", "batch_2", "batch_all", "group_by", "polars_filter") + in ( + "join_2", + "join_3", + "merge_join", + "batch_2", + "batch_all", + "group_by", + "polars_filter", + ) ], ids=lambda c: c.label, ) def test_barrier_async_rejects_wrong_number_of_pipeline_hashes(case: Case) -> None: - """A barrier ``async_execute`` needs one input pipeline hash per input channel.""" + """An ``async_execute`` that uses the hashes needs one per input channel.""" sources = case.make_sources() + n_inputs = len(sources) async def run() -> None: channels: list[Channel[Row]] = [Channel(buffer_size=64) for _ in sources] @@ -507,7 +531,8 @@ async def run() -> None: input_pipeline_hashes=[s.pipeline_hash() for s in sources] * 2, ) - with pytest.raises(ValueError, match="must match inputs length"): + message = rf"length \({2 * n_inputs}\) must match inputs length \({n_inputs}\)" + with pytest.raises(ValueError, match=message): asyncio.run(run()) From fc2ce809438758a58788635f1d83a63b4fc18e05 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:22:16 -0800 Subject: [PATCH 10/13] refactor(operators)!: require input_pipeline_hashes on every operator async_execute (ITL-906) Remove _pair_input_pipeline_hashes: the keyword is required, with no fallback to a schema-only hash (barrier paths) or to positional suffixes (Join). Barrier inputs are rebuilt by _materialize_input_stream; the output-holder _materialize_to_stream takes no hash. Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-906.md | 7 + design_issues/ITL-906-O7.md | 11 +- src/orcapod/core/operators/base.py | 27 ++- src/orcapod/core/operators/batch.py | 17 +- .../core/operators/column_selection.py | 54 ++++- src/orcapod/core/operators/drop_nulls.py | 9 +- src/orcapod/core/operators/index.py | 6 +- src/orcapod/core/operators/join.py | 27 ++- src/orcapod/core/operators/mappers.py | 28 ++- src/orcapod/core/operators/pick.py | 6 +- src/orcapod/core/operators/semijoin.py | 12 +- .../core/operators/static_output_pod.py | 74 +++--- tests/test_channels/test_async_execute.py | 81 +++++-- .../test_native_async_operators.py | 221 +++++++++++++++--- tests/test_channels/test_pipeline_example.py | 33 ++- .../operators/test_chained_aggregation.py | 6 +- .../operators/test_index_pick_arrow_types.py | 12 +- .../operators/test_materialize_source_info.py | 7 +- tests/test_core/test_regression_fixes.py | 15 +- 19 files changed, 502 insertions(+), 151 deletions(-) diff --git a/changelog.d/ITL-906.md b/changelog.d/ITL-906.md index fdfc7f8c9..b37666de6 100644 --- a/changelog.d/ITL-906.md +++ b/changelog.d/ITL-906.md @@ -11,6 +11,13 @@ orchestrator now uses the input's pipeline hash, and its names and record ids ar the sync orchestrator has always produced. Nothing a sync run produces changes, and graphs where these operators read a source directly are unaffected. +Every operator's `async_execute` now requires the keyword argument `input_pipeline_hashes`, +one pipeline hash per input, as the operator node always passes it; a missing argument raises +`TypeError` and a count that does not match the inputs raises `ValueError`. Before, these +operators fell back without it to the schema-only hash, and `Join`'s async path to names built +from the input positions (`::0`, `::1`, ...), neither of which the sync path uses. Code that +calls an operator's `async_execute` directly must pass the inputs' pipeline hashes. + This also fixes ITL-885: a store written by one orchestrator can now be extended by the other. Before, the function node downstream of such an operator wrote two sets of system-tag columns into one pipeline table, and the second run failed while storing its diff --git a/design_issues/ITL-906-O7.md b/design_issues/ITL-906-O7.md index 95778d0c5..956170db4 100644 --- a/design_issues/ITL-906-O7.md +++ b/design_issues/ITL-906-O7.md @@ -16,10 +16,13 @@ keyword the operator node passes was ignored. A function node downstream then wr of system-tag columns into one pipeline table, and a run on one orchestrator could not extend a store written by the other (ITL-885). -**Fix:** `_materialize_to_stream` takes the input's pipeline hash and returns a stream that -reports it (`_RowsOfStream`); every barrier path passes `input_pipeline_hashes`, and a hash -count that does not match the inputs raises `ValueError`. `Batch.async_execute` declares the -keyword instead of `**kwargs`. Async names and record ids now equal the sync ones; nothing a +**Fix:** a barrier path rebuilds each input with `_materialize_input_stream(rows, +pipeline_hash)`, a stream that reports the input's pipeline hash (`_RowsOfStream`). +`input_pipeline_hashes` is a required keyword of every operator's `async_execute`, with no +default and no fallback: the barrier paths and `Join` (whose async path had named system tags +after the input positions `0`, `1`, ... when no hashes were given) raise `ValueError` when the +count does not match the inputs, and the per-row operators and `SemiJoin` take it and do not +use it. `_materialize_to_stream(rows)` only holds output rows. Async names and record ids now equal the sync ones; nothing a sync run produces moves, and no golden value moves (the golden cases run the sync path). `tests/test_core/operators/test_async_sync_equivalence.py` runs every operator with an async path over inputs behind an operator and behind a function pod; diff --git a/src/orcapod/core/operators/base.py b/src/orcapod/core/operators/base.py index 77390dd09..cc9c5e453 100644 --- a/src/orcapod/core/operators/base.py +++ b/src/orcapod/core/operators/base.py @@ -115,7 +115,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Barrier-mode: collect single input, run unary_static_process, emit. @@ -132,16 +132,18 @@ async def async_execute( ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: - (pipeline_hash,) = self._pair_input_pipeline_hashes( - len(inputs), input_pipeline_hashes - ) + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) rows = await inputs[0].collect() if not rows: # No rows in, no rows out: an operator never synthesizes a row. # The typed empty stream the sync path returns needs a schema the # channel does not carry (ITL-858), so nothing is emitted (ITL-865). return - stream = self._materialize_to_stream(rows, pipeline_hash) + stream = self._materialize_input_stream(rows, input_pipeline_hashes[0]) result = self.static_process(stream) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -220,7 +222,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Barrier-mode: collect both inputs concurrently, run binary_static_process, emit. @@ -237,9 +239,12 @@ async def async_execute( ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: - left_hash, right_hash = self._pair_input_pipeline_hashes( - len(inputs), input_pipeline_hashes - ) + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) + left_hash, right_hash = input_pipeline_hashes left_rows, right_rows = await asyncio.gather( inputs[0].collect(), inputs[1].collect() ) @@ -249,8 +254,8 @@ async def async_execute( # empty stream needs a schema the channels do not carry (ITL-858), # so nothing is emitted (ITL-865). return - left_stream = self._materialize_to_stream(left_rows, left_hash) - right_stream = self._materialize_to_stream(right_rows, right_hash) + left_stream = self._materialize_input_stream(left_rows, left_hash) + right_stream = self._materialize_input_stream(right_rows, right_hash) result = self.static_process(left_stream, right_stream) for tag, data in result.iter_data(): await output.send((tag, data)) diff --git a/src/orcapod/core/operators/batch.py b/src/orcapod/core/operators/batch.py index 255d97c04..2fc0fe32b 100644 --- a/src/orcapod/core/operators/batch.py +++ b/src/orcapod/core/operators/batch.py @@ -153,7 +153,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Streaming batch: emit full batches as they accumulate. @@ -173,14 +173,17 @@ async def async_execute( ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: - (pipeline_hash,) = self._pair_input_pipeline_hashes( - len(inputs), input_pipeline_hashes - ) + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) + pipeline_hash = input_pipeline_hashes[0] if self.batch_size == 0: # Must collect all rows — barrier fallback rows = await inputs[0].collect() if rows: - stream = self._materialize_to_stream(rows, pipeline_hash) + stream = self._materialize_input_stream(rows, pipeline_hash) result = self.unary_static_process(stream) for tag, data in result.iter_data(): await output.send((tag, data)) @@ -190,7 +193,7 @@ async def async_execute( async for tag, data in inputs[0]: batch.append((tag, data)) if len(batch) >= self.batch_size: - stream = self._materialize_to_stream(batch, pipeline_hash) + stream = self._materialize_input_stream(batch, pipeline_hash) result = self.unary_static_process(stream) for out_tag, out_data in result.iter_data(): await output.send((out_tag, out_data)) @@ -198,7 +201,7 @@ async def async_execute( # Flush partial batch if batch and not self.drop_partial_batch: - stream = self._materialize_to_stream(batch, pipeline_hash) + stream = self._materialize_input_stream(batch, pipeline_hash) result = self.unary_static_process(stream) for out_tag, out_data in result.iter_data(): await output.send((out_tag, out_data)) diff --git a/src/orcapod/core/operators/column_selection.py b/src/orcapod/core/operators/column_selection.py index e00f440f2..f3ed67c56 100644 --- a/src/orcapod/core/operators/column_selection.py +++ b/src/orcapod/core/operators/column_selection.py @@ -8,7 +8,7 @@ from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig, Schema +from orcapod.types import ColumnConfig, ContentHash, Schema from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -114,9 +114,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: select tag columns per row without materializing.""" + """Streaming: select tag columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: tags_to_drop: list[str] | None = None async for tag, data in inputs[0]: @@ -233,9 +242,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: select data columns per row without materializing.""" + """Streaming: select data columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: pkts_to_drop: list[str] | None = None async for tag, data in inputs[0]: @@ -347,9 +365,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: drop tag columns per row without materializing.""" + """Streaming: drop tag columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: effective_drops: list[str] | None = None async for tag, data in inputs[0]: @@ -468,9 +495,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: drop data columns per row without materializing.""" + """Streaming: drop data columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: effective_drops: list[str] | None = None async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/drop_nulls.py b/src/orcapod/core/operators/drop_nulls.py index b938653f1..437fbfcff 100644 --- a/src/orcapod/core/operators/drop_nulls.py +++ b/src/orcapod/core/operators/drop_nulls.py @@ -14,7 +14,7 @@ from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig, DataType, Schema +from orcapod.types import ColumnConfig, ContentHash, DataType, Schema from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -313,7 +313,8 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Streaming: test and narrow each row as it arrives, without materializing. @@ -325,7 +326,9 @@ async def async_execute( Args: inputs: Single-element sequence of readable channels. output: Channel to send surviving rows to. - **kwargs: Ignored; present for protocol compliance. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. """ try: async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/index.py b/src/orcapod/core/operators/index.py index a06ed77fa..23c09b676 100644 --- a/src/orcapod/core/operators/index.py +++ b/src/orcapod/core/operators/index.py @@ -379,7 +379,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Process packets one at a time as they arrive (true streaming). @@ -389,7 +389,9 @@ async def async_execute( Args: inputs: Single-element sequence of readable channels. output: Channel to send transformed packets to. - input_pipeline_hashes: Ignored; present for protocol compliance. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. """ try: async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/join.py b/src/orcapod/core/operators/join.py index 7da6933c8..b6ac78d5b 100644 --- a/src/orcapod/core/operators/join.py +++ b/src/orcapod/core/operators/join.py @@ -343,7 +343,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Async streaming join with pairwise iterative semantics. @@ -363,26 +363,25 @@ async def async_execute( inputs: Readable channels, one per upstream. output: Writable channel for downstream. input_pipeline_hashes: Pipeline hash for each input, - positionally matching ``inputs``. Required for - correct system-tag renaming with 2+ inputs. + positionally matching ``inputs``. With 2+ inputs, each input's + system-tag columns are renamed after its hash, as on the sync + path. + + Raises: + ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) if len(inputs) == 1: async for tag, data in inputs[0]: await output.send((tag, data)) return - n = len(inputs) - if input_pipeline_hashes is not None and len(input_pipeline_hashes) != n: - raise ValueError( - f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " - f"must match inputs length ({n})" - ) - suffixes = ( - self._compute_system_tag_suffixes(input_pipeline_hashes) - if input_pipeline_hashes is not None - else [str(i) for i in range(n)] - ) + suffixes = self._compute_system_tag_suffixes(input_pipeline_hashes) await self._streaming_join(inputs, output, suffixes) finally: await output.close() diff --git a/src/orcapod/core/operators/mappers.py b/src/orcapod/core/operators/mappers.py index 7e16aac27..48db5c234 100644 --- a/src/orcapod/core/operators/mappers.py +++ b/src/orcapod/core/operators/mappers.py @@ -7,7 +7,7 @@ from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.system_constants import constants -from orcapod.types import ColumnConfig, Schema +from orcapod.types import ColumnConfig, ContentHash, Schema from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -127,9 +127,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: rename data columns per row without materializing.""" + """Streaming: rename data columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: rename_map: dict[str, str] | None = None unmapped: list[str] | None = None @@ -267,9 +276,18 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: - """Streaming: rename tag columns per row without materializing.""" + """Streaming: rename tag columns per row without materializing. + + Args: + inputs: The one readable input channel. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. + """ try: rename_map: dict[str, str] | None = None unmapped: list[str] | None = None diff --git a/src/orcapod/core/operators/pick.py b/src/orcapod/core/operators/pick.py index ef65e4d4c..b69f071dc 100644 --- a/src/orcapod/core/operators/pick.py +++ b/src/orcapod/core/operators/pick.py @@ -393,7 +393,7 @@ async def async_execute( inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Process packets one at a time as they arrive (true streaming). @@ -403,7 +403,9 @@ async def async_execute( Args: inputs: Single-element sequence of readable channels. output: Channel to send transformed packets to. - input_pipeline_hashes: Ignored; present for protocol compliance. + input_pipeline_hashes: Not used: this operator keeps system-tag + names as they are, so no column is named after an input's + pipeline hash. """ try: async for tag, data in inputs[0]: diff --git a/src/orcapod/core/operators/semijoin.py b/src/orcapod/core/operators/semijoin.py index 1b9a42aca..532363b23 100644 --- a/src/orcapod/core/operators/semijoin.py +++ b/src/orcapod/core/operators/semijoin.py @@ -7,7 +7,7 @@ from orcapod.core.streams import ArrowTableStream from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol -from orcapod.types import ColumnConfig, Schema +from orcapod.types import ColumnConfig, ContentHash, Schema from orcapod.utils import arrow_utils, schema_utils from orcapod.utils.lazy_module import LazyModule @@ -195,7 +195,8 @@ async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], - **kwargs: Any, + *, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Build-probe: collect right input, then stream left through a hash lookup. @@ -211,6 +212,13 @@ async def async_execute( of each input, since channels carry no schema. When either input is empty there is nothing to check, so a key type mismatch the sync path reports gives an empty result here (as in ``Join``'s async path). + + Args: + inputs: The left and right readable input channels. + output: Writable channel for downstream consumption. + input_pipeline_hashes: Not used: the left rows pass through with + their system-tag names, so no column is named after an + input's pipeline hash. """ try: left_ch, right_ch = inputs[0], inputs[1] diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 2c4444dff..e9f182097 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -245,23 +245,18 @@ def _rows_to_table( @staticmethod def _materialize_to_stream( rows: Sequence[tuple[TagProtocol, DataProtocol]], - pipeline_hash: ContentHash | None = None, ) -> StreamProtocol: """Materialize a list of (Tag, Data) pairs into an ArrowTableStream. - Used by the barrier-mode ``async_execute`` to convert collected - channel items back into a stream suitable for ``static_process``. - Every row keeps its own source info (see ``_rows_to_table``). + Holds an operator's or a node's OUTPUT rows, whose stream identity is + given by whoever holds them (the node, the ``DynamicPodStream``), so the + stream has no producer and its own pipeline hash is never read. Rows + that stand for an operator's INPUT go through + ``_materialize_input_stream``. Every row keeps its own source info (see + ``_rows_to_table``). Args: rows: Non-empty sequence of (Tag, Data) pairs. - pipeline_hash: The pipeline hash of the stream the rows came from. - The rebuilt stream reports it as its own, so an operator that - names system-tag columns after its input's pipeline hash - (``MergeJoin``, ``Batch``, ``GroupBy``) names them as the sync - path does (ITL-906). ``None`` leaves the rebuilt stream's own - hash, which is schema-only: equal to the input's only when the - input is a root source. Returns: A stream over the rows, in order. @@ -272,41 +267,40 @@ def _materialize_to_stream( table, tag_keys = StaticOutputOperatorPod._rows_to_table(rows) # No ``source_info`` argument: the table's own ``_source_*`` columns # pass through, one value per row. - if pipeline_hash is None: - return ArrowTableStream(table, tag_columns=tag_keys) - return _RowsOfStream(table, tag_columns=tag_keys, pipeline_hash=pipeline_hash) + return ArrowTableStream(table, tag_columns=tag_keys) @staticmethod - def _pair_input_pipeline_hashes( - n_inputs: int, input_pipeline_hashes: Sequence[ContentHash] | None - ) -> list[ContentHash | None]: - """The pipeline hash of each input, positionally; ``None`` each when none given. + def _materialize_input_stream( + rows: Sequence[tuple[TagProtocol, DataProtocol]], + pipeline_hash: ContentHash, + ) -> StreamProtocol: + """Rebuild an input stream from the rows a barrier ``async_execute`` collected. + + The rebuilt stream reports ``pipeline_hash`` as its own, so an operator + that names system-tag columns after its input's pipeline hash + (``MergeJoin``, ``Batch``, ``GroupBy``) names them as the sync path + does (ITL-906). Every row keeps its own source info (see + ``_rows_to_table``). Args: - n_inputs: The number of input channels. - input_pipeline_hashes: The ``async_execute`` keyword of that name. + rows: Non-empty sequence of (Tag, Data) pairs. + pipeline_hash: The pipeline hash of the stream the rows came from. Returns: - One entry per input. + A stream over the rows, in order. Raises: - ValueError: If ``input_pipeline_hashes`` does not have one hash per input. + ValueError: If ``rows`` is empty. """ - if input_pipeline_hashes is None: - return [None] * n_inputs - if len(input_pipeline_hashes) != n_inputs: - n_hashes = len(input_pipeline_hashes) - raise ValueError( - f"input_pipeline_hashes length ({n_hashes}) must match inputs length ({n_inputs})" - ) - return list(input_pipeline_hashes) + table, tag_keys = StaticOutputOperatorPod._rows_to_table(rows) + return _RowsOfStream(table, tag_columns=tag_keys, pipeline_hash=pipeline_hash) async def async_execute( self, inputs: Sequence[ReadableChannel[tuple[TagProtocol, DataProtocol]]], output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, - input_pipeline_hashes: Sequence[ContentHash] | None = None, + input_pipeline_hashes: Sequence[ContentHash], ) -> None: """Default barrier-mode async execution. @@ -315,22 +309,26 @@ async def async_execute( input has no rows, nothing is emitted (ITL-865): an operator whose result is not empty then must override this method. + Every operator's ``async_execute`` takes ``input_pipeline_hashes``, with + no default: the operator node passes it for every operator alike. + Args: inputs: Readable channels, one per upstream node. output: Writable channel for downstream consumption. input_pipeline_hashes: Pipeline hash for each input stream, positionally matching ``inputs``. Each rebuilt input stream reports its hash, so system-tag column names equal the sync - path's (ITL-906). Without it, the rebuilt streams' schema-only - hashes are used, which equal the inputs' only for root sources. + path's (ITL-906). Raises: ValueError: If ``input_pipeline_hashes`` does not have one hash per input. """ try: - hashes = self._pair_input_pipeline_hashes( - len(inputs), input_pipeline_hashes - ) + if len(input_pipeline_hashes) != len(inputs): + raise ValueError( + f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) " + f"must match inputs length ({len(inputs)})" + ) all_rows = await asyncio.gather(*(ch.collect() for ch in inputs)) if not all(all_rows): # Every barrier operator that exists gives no rows when an input has @@ -338,8 +336,8 @@ async def async_execute( # the channels do not carry (ITL-858), so nothing is emitted (ITL-865). return streams = [ - self._materialize_to_stream(rows, pipeline_hash) - for rows, pipeline_hash in zip(all_rows, hashes) + self._materialize_input_stream(rows, pipeline_hash) + for rows, pipeline_hash in zip(all_rows, input_pipeline_hashes) ] result = self.static_process(*streams) for tag, data in result.iter_data(): diff --git a/tests/test_channels/test_async_execute.py b/tests/test_channels/test_async_execute.py index 82760f258..b00408c06 100644 --- a/tests/test_channels/test_async_execute.py +++ b/tests/test_channels/test_async_execute.py @@ -228,7 +228,11 @@ async def test_select_tag_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -244,7 +248,11 @@ async def test_select_data_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -262,7 +270,11 @@ async def test_drop_data_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -288,7 +300,11 @@ async def test_drop_tag_columns(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 2 @@ -306,7 +322,11 @@ async def test_map_tags(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -323,7 +343,11 @@ async def test_map_data(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -341,7 +365,11 @@ async def test_polars_filter(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 1 @@ -358,7 +386,11 @@ async def test_batch_operator(self): output_ch = Channel(buffer_size=16) await feed_stream_to_channel(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 # 6 rows / batch_size=2 @@ -390,7 +422,11 @@ async def test_semi_join(self): await feed_stream_to_channel(left, left_ch) await feed_stream_to_channel(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), right.pipeline_hash()], + ) results = await output_ch.reader.collect() ids = sorted(tag.as_dict()["id"] for tag, _ in results) @@ -422,7 +458,11 @@ async def test_merge_join(self): await feed_stream_to_channel(left, left_ch) await feed_stream_to_channel(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), right.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 2 @@ -460,7 +500,11 @@ async def test_two_way_join(self): await feed_stream_to_channel(left, left_ch) await feed_stream_to_channel(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), right.pipeline_hash()], + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -674,7 +718,13 @@ async def source(): async with asyncio.TaskGroup() as tg: tg.create_task(source()) - tg.create_task(filter_op.async_execute([ch1.reader], ch2.writer)) + tg.create_task( + filter_op.async_execute( + [ch1.reader], + ch2.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) + ) tg.create_task(func_pod.async_execute([ch2.reader], ch3.writer)) results = await ch3.reader.collect() @@ -727,7 +777,12 @@ async def push(stream, ch): tg.create_task(push(right_stream, ch_right)) tg.create_task( join_op.async_execute( - [ch_left.reader, ch_right.reader], ch_joined.writer + [ch_left.reader, ch_right.reader], + ch_joined.writer, + input_pipeline_hashes=[ + left_stream.pipeline_hash(), + right_stream.pipeline_hash(), + ], ) ) tg.create_task(func_pod.async_execute([ch_joined.reader], ch_out.writer)) diff --git a/tests/test_channels/test_native_async_operators.py b/tests/test_channels/test_native_async_operators.py index b610f2b2d..68d681681 100644 --- a/tests/test_channels/test_native_async_operators.py +++ b/tests/test_channels/test_native_async_operators.py @@ -40,7 +40,9 @@ SemiJoin, ) from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.protocols.core_protocols import StreamProtocol from orcapod.system_constants import constants +from orcapod.types import ContentHash # --------------------------------------------------------------------------- @@ -155,6 +157,21 @@ def make_disjoint_stream() -> ArrowTableStream: return ArrowTableStream(table, tag_columns=["animal"]) +def hashes(*streams: StreamProtocol) -> list[ContentHash]: + """The pipeline hash of each input stream, as the operator node passes them.""" + return [stream.pipeline_hash() for stream in streams] + + +def root_hash(tag: str, data: str) -> ContentHash: + """The pipeline hash of a root stream with one int64 tag and one int64 data column. + + For an input channel fed by hand, or one that delivers no rows: the operator + node passes every input's pipeline hash, so the tests do too. + """ + table = pa.table({tag: pa.array([0], pa.int64()), data: pa.array([0], pa.int64())}) + return ArrowTableStream(table, tag_columns=[tag]).pipeline_hash() + + async def feed(stream: ArrowTableStream, ch: Channel) -> None: """Push all (tag, data) from a stream into a channel, then close.""" for tag, data in stream.iter_data(): @@ -167,7 +184,9 @@ async def run_unary(op, stream: ArrowTableStream) -> list[tuple]: input_ch = Channel(buffer_size=1024) output_ch = Channel(buffer_size=1024) await feed(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], output_ch.writer, input_pipeline_hashes=hashes(stream) + ) return await output_ch.reader.collect() @@ -180,7 +199,11 @@ async def run_binary( output_ch = Channel(buffer_size=1024) await feed(left, left_ch) await feed(right, right_ch) - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=hashes(left, right), + ) return await output_ch.reader.collect() @@ -237,7 +260,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = SelectTagColumns(columns=["region"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -326,7 +353,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = SelectDataColumns(columns=["weight"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -411,7 +442,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = DropTagColumns(columns=["region"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -473,7 +508,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = DropDataColumns(columns=["legs"]) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -569,7 +608,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = MapTags(name_map={"region": "area"}) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -682,7 +725,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = MapData(name_map={"weight": "mass"}) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -773,7 +820,11 @@ async def test_empty_input(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = Batch(batch_size=2) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -919,7 +970,11 @@ async def test_empty_left_returns_empty(self): await feed(right, right_ch) op = SemiJoin() - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value"), right.pipeline_hash()], + ) results = await output_ch.reader.collect() assert results == [] @@ -937,7 +992,11 @@ async def test_empty_right_returns_empty_or_all(self): await right_ch.writer.close() op = SemiJoin() - await op.async_execute([left_ch.reader, right_ch.reader], output_ch.writer) + await op.async_execute( + [left_ch.reader, right_ch.reader], + output_ch.writer, + input_pipeline_hashes=[left.pipeline_hash(), root_hash("id", "value")], + ) results = await output_ch.reader.collect() # With empty right and no schema information available, # the implementation falls back to passing left through @@ -998,7 +1057,9 @@ async def test_single_input_passthrough(self): input_ch = Channel(buffer_size=64) output_ch = Channel(buffer_size=64) await feed(stream, input_ch) - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], output_ch.writer, input_pipeline_hashes=hashes(stream) + ) results = await output_ch.reader.collect() assert len(results) == 3 @@ -1077,7 +1138,11 @@ async def test_three_way_join(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) results = await out.reader.collect() assert len(results) == 2 @@ -1114,7 +1179,11 @@ async def test_empty_input_single(self): output_ch = Channel(buffer_size=4) await input_ch.writer.close() op = Join() - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[root_hash("id", "value")], + ) results = await output_ch.reader.collect() assert results == [] @@ -1171,7 +1240,11 @@ async def test_matches_sync_three_way(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) async_results = await out.reader.collect() assert len(async_results) == len(sync_results) @@ -1197,7 +1270,15 @@ async def test_three_way_streams_before_all_closed(self): # Start the join in the background join_task = asyncio.create_task( - op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=[ + root_hash("id", "a"), + root_hash("id", "b"), + root_hash("id", "c"), + ], + ) ) # Send one matching row from each side @@ -1235,7 +1316,11 @@ async def test_four_way_join(self): out = Channel(buffer_size=64) for s, ch in zip(streams, channels): await feed(s, ch) - await op.async_execute([ch.reader for ch in channels], out.writer) + await op.async_execute( + [ch.reader for ch in channels], + out.writer, + input_pipeline_hashes=hashes(*streams), + ) results = await out.reader.collect() assert len(results) == 2 @@ -1255,7 +1340,15 @@ async def test_three_way_partial_match_no_premature_emit(self): from orcapod.core.datagrams import Data, Tag join_task = asyncio.create_task( - op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=[ + root_hash("id", "a"), + root_hash("id", "b"), + root_hash("id", "c"), + ], + ) ) # Send matching rows from only 2 of 3 sides @@ -1299,7 +1392,11 @@ async def test_three_way_empty_side_produces_nothing(self): await feed(s1, ch1) await feed(s2, ch2) await ch3.writer.close() # empty third input - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=[*hashes(s1, s2), root_hash("id", "c")], + ) results = await out.reader.collect() assert results == [] @@ -1333,7 +1430,11 @@ async def test_three_way_no_shared_tags_cartesian(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out_ch.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out_ch.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) results = await out_ch.reader.collect() # 2 × 1 × 2 = 4 cartesian product @@ -1372,7 +1473,9 @@ async def test_duplicate_tag_keys_cross_product(self): out = Channel(buffer_size=64) await feed(s1, ch1) await feed(s2, ch2) - await op.async_execute([ch1.reader, ch2.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader], out.writer, input_pipeline_hashes=hashes(s1, s2) + ) async_results = await out.reader.collect() assert len(async_results) == 6 @@ -1424,7 +1527,11 @@ async def test_three_way_duplicate_keys_cross_product(self): await feed(s1, ch1) await feed(s2, ch2) await feed(s3, ch3) - await op.async_execute([ch1.reader, ch2.reader, ch3.reader], out.writer) + await op.async_execute( + [ch1.reader, ch2.reader, ch3.reader], + out.writer, + input_pipeline_hashes=hashes(s1, s2, s3), + ) async_results = await out.reader.collect() assert len(async_results) == 4 @@ -1537,7 +1644,11 @@ async def test_buffered_rows_both_sides_emit_on_reindex(self): from orcapod.core.datagrams import Data, Tag join_task = asyncio.create_task( - op.async_execute([ch1.reader, ch2.reader], out.writer) + op.async_execute( + [ch1.reader, ch2.reader], + out.writer, + input_pipeline_hashes=[root_hash("id", "a"), root_hash("id", "b")], + ) ) # Buffer 2 rows on side 0 before side 1 sends anything @@ -1588,8 +1699,18 @@ async def source(): async with asyncio.TaskGroup() as tg: tg.create_task(source()) - tg.create_task(select_op.async_execute([ch1.reader], ch2.writer)) - tg.create_task(map_op.async_execute([ch2.reader], ch3.writer)) + tg.create_task( + select_op.async_execute( + [ch1.reader], ch2.writer, input_pipeline_hashes=hashes(stream) + ) + ) + tg.create_task( + map_op.async_execute( + [ch2.reader], + ch3.writer, + input_pipeline_hashes=hashes(select_op(stream)), + ) + ) results = await ch3.reader.collect() assert len(results) == 3 @@ -1633,9 +1754,19 @@ async def push(stream, ch): tg.create_task(push(left, ch_l)) tg.create_task(push(right, ch_r)) tg.create_task( - join_op.async_execute([ch_l.reader, ch_r.reader], ch_joined.writer) + join_op.async_execute( + [ch_l.reader, ch_r.reader], + ch_joined.writer, + input_pipeline_hashes=hashes(left, right), + ) + ) + tg.create_task( + select_op.async_execute( + [ch_joined.reader], + ch_out.writer, + input_pipeline_hashes=hashes(join_op(left, right)), + ) ) - tg.create_task(select_op.async_execute([ch_joined.reader], ch_out.writer)) results = await ch_out.reader.collect() assert len(results) == 3 @@ -1666,9 +1797,19 @@ async def push(stream, ch): tg.create_task(push(left, ch_l)) tg.create_task(push(right, ch_r)) tg.create_task( - semi_op.async_execute([ch_l.reader, ch_r.reader], ch_semi.writer) + semi_op.async_execute( + [ch_l.reader, ch_r.reader], + ch_semi.writer, + input_pipeline_hashes=hashes(left, right), + ) + ) + tg.create_task( + batch_op.async_execute( + [ch_semi.reader], + ch_out.writer, + input_pipeline_hashes=hashes(semi_op(left, right)), + ) ) - tg.create_task(batch_op.async_execute([ch_semi.reader], ch_out.writer)) results = await ch_out.reader.collect() # SemiJoin produces 2 rows (id=[2,3]), Batch(2) → 1 batch @@ -1699,9 +1840,25 @@ async def source(): async with asyncio.TaskGroup() as tg: tg.create_task(source()) - tg.create_task(drop_op.async_execute([ch1.reader], ch2.writer)) - tg.create_task(map_op.async_execute([ch2.reader], ch3.writer)) - tg.create_task(select_op.async_execute([ch3.reader], ch4.writer)) + tg.create_task( + drop_op.async_execute( + [ch1.reader], ch2.writer, input_pipeline_hashes=hashes(stream) + ) + ) + tg.create_task( + map_op.async_execute( + [ch2.reader], + ch3.writer, + input_pipeline_hashes=hashes(drop_op(stream)), + ) + ) + tg.create_task( + select_op.async_execute( + [ch3.reader], + ch4.writer, + input_pipeline_hashes=hashes(map_op(drop_op(stream))), + ) + ) results = await ch4.reader.collect() assert len(results) == 3 diff --git a/tests/test_channels/test_pipeline_example.py b/tests/test_channels/test_pipeline_example.py index 129ae3e25..e4b1bc2e0 100644 --- a/tests/test_channels/test_pipeline_example.py +++ b/tests/test_channels/test_pipeline_example.py @@ -196,6 +196,10 @@ async def push_source(stream: ArrowTableStream, ch: Channel): join_op.async_execute( [ch_students.reader, ch_grades.reader], ch_joined.writer, + input_pipeline_hashes=[ + make_students().pipeline_hash(), + make_grades().pipeline_hash(), + ], ) ) @@ -204,6 +208,9 @@ async def push_source(stream: ArrowTableStream, ch: Channel): filter_op.async_execute( [ch_joined.reader], ch_filtered.writer, + input_pipeline_hashes=[ + join_op(make_students(), make_grades()).pipeline_hash() + ], ) ) @@ -253,12 +260,19 @@ async def push_source(stream, ch): join_op.async_execute( [ch_students.reader, ch_grades.reader], ch_joined.writer, + input_pipeline_hashes=[ + make_students().pipeline_hash(), + make_grades().pipeline_hash(), + ], ) ) tg.create_task( filter_op.async_execute( [ch_joined.reader], ch_filtered.writer, + input_pipeline_hashes=[ + join_op(make_students(), make_grades()).pipeline_hash() + ], ) ) tg.create_task( @@ -322,9 +336,24 @@ async def push(stream, ch): tg.create_task(push(make_students(), ch_s)) tg.create_task(push(make_grades(), ch_g)) tg.create_task( - join_op.async_execute([ch_s.reader, ch_g.reader], ch_j.writer) + join_op.async_execute( + [ch_s.reader, ch_g.reader], + ch_j.writer, + input_pipeline_hashes=[ + make_students().pipeline_hash(), + make_grades().pipeline_hash(), + ], + ) + ) + tg.create_task( + filter_op.async_execute( + [ch_j.reader], + ch_f.writer, + input_pipeline_hashes=[ + join_op(make_students(), make_grades()).pipeline_hash() + ], + ) ) - tg.create_task(filter_op.async_execute([ch_j.reader], ch_f.writer)) tg.create_task(grade_pod.async_execute([ch_f.reader], ch_o.writer)) return { diff --git a/tests/test_core/operators/test_chained_aggregation.py b/tests/test_core/operators/test_chained_aggregation.py index 2b8d12c4c..544c511e4 100644 --- a/tests/test_core/operators/test_chained_aggregation.py +++ b/tests/test_core/operators/test_chained_aggregation.py @@ -211,7 +211,11 @@ async def run() -> Rows: for row in upstream.iter_data(): await source.writer.send(row) await source.writer.close() - await op.async_execute([source.reader], sink.writer) + await op.async_execute( + [source.reader], + sink.writer, + input_pipeline_hashes=[upstream.pipeline_hash()], + ) return await sink.reader.collect() return asyncio.run(run()) diff --git a/tests/test_core/operators/test_index_pick_arrow_types.py b/tests/test_core/operators/test_index_pick_arrow_types.py index 82579b8b6..78cd3ab7c 100644 --- a/tests/test_core/operators/test_index_pick_arrow_types.py +++ b/tests/test_core/operators/test_index_pick_arrow_types.py @@ -471,7 +471,11 @@ async def _run_async(op: UnaryOperator, source: StreamProtocol) -> list[Row]: for tag, data in source.iter_data(): await input_ch.writer.send((tag, data)) await input_ch.writer.close() - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[source.pipeline_hash()], + ) return await output_ch.reader.collect() @@ -694,7 +698,11 @@ async def test_row_time_code_never_reads_a_context( for row in source.iter_data(): await input_ch.writer.send(row) await input_ch.writer.close() - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[source.pipeline_hash()], + ) async_rows = await output_ch.reader.collect() assert len(sync_rows) == len(async_rows) == 2 diff --git a/tests/test_core/operators/test_materialize_source_info.py b/tests/test_core/operators/test_materialize_source_info.py index b26a742a2..59a7c9f8b 100644 --- a/tests/test_core/operators/test_materialize_source_info.py +++ b/tests/test_core/operators/test_materialize_source_info.py @@ -391,7 +391,12 @@ async def go() -> list[Row]: empty: Channel[Row] = Channel(buffer_size=4) await empty.writer.close() output: Channel[Row] = Channel(buffer_size=4) - await make_op().async_execute([empty.reader], output.writer) + # An input that delivers no rows still has a pipeline hash. + await make_op().async_execute( + [empty.reader], + output.writer, + input_pipeline_hashes=[_source().pipeline_hash()], + ) return await output.reader.collect() # The generic barrier path (e.g. ``PolarsFilter``) still raises on zero rows, diff --git a/tests/test_core/test_regression_fixes.py b/tests/test_core/test_regression_fixes.py index 920c9173b..57680adc8 100644 --- a/tests/test_core/test_regression_fixes.py +++ b/tests/test_core/test_regression_fixes.py @@ -146,7 +146,11 @@ async def test_operator_closes_channel_on_static_process_error(self): await feed_stream_to_channel(stream, input_ch) with pytest.raises(Exception): - await op.async_execute([input_ch.reader], output_ch.writer) + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[stream.pipeline_hash()], + ) # Channel should be closed — collect should not hang. results = await output_ch.reader.collect() @@ -166,8 +170,13 @@ async def test_static_output_pod_closes_channel_on_empty_input(self): await input_ch.writer.close() # empty input - # Streaming async_execute handles empty input gracefully. - await op.async_execute([input_ch.reader], output_ch.writer) + # Streaming async_execute handles empty input gracefully. An input + # that delivers no rows still has a pipeline hash. + await op.async_execute( + [input_ch.reader], + output_ch.writer, + input_pipeline_hashes=[make_stream(3).pipeline_hash()], + ) # Channel should be closed and empty. results = await output_ch.reader.collect() From 467d9a24b48902c3f558eaed0770abbd21a22266 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:28:02 -0800 Subject: [PATCH 11/13] test(operators): narrow the pyright ignore on the missing-keyword call Co-Authored-By: Claude Opus 5.5 --- tests/test_core/operators/test_async_sync_equivalence.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 3ebb26355..ed95210d5 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -490,7 +490,7 @@ def test_async_execute_requires_input_pipeline_hashes(case: Case) -> None: channels: list[Channel[Row]] = [Channel() for _ in case.make_sources()] output: Channel[Row] = Channel() with pytest.raises(TypeError, match="input_pipeline_hashes"): - case.make_op().async_execute( # pyright: ignore[reportCallIssue, reportUnusedCoroutine] # the missing keyword is the point + case.make_op().async_execute( # pyright: ignore[reportCallIssue] # the missing keyword is the point [channel.reader for channel in channels], output.writer ) From 5a05ae7e1f19ad707f3a7e07affa8788458e5605 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:40:23 -0800 Subject: [PATCH 12/13] docs(plan): ITL-906 review round progress Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-11-12-itl-906-async-system-tag-names.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md index 5f90420a7..16a57ab7e 100644 --- a/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md +++ b/superpowers/plans/2026-11-12-itl-906-async-system-tag-names.md @@ -143,7 +143,8 @@ Decision (developer "Go with A", coordinator approval 2026-11-12): REQUIRE the h - Changelog ITL-906: add Join's old fallback. PR body: signature uniform and why. ### Steps -- [ ] Failing tests (`tests/test_core/operators/test_async_sync_equivalence.py`): each barrier +- [x] Failing tests (`tests/test_core/operators/test_async_sync_equivalence.py`): each barrier operator + Join without hashes raises `TypeError`; wrong length raises `ValueError` naming both. -- [ ] Implement; update direct test callers; design issue Fix note; changelog. -- [ ] Full suite 3.12, 3.11, 3.13; ruff; basedpyright; merge origin/main; one push; reply. +- [x] Implement; update direct test callers; design issue Fix note; changelog. +- [x] Full suite 3.12, 3.11, 3.13 (10167 passed, 109 skipped, 22 xfailed each); ruff; basedpyright exit 0; merged origin/main (290889e8). +- [ ] Push, PR body, reply on thread 4557365620, CodeRabbit on new head, CI green. From 4e70a3877ead765c548db5467b43ddec86ff19db Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:58:34 -0800 Subject: [PATCH 13/13] docs(operators): name the operators that check the hash count; document _RowsOfStream.pipeline_hash Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-906.md | 5 ++++- src/orcapod/core/operators/static_output_pod.py | 6 ++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/changelog.d/ITL-906.md b/changelog.d/ITL-906.md index b37666de6..ac9b37d6e 100644 --- a/changelog.d/ITL-906.md +++ b/changelog.d/ITL-906.md @@ -13,7 +13,10 @@ where these operators read a source directly are unaffected. Every operator's `async_execute` now requires the keyword argument `input_pipeline_hashes`, one pipeline hash per input, as the operator node always passes it; a missing argument raises -`TypeError` and a count that does not match the inputs raises `ValueError`. Before, these +`TypeError`. The operators that use the hashes (`Join`, `MergeJoin`, `Batch`, `GroupBy`, +`PolarsFilter`, and any operator on the default barrier path) raise `ValueError` when the count +does not match the inputs; the per-row operators and `SemiJoin` take the argument without +using it. Before, these operators fell back without it to the schema-only hash, and `Join`'s async path to names built from the input positions (`::0`, `::1`, ...), neither of which the sync path uses. Code that calls an operator's `async_execute` directly must pass the inputs' pipeline hashes. diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index e9f182097..748f41381 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -374,6 +374,12 @@ def pipeline_hash( It was computed by the operator node with the hasher of its data context, so it is returned whatever ``hasher`` is given. + + Args: + hasher: Not used. + + Returns: + The input pipeline hash the operator node passed. """ return self._input_pipeline_hash