From 247c552d925b35b5bdf2b32f82eab6a1edbea177 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 03:59:54 -0800 Subject: [PATCH 1/7] docs(plan): ITL-896 async operator read-back plan Co-Authored-By: Claude Opus 5.5 --- ...6-11-11-itl-896-async-operator-readback.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 superpowers/plans/2026-11-11-itl-896-async-operator-readback.md diff --git a/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md b/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md new file mode 100644 index 000000000..86ab7da95 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md @@ -0,0 +1,79 @@ +# ITL-896: operator node reads back empty after an async job run — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** After `PipelineJob.run(orchestrator=AsyncPipelineOrchestrator())`, `job.nodes[]` +returns the same rows from `as_table()` / `iter_data()` as after a sync run. + +**Architecture:** `OperatorJobNode` keeps its result only in memory (`_cached_output_stream`). +`execute()` (sync orchestrator) always keeps it; `async_execute()` keeps it only in +`CacheMode.LOG`, and inside a job every operator is `CacheMode.OFF`. Fix (option A, approved by +the coordinator 2026-11-11): `async_execute` collects the rows it forwards in every mode +(OFF, LOG, and the no-database branch) and stores them on the node with the same +`StaticOutputOperatorPod._materialize_to_stream` the sync path uses, regardless of +`materialize_results`. Nothing new is written to any store; no hash, path or schema changes. + +**Tech Stack:** Python, PyArrow, asyncio, pytest. + +## Root cause (STEP 1, done) + +- Reproduced through the public interface on `cf4f67c7` for select/drop/select-tag/map-data/ + map-tags/polars_filter(constraint)/Join/MergeJoin/SemiJoin/Batch/GroupBy and an + operator→function-pod chain, in-memory and Delta: sync gives the rows, async gives 0 rows, + no error. `get_all_records()` is `None` on both paths (operators in a job are + `CacheMode.OFF`, nothing is stored). +- Script: scratchpad `repro.py` (not committed). +- `get_all_records()` stays `None` for an operator node that stored nothing (approved); + document that operator nodes in a job keep results in memory only and correct the + `PipelineJob.run` `materialize_results` docstring. + +## Open question to coordinator + +After `clear_cache()` an operator node returns a zero-row table on both paths today (the same +"not yet run" state as before `run()`). Coordinator asked to be told before changing it. + +## Files + +- Modify: `src/orcapod/core/nodes/operator_node.py` (`OperatorJobNode.async_execute`) +- Modify: `src/orcapod/pipeline/job.py` (`run` docstring) +- Create: `tests/test_pipeline/test_operator_node_async_readback.py` +- Modify: `docs/concepts/function-pods.md` (node read-back / `get_all_records` for operators) +- Modify: `CHANGELOG.md` (`[0.3.0] — unreleased`, Fixed) +- Modify: `DESIGN_ISSUES.md` (one entry appended at the end, resolved) + +### Task 1: failing tests + +- [ ] Write `tests/test_pipeline/test_operator_node_async_readback.py`: parametrize over every + operator (built with the stream convenience methods / operator call, labelled `"op"`), + store in {in-memory, Delta}, `materialize_results` in {True, False}. Run the same job twice + (fresh store each) with sync and async orchestrators. Assert: + - async `as_table(all_info=True)` sorted by tag columns equals sync (schema incl. nullability + and column order, values, system tags); + - `iter_data()` row count equal and per-row tag/data `content_hash()` multisets equal; + - row count > 0 (guard against both being empty); + - `get_all_records()` is `None` on both. +- [ ] Run, confirm async cases FAIL (0 rows) and sync cases pass. +- [ ] Commit WIP. + +### Task 2: fix + +- [ ] In `OperatorJobNode.async_execute`, no-DB branch: wrap the output writer to collect rows, + then set `_cached_output_stream` + `_update_modified_time()`. +- [ ] DB branch OFF/LOG: always collect; after the TaskGroup, if rows were collected set + `_cached_output_stream = _materialize_to_stream(collected)`; store only in LOG. + `_update_modified_time()` as today. +- [ ] Run new tests + `tests/test_core/operators tests/test_pipeline tests/test_channels`. +- [ ] Commit. + +### Task 3: docs, CHANGELOG, DESIGN_ISSUES + +- [ ] function-pods.md: operator nodes keep their output in memory for the life of the job + object, on both orchestrators; nothing is stored, so `get_all_records()` returns `None`. +- [ ] job.py `run` docstring: correct the `get_all_records()` claim. +- [ ] CHANGELOG Fixed entry; DESIGN_ISSUES entry (resolved, Fix note). +- [ ] `uv run pytest tests/test_docs` + mkdocs strict. + +### Task 4: verify & ship + +- [ ] ruff check/format, basedpyright exit code, full `uv run pytest tests/`, golden hashes unchanged. +- [ ] Merge origin/main, push, PR (Fixes ITL-896), link, CI, CodeRabbit (inline + review bodies). From d6dbe7618c3946d45228d4863f0f75febfb76bd8 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 04:04:17 -0800 Subject: [PATCH 2/7] fix(operator_node): keep an operator node's output after an async run (ITL-896) WIP: failing tests for zero-row, MergeJoin-after-operator and clear_cache types pending a coordinator decision. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/nodes/operator_node.py | 45 ++-- .../test_operator_node_async_readback.py | 248 ++++++++++++++++++ 2 files changed, 271 insertions(+), 22 deletions(-) create mode 100644 tests/test_pipeline/test_operator_node_async_readback.py diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index f343ba67f..629910349 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -1192,13 +1192,18 @@ async def async_execute( ) -> None: """Async execution with cache mode handling when DB is attached. - Without a database, delegates to the wrapped operator's - ``async_execute``. + Every mode except REPLAY runs the wrapped operator's ``async_execute``, + forwards its rows downstream and keeps them on the node, exactly as + ``execute()`` does, so ``iter_data()`` and ``as_table()`` return them + after the run whichever orchestrator ran the node. + + Without a database, or with one in OFF mode, nothing is stored. With a database: - REPLAY: emit from DB, close output. - - OFF: delegate to operator, forward results. - - LOG: delegate to operator, forward + collect results, then store in DB. + - OFF: delegate to operator, forward and keep results. + - LOG: delegate to operator, forward and keep results, then store + them in the DB. Args: inputs: Sequence of readable channels from upstream nodes. @@ -1212,20 +1217,13 @@ async def async_execute( node_hash = self.content_hash().to_string() ctx_obs = obs.contextualize(*self.node_identity_path) - if self._pipeline_database is None: - # Simple delegation without DB - ctx_obs.on_node_start(node_label, node_hash) - hashes = [s.pipeline_hash() for s in self._input_streams] - await self._operator.async_execute( - inputs, output, input_pipeline_hashes=hashes - ) - ctx_obs.on_node_end(node_label, node_hash) - return - try: ctx_obs.on_node_start(node_label, node_hash) - if self._cache_mode == CacheMode.REPLAY: + if ( + self._pipeline_database is not None + and self._cache_mode == CacheMode.REPLAY + ): self._replay_from_cache() assert self._cached_output_stream is not None for tag, data in self._cached_output_stream.iter_data(): @@ -1233,15 +1231,13 @@ async def async_execute( ctx_obs.on_node_end(node_label, node_hash) return # finally block closes output - # OFF or LOG: delegate to operator, forward results downstream + # Delegate to the operator, forward its rows downstream and keep them. intermediate: Channel[tuple[TagProtocol, DataProtocol]] = Channel() - should_collect = self._cache_mode == CacheMode.LOG collected: list[tuple[TagProtocol, DataProtocol]] = [] async def forward() -> None: async for item in intermediate.reader: - if should_collect: - collected.append(item) + collected.append(item) await output.send(item) hashes = [s.pipeline_hash() for s in self._input_streams] @@ -1255,11 +1251,16 @@ async def forward() -> None: ) tg.create_task(forward()) - # TaskGroup has completed — store if LOG mode (sync DB write, post-hoc) - if should_collect and collected: + # TaskGroup has completed. Keep the output on the node, as execute() + # does, and store it in LOG mode (sync DB write, post-hoc). + if collected: stream = StaticOutputOperatorPod._materialize_to_stream(collected) self._cached_output_stream = stream - self._store_output_stream(stream) + if ( + self._pipeline_database is not None + and self._cache_mode == CacheMode.LOG + ): + self._store_output_stream(stream) self._update_modified_time() diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py new file mode 100644 index 000000000..2334f1332 --- /dev/null +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -0,0 +1,248 @@ +"""After a job run, an operator node reads back the same rows on both orchestrators (ITL-896). + +``job.nodes[label]`` is a stream: ``as_table()`` and ``iter_data()`` must return the rows the +operator produced, whether the job ran on ``SyncPipelineOrchestrator`` or +``AsyncPipelineOrchestrator``, with either store and either ``materialize_results`` setting. +An operator node in a job stores nothing (``cache_mode`` is ``OFF``), so ``get_all_records()`` +returns ``None`` on both. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod.core.operators import ( + Batch, + DropDataColumns, + DropTagColumns, + GroupBy, + Join, + MapData, + MapTags, + MergeJoin, + PolarsFilter, + SelectDataColumns, + SelectTagColumns, + SemiJoin, +) +from orcapod.core.data_function import PythonDataFunction +from orcapod.core.function_pod import FunctionPod +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator +from orcapod.pipeline.job import PipelineJob +from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator +from orcapod.protocols.core_protocols import StreamProtocol + +ORCHESTRATORS = { + "sync": SyncPipelineOrchestrator, + "async": AsyncPipelineOrchestrator, +} + + +def _source_a() -> ArrowTableSource: + table = pa.table( + { + "id": pa.array([1, 2, 3], type=pa.int8()), + "grp": pa.array(["g1", "g1", "g2"], type=pa.large_string()), + "x": pa.array([10, 20, 30], type=pa.int8()), + "s": pa.array(["p", None, "r"], type=pa.string()), + }, + schema=pa.schema( + [ + pa.field("id", pa.int8(), nullable=False), + pa.field("grp", pa.large_string(), nullable=False), + pa.field("x", pa.int8(), nullable=False), + pa.field("s", pa.string(), nullable=True), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id", "grp"], source_id="a") + + +def _source_b() -> ArrowTableSource: + table = pa.table( + { + "id": pa.array([1, 2, 3], type=pa.int8()), + "z": pa.array([5, 6, 7], type=pa.int16()), + }, + schema=pa.schema( + [ + pa.field("id", pa.int8(), nullable=False), + pa.field("z", pa.int16(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="b") + + +def _source_c() -> ArrowTableSource: + table = pa.table( + { + "id": pa.array([1, 2, 3], type=pa.int8()), + "grp": pa.array(["g1", "g1", "g2"], type=pa.large_string()), + "x": pa.array([11, 21, 31], type=pa.int8()), + }, + schema=pa.schema( + [ + pa.field("id", pa.int8(), nullable=False), + pa.field("grp", pa.large_string(), nullable=False), + pa.field("x", pa.int8(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id", "grp"], source_id="c") + + +def _double_x(x: int) -> int: + return x * 2 + + +@dataclass(frozen=True) +class Case: + """An operator applied inside a job; the operator node under test is labelled ``op``.""" + + name: str + build: Callable[[StreamProtocol, StreamProtocol, StreamProtocol], Any] + expected_rows: int + + +CASES = [ + Case( + "select_data_columns", + lambda a, b, c: SelectDataColumns(["x"])(a, label="op"), + 3, + ), + Case("drop_data_columns", lambda a, b, c: DropDataColumns(["s"])(a, label="op"), 3), + Case( + "select_tag_columns", + lambda a, b, c: SelectTagColumns(["id"])(a, label="op"), + 3, + ), + Case("drop_tag_columns", lambda a, b, c: DropTagColumns(["grp"])(a, label="op"), 3), + Case("map_data", lambda a, b, c: MapData({"x": "w"})(a, label="op"), 3), + Case("map_tags", lambda a, b, c: MapTags({"grp": "group"})(a, label="op"), 3), + Case( + "polars_filter", + lambda a, b, c: PolarsFilter(constraints={"grp": "g1"})(a, label="op"), + 2, + ), + Case( + "polars_filter_no_match", + lambda a, b, c: PolarsFilter(constraints={"grp": "none"})(a, label="op"), + 0, + ), + Case("join", lambda a, b, c: Join()(a, b, label="op"), 3), + Case( + "merge_join", + lambda a, b, c: MergeJoin()(SelectDataColumns(["x"])(a), c, label="op"), + 3, + ), + Case("semi_join", lambda a, b, c: SemiJoin()(a, b, label="op"), 3), + Case("batch", lambda a, b, c: Batch(batch_size=2)(a, label="op"), 2), + Case("group_by", lambda a, b, c: GroupBy(by=["grp"])(a, label="op"), 2), + Case( + "operator_feeding_function_pod", + lambda a, b, c: FunctionPod( + PythonDataFunction(_double_x, output_keys="x2") + )(SelectDataColumns(["x"])(a, label="op"), label="double"), + 3, + ), +] + + +def _make_store(kind: str, tmp_path: Path) -> Any: + if kind == "in_memory": + return InMemoryArrowDatabase() + return DeltaTableDatabase(base_path=tmp_path / "store") + + +def _run_job( + case: Case, + orchestrator: str, + store_kind: str, + tmp_path: Path, + materialize_results: bool = True, +) -> PipelineJob: + job = PipelineJob( + name="itl896", store=_make_store(store_kind, tmp_path / orchestrator) + ) + with job: + case.build(_source_a(), _source_b(), _source_c()) + job.run( + orchestrator=ORCHESTRATORS[orchestrator](), + materialize_results=materialize_results, + ) + return job + + +def _table_rows(table: pa.Table) -> list[dict[str, Any]]: + """Rows of ``table`` in a canonical order, so arrival order does not matter.""" + return sorted(table.to_pylist(), key=repr) + + +def _row_hashes(node: Any) -> list[tuple[str, str]]: + return sorted( + (tag.content_hash().to_string(), data.content_hash().to_string()) + for tag, data in node.iter_data() + ) + + +@pytest.mark.parametrize("materialize_results", [True, False]) +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("case", CASES, ids=lambda case: case.name) +def test_operator_node_reads_back_same_rows_after_async_run( + case: Case, store_kind: str, materialize_results: bool, tmp_path: Path +) -> None: + sync_node = _run_job( + case, "sync", store_kind, tmp_path, materialize_results + ).nodes["op"] + async_node = _run_job( + case, "async", store_kind, tmp_path, materialize_results + ).nodes["op"] + + sync_table = sync_node.as_table(all_info=True) + async_table = async_node.as_table(all_info=True) + + assert sync_table.num_rows == case.expected_rows + assert async_table.num_rows == case.expected_rows + # Types, nullability and column order, including system tags and source info. + assert async_table.schema == sync_table.schema + assert _table_rows(async_table) == _table_rows(sync_table) + assert _table_rows(async_node.as_table()) == _table_rows(sync_node.as_table()) + + assert len(list(async_node.iter_data())) == case.expected_rows + assert _row_hashes(async_node) == _row_hashes(sync_node) + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_operator_node_stores_nothing_so_get_all_records_is_none( + orchestrator: str, tmp_path: Path +) -> None: + node = _run_job(CASES[0], orchestrator, "in_memory", tmp_path).nodes["op"] + + assert node.as_table().num_rows == 3 + assert node.get_all_records() is None + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_clear_cache_returns_operator_node_to_not_yet_run( + orchestrator: str, tmp_path: Path +) -> None: + node = _run_job(CASES[0], orchestrator, "in_memory", tmp_path).nodes["op"] + schema_after_run = node.as_table().schema + + # An operator node keeps its output in memory only: clear_cache() discards it, and + # the node reads as a zero-row table with its schema, as before the first run. + node.clear_cache() + + table = node.as_table() + assert table.num_rows == 0 + assert table.schema == schema_after_run + assert list(node.iter_data()) == [] From 303de51d1c2ae347fb925713c30299a0aa2c5ce9 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 04:11:12 -0800 Subject: [PATCH 3/7] fix(operator_node): keep an operator node's output after an async run (ITL-896) OperatorJobNode.async_execute kept the rows it forwarded only in cache_mode LOG, so after an async job run an operator node read back the zero-row 'not yet run' table. It now keeps them in every mode except REPLAY, as execute() does. Nothing new is stored. Strict xfails pin the pre-existing findings this exposes: zero-row async output types (ITL-858/ITL-865), the zero-row table widening types (ITL-912) and async MergeJoin system-tag names after a non-source input (ITL-906). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 7 + DESIGN_ISSUES.md | 29 +++ docs/concepts/function-pods.md | 7 + src/orcapod/core/nodes/operator_node.py | 10 +- src/orcapod/pipeline/job.py | 8 +- .../test_operator_node_async_readback.py | 170 +++++++++++++----- 6 files changed, 184 insertions(+), 47 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffc35f547..a40823ce7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,13 @@ ### Fixed +- After a job ran on `AsyncPipelineOrchestrator`, an operator node (`job.nodes[label]` for + `Join`, `MergeJoin`, `SemiJoin`, `Batch`, `GroupBy`, `MapTags`, `MapData`, `PolarsFilter` and + the column-selection operators) returned no rows from `as_table()` and `iter_data()`, without + an error. It now returns the same rows as after a sync run, whatever `materialize_results` + is. An operator node in a job keeps its output in memory and stores nothing, so + `get_all_records()` returns `None` for it on both orchestrators. (ITL-896) + - A failed store write fails the run. `PipelineJob.run()` used to return normally when a node's result or record could not be written (the row was skipped like a crashed function under the default `error_policy="continue"`), so the node read back fewer rows than it diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index ba4c26c34..8a260f98f 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2293,3 +2293,32 @@ xfail. The async path checks key types on the first row of each input, because So when either input is empty (left or right), a key type mismatch that the sync path reports yields an empty result on the async path instead of an error. `Join`'s async key check (`_check_shared_key_types`) has the same limit. + +--- + +## `src/orcapod/core/nodes/operator_node.py` — operator node read-back after an async run + +### OR1 — An operator node read back empty after an async job run +**Status:** resolved +**Severity:** high +**Issue:** ITL-896 + +An operator node keeps its output only in memory; inside a `PipelineJob` every operator runs +with `cache_mode=OFF` and stores nothing. The sync orchestrator calls `OperatorJobNode.execute()`, +which always keeps the materialized output on the node. The async orchestrator calls +`OperatorJobNode.async_execute()`, which kept the rows only in `CacheMode.LOG`; in `OFF` (and with +no database) it forwarded them downstream and dropped them. `iter_data()` and `as_table()` then +fell back to the database, which they read only in `REPLAY`, and returned the zero-row +"not yet run" table without an error. Every operator was affected, on every store. + +**Fix:** `async_execute` collects the rows it forwards in every mode except `REPLAY` and keeps +them on the node with `StaticOutputOperatorPod._materialize_to_stream`, as the sync path does, +whatever `materialize_results` is; `LOG` still stores them. Nothing new is stored, and no hash, +path or stored schema changes. `get_all_records()` still returns `None` for an operator node in a +job, since it stores nothing. `clear_cache()` still returns the node to its zero-row "not yet run" +state on both paths (a function node instead reloads from its store). +Regressions: `tests/test_pipeline/test_operator_node_async_readback.py`, which compares the sync +and async reads of every operator. Open, pinned by strict xfails there: a zero-row async result +reads back the table built from the Python output schema (ITL-858, ITL-865); that table widens +`int8` to `int64` and changes nullability on both paths (ITL-912); `MergeJoin` after a non-source +input names its system-tag columns differently on the async path (ITL-906). diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 61ea20fca..bc5ca2b31 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -228,6 +228,13 @@ A node from a job that has run also provides: - **`as_source()`** -- a `DerivedSource` backed by those stored records, usable as the input of another pipeline +An operator node (a `Join`, `Batch`, column selection and so on, applied inside the job) is +read the same way: after `job.run()` its `as_table()` and `iter_data()` return the rows it +produced, on both the synchronous and the asynchronous orchestrator. It keeps that output in +memory and stores nothing, so its `get_all_records()` returns `None`. Calling `clear_cache()` +on it discards the output, and it then reads as a zero-row table, as before the first run, +until the job runs again. + ### When the store cannot be written A failed write to the store always fails the run. If a node's results or records cannot be diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 629910349..ff8b3d5a4 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -1058,9 +1058,10 @@ def run(self) -> None: def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Return an iterator over (tag, data) pairs. - Read-only: never triggers computation. Returns empty before ``run()`` - or ``execute()`` populates the cache. Call ``node.is_stale`` before - iterating if you need to detect outdated cached data. + Read-only: never triggers computation. Returns empty before ``run()``, + ``execute()`` or ``async_execute()`` populates the cache, and after + ``clear_cache()``. Call ``node.is_stale`` before iterating if you need + to detect outdated cached data. """ if self._cached_output_stream is not None: return self._cached_output_stream.iter_data() @@ -1078,7 +1079,8 @@ def as_table( """Return the output as a PyArrow table. Read-only: never triggers computation. Returns a zero-row table - with correct schema before ``run()`` or ``execute()`` is called. + with the node's columns before ``run()``, ``execute()`` or + ``async_execute()`` is called, and after ``clear_cache()``. """ if self._cached_output_stream is not None: return self._cached_output_stream.as_table( diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 7fe706f82..2e5eeedf7 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -755,9 +755,11 @@ def run( the duration of the run. When ``False`` each node's buffer is released as soon as the node completes, reducing peak memory. This setting does not affect database persistence or post-run - result access — DB records are written regardless, and - ``job.nodes["label"].get_all_records()`` works the same in - either case. + result access: DB records are written regardless, and + ``job.nodes["label"]`` returns the same rows from + ``as_table()`` and ``iter_data()`` in either case, on either + orchestrator. An operator node keeps its output in memory and + stores nothing, so its ``get_all_records()`` returns ``None``. Returns: ``self`` — the same ``PipelineJob`` instance, with diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index 2334f1332..3e2db0ea9 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -12,7 +12,6 @@ from collections.abc import Callable from dataclasses import dataclass from pathlib import Path -from typing import Any import pyarrow as pa import pytest @@ -31,14 +30,15 @@ SelectTagColumns, SemiJoin, ) -from orcapod.core.data_function import PythonDataFunction -from orcapod.core.function_pod import FunctionPod +from orcapod import function_pod +from orcapod.core.nodes import OperatorJobNode from orcapod.core.sources import ArrowTableSource from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator from orcapod.pipeline.job import PipelineJob from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol ORCHESTRATORS = { "sync": SyncPipelineOrchestrator, @@ -100,7 +100,8 @@ def _source_c() -> ArrowTableSource: return ArrowTableSource(table, tag_columns=["id", "grp"], source_id="c") -def _double_x(x: int) -> int: +@function_pod(output_keys="x2") +def double_x(x: int) -> int: return x * 2 @@ -109,7 +110,7 @@ class Case: """An operator applied inside a job; the operator node under test is labelled ``op``.""" name: str - build: Callable[[StreamProtocol, StreamProtocol, StreamProtocol], Any] + build: Callable[[StreamProtocol, StreamProtocol, StreamProtocol], object] expected_rows: int @@ -133,31 +134,22 @@ class Case: lambda a, b, c: PolarsFilter(constraints={"grp": "g1"})(a, label="op"), 2, ), - Case( - "polars_filter_no_match", - lambda a, b, c: PolarsFilter(constraints={"grp": "none"})(a, label="op"), - 0, - ), Case("join", lambda a, b, c: Join()(a, b, label="op"), 3), - Case( - "merge_join", - lambda a, b, c: MergeJoin()(SelectDataColumns(["x"])(a), c, label="op"), - 3, - ), + Case("merge_join", lambda a, b, c: MergeJoin()(a, c, label="op"), 3), Case("semi_join", lambda a, b, c: SemiJoin()(a, b, label="op"), 3), Case("batch", lambda a, b, c: Batch(batch_size=2)(a, label="op"), 2), Case("group_by", lambda a, b, c: GroupBy(by=["grp"])(a, label="op"), 2), Case( "operator_feeding_function_pod", - lambda a, b, c: FunctionPod( - PythonDataFunction(_double_x, output_keys="x2") - )(SelectDataColumns(["x"])(a, label="op"), label="double"), + lambda a, b, c: double_x.pod( + SelectDataColumns(["x"])(a, label="op"), label="double" + ), 3, ), ] -def _make_store(kind: str, tmp_path: Path) -> Any: +def _make_store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: if kind == "in_memory": return InMemoryArrowDatabase() return DeltaTableDatabase(base_path=tmp_path / "store") @@ -174,20 +166,26 @@ def _run_job( name="itl896", store=_make_store(store_kind, tmp_path / orchestrator) ) with job: - case.build(_source_a(), _source_b(), _source_c()) - job.run( + _ = case.build(_source_a(), _source_b(), _source_c()) + _ = job.run( orchestrator=ORCHESTRATORS[orchestrator](), materialize_results=materialize_results, ) return job -def _table_rows(table: pa.Table) -> list[dict[str, Any]]: +def _op_node(job: PipelineJob) -> OperatorJobNode: + node: object = job.nodes["op"] + assert isinstance(node, OperatorJobNode) + return node + + +def _table_rows(table: pa.Table) -> list[dict[str, object]]: """Rows of ``table`` in a canonical order, so arrival order does not matter.""" return sorted(table.to_pylist(), key=repr) -def _row_hashes(node: Any) -> list[tuple[str, str]]: +def _row_hashes(node: OperatorJobNode) -> list[tuple[str, str]]: return sorted( (tag.content_hash().to_string(), data.content_hash().to_string()) for tag, data in node.iter_data() @@ -196,16 +194,16 @@ def _row_hashes(node: Any) -> list[tuple[str, str]]: @pytest.mark.parametrize("materialize_results", [True, False]) @pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) -@pytest.mark.parametrize("case", CASES, ids=lambda case: case.name) +@pytest.mark.parametrize("case", CASES, ids=[case.name for case in CASES]) def test_operator_node_reads_back_same_rows_after_async_run( case: Case, store_kind: str, materialize_results: bool, tmp_path: Path ) -> None: - sync_node = _run_job( - case, "sync", store_kind, tmp_path, materialize_results - ).nodes["op"] - async_node = _run_job( - case, "async", store_kind, tmp_path, materialize_results - ).nodes["op"] + sync_node = _op_node( + _run_job(case, "sync", store_kind, tmp_path, materialize_results) + ) + async_node = _op_node( + _run_job(case, "async", store_kind, tmp_path, materialize_results) + ) sync_table = sync_node.as_table(all_info=True) async_table = async_node.as_table(all_info=True) @@ -225,24 +223,116 @@ def test_operator_node_reads_back_same_rows_after_async_run( def test_operator_node_stores_nothing_so_get_all_records_is_none( orchestrator: str, tmp_path: Path ) -> None: - node = _run_job(CASES[0], orchestrator, "in_memory", tmp_path).nodes["op"] + node = _op_node(_run_job(CASES[0], orchestrator, "in_memory", tmp_path)) assert node.as_table().num_rows == 3 assert node.get_all_records() is None -@pytest.mark.parametrize("orchestrator", ["sync", "async"]) -def test_clear_cache_returns_operator_node_to_not_yet_run( - orchestrator: str, tmp_path: Path +NO_MATCH = Case( + "polars_filter_no_match", + lambda a, b, c: PolarsFilter(constraints={"grp": "none"})(a, label="op"), + 0, +) + +MERGE_JOIN_AFTER_OPERATOR = Case( + "merge_join_after_operator", + lambda a, b, c: MergeJoin()(SelectDataColumns(["x"])(a), c, label="op"), + 3, +) + + +def _sync_and_async_nodes( + case: Case, tmp_path: Path +) -> tuple[OperatorJobNode, OperatorJobNode]: + sync_node = _op_node(_run_job(case, "sync", "in_memory", tmp_path)) + async_node = _op_node(_run_job(case, "async", "in_memory", tmp_path)) + return sync_node, async_node + + +def test_operator_node_with_no_rows_reads_back_empty_after_async_run( + tmp_path: Path, ) -> None: - node = _run_job(CASES[0], orchestrator, "in_memory", tmp_path).nodes["op"] - schema_after_run = node.as_table().schema + sync_node, async_node = _sync_and_async_nodes(NO_MATCH, tmp_path) + + sync_table = sync_node.as_table() + async_table = async_node.as_table() + + assert sync_table.num_rows == 0 + assert async_table.num_rows == 0 + assert async_table.column_names == sync_table.column_names + assert list(async_node.iter_data()) == [] + + +@pytest.mark.xfail( + strict=True, + reason=( + "ITL-858/ITL-865: no row reaches the async node, so it reads back the table " + "built from the Python output schema: int8 becomes int64, and the system-tag " + "and source-info columns are missing" + ), +) +def test_operator_node_with_no_rows_keeps_columns_and_types_after_async_run( + tmp_path: Path, +) -> None: + sync_node, async_node = _sync_and_async_nodes(NO_MATCH, tmp_path) + + assert async_node.as_table(all_info=True).schema == ( + sync_node.as_table(all_info=True).schema + ) + + +@pytest.mark.xfail( + strict=True, + reason=( + "ITL-906: the async MergeJoin names system-tag columns from the schema-only " + "hash of its materialized inputs, not from its upstreams' pipeline hashes" + ), +) +def test_merge_join_after_an_operator_reads_back_same_system_tags_after_async_run( + tmp_path: Path, +) -> None: + sync_node, async_node = _sync_and_async_nodes(MERGE_JOIN_AFTER_OPERATOR, tmp_path) + + assert async_node.as_table(all_info=True).column_names == ( + sync_node.as_table(all_info=True).column_names + ) + + +def _cleared_table(orchestrator: str, tmp_path: Path) -> tuple[pa.Table, pa.Table]: + """The node's table after a run, and after ``clear_cache()``.""" + node = _op_node(_run_job(CASES[0], orchestrator, "in_memory", tmp_path)) + after_run = node.as_table() # An operator node keeps its output in memory only: clear_cache() discards it, and - # the node reads as a zero-row table with its schema, as before the first run. + # the node reads as a zero-row table, as before the first run. node.clear_cache() - table = node.as_table() - assert table.num_rows == 0 - assert table.schema == schema_after_run assert list(node.iter_data()) == [] + return after_run, node.as_table() + + +def test_clear_cache_returns_operator_node_to_not_yet_run(tmp_path: Path) -> None: + sync_after_run, sync_cleared = _cleared_table("sync", tmp_path / "sync") + async_after_run, async_cleared = _cleared_table("async", tmp_path / "async") + + assert sync_after_run.num_rows == async_after_run.num_rows == 3 + assert sync_cleared.num_rows == async_cleared.num_rows == 0 + assert sync_cleared.column_names == sync_after_run.column_names + assert async_cleared.schema == sync_cleared.schema + + +@pytest.mark.xfail( + strict=True, + reason=( + "ITL-912: the zero-row table built from the Python output schema widens " + "int8 to int64 and changes nullability" + ), +) +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_clear_cache_keeps_operator_node_arrow_types( + orchestrator: str, tmp_path: Path +) -> None: + after_run, cleared = _cleared_table(orchestrator, tmp_path) + + assert cleared.schema == after_run.schema From 475d4cad55731f3368191d43c696543b43f8cd15 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 04:39:11 -0800 Subject: [PATCH 4/7] fix(operator_node): clear an operator node's output on an empty async rerun (ITL-896) Address CodeRabbit on PR #302: an async run with no rows left the previous run's rows on the node. A run now always replaces the output. Regression tests rerun a job with no rows and with other rows on both orchestrators. Qualify the no-storage wording to the default cache_mode=OFF and add docstrings to the new tests. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 +- DESIGN_ISSUES.md | 6 +- docs/concepts/function-pods.md | 8 +-- src/orcapod/core/nodes/operator_node.py | 5 +- src/orcapod/pipeline/job.py | 6 +- .../test_operator_node_async_readback.py | 72 ++++++++++++++++++- 6 files changed, 90 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 145e71be6..8f54ff729 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,8 +14,10 @@ `Join`, `MergeJoin`, `SemiJoin`, `Batch`, `GroupBy`, `MapTags`, `MapData`, `PolarsFilter` and the column-selection operators) returned no rows from `as_table()` and `iter_data()`, without an error. It now returns the same rows as after a sync run, whatever `materialize_results` - is. An operator node in a job keeps its output in memory and stores nothing, so - `get_all_records()` returns `None` for it on both orchestrators. (ITL-896) + is, and a later run replaces those rows even when it produces none. With the default + `cache_mode=OFF` (the only mode a job sets) an operator node keeps its output in memory and + stores nothing, so `get_all_records()` returns `None` for it on both orchestrators. + (ITL-896) - A failed store write fails the run. `PipelineJob.run()` used to return normally when a node's result or record could not be written (the row was skipped like a crashed function diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 42d31fb3e..9a7f6bb55 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2349,8 +2349,10 @@ fell back to the database, which they read only in `REPLAY`, and returned the ze **Fix:** `async_execute` collects the rows it forwards in every mode except `REPLAY` and keeps them on the node with `StaticOutputOperatorPod._materialize_to_stream`, as the sync path does, whatever `materialize_results` is; `LOG` still stores them. Nothing new is stored, and no hash, -path or stored schema changes. `get_all_records()` still returns `None` for an operator node in a -job, since it stores nothing. `clear_cache()` still returns the node to its zero-row "not yet run" +path or stored schema changes. A run with no rows clears the previous run's output, so a node +never serves stale rows (CodeRabbit on PR #302). `get_all_records()` still returns `None` for an +operator node in a job, since with the default `cache_mode=OFF` (the only mode a job sets) it +stores nothing. `clear_cache()` still returns the node to its zero-row "not yet run" state on both paths (a function node instead reloads from its store). Regressions: `tests/test_pipeline/test_operator_node_async_readback.py`, which compares the sync and async reads of every operator. Open, pinned by strict xfails there: a zero-row async result diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index a14d8bf2d..508463a18 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -231,10 +231,10 @@ A node from a job that has run also provides: An operator node (a `Join`, `Batch`, column selection and so on, applied inside the job) is read the same way: after `job.run()` its `as_table()` and `iter_data()` return the rows it -produced, on both the synchronous and the asynchronous orchestrator. It keeps that output in -memory and stores nothing, so its `get_all_records()` returns `None`. Calling `clear_cache()` -on it discards the output, and it then reads as a zero-row table, as before the first run, -until the job runs again. +produced in the latest run, on both the synchronous and the asynchronous orchestrator. It +keeps that output in memory and, with the default `cache_mode=OFF`, stores nothing, so its +`get_all_records()` returns `None`. Calling `clear_cache()` on it discards the output, and it +then reads as a zero-row table, as before the first run, until the job runs again. ### When the store cannot be written diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index ff8b3d5a4..2353f9267 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -1254,7 +1254,10 @@ async def forward() -> None: tg.create_task(forward()) # TaskGroup has completed. Keep the output on the node, as execute() - # does, and store it in LOG mode (sync DB write, post-hoc). + # does, and store it in LOG mode (sync DB write, post-hoc). A run + # with no rows replaces an earlier run's output too: the node then + # reads as a zero-row table. + self._cached_output_stream = None if collected: stream = StaticOutputOperatorPod._materialize_to_stream(collected) self._cached_output_stream = stream diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 2e5eeedf7..41e5cff5e 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -758,8 +758,10 @@ def run( result access: DB records are written regardless, and ``job.nodes["label"]`` returns the same rows from ``as_table()`` and ``iter_data()`` in either case, on either - orchestrator. An operator node keeps its output in memory and - stores nothing, so its ``get_all_records()`` returns ``None``. + orchestrator. An operator node with the default + ``cache_mode=OFF`` (the only mode a job sets) keeps its output + in memory and stores nothing, so its ``get_all_records()`` + returns ``None``. Returns: ``self`` — the same ``PipelineJob`` instance, with diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index 3e2db0ea9..c3c314053 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -3,8 +3,8 @@ ``job.nodes[label]`` is a stream: ``as_table()`` and ``iter_data()`` must return the rows the operator produced, whether the job ran on ``SyncPipelineOrchestrator`` or ``AsyncPipelineOrchestrator``, with either store and either ``materialize_results`` setting. -An operator node in a job stores nothing (``cache_mode`` is ``OFF``), so ``get_all_records()`` -returns ``None`` on both. +An operator node in a job runs with the default ``cache_mode=OFF`` and stores nothing, so +``get_all_records()`` returns ``None`` on both. """ from __future__ import annotations @@ -12,6 +12,7 @@ from collections.abc import Callable from dataclasses import dataclass from pathlib import Path +from typing import cast import pyarrow as pa import pytest @@ -47,6 +48,7 @@ def _source_a() -> ArrowTableSource: + """Two tags and two data columns: non-nullable ``int8`` and a nullable ``string``.""" table = pa.table( { "id": pa.array([1, 2, 3], type=pa.int8()), @@ -67,6 +69,7 @@ def _source_a() -> ArrowTableSource: def _source_b() -> ArrowTableSource: + """Shares the ``id`` tag with ``_source_a`` and adds an ``int16`` column for joins.""" table = pa.table( { "id": pa.array([1, 2, 3], type=pa.int8()), @@ -83,6 +86,7 @@ def _source_b() -> ArrowTableSource: def _source_c() -> ArrowTableSource: + """Same tags and ``x`` column as ``_source_a``, so ``MergeJoin`` merges ``x``.""" table = pa.table( { "id": pa.array([1, 2, 3], type=pa.int8()), @@ -102,6 +106,7 @@ def _source_c() -> ArrowTableSource: @function_pod(output_keys="x2") def double_x(x: int) -> int: + """Double ``x``; the function pod downstream of an operator node.""" return x * 2 @@ -150,6 +155,7 @@ class Case: def _make_store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + """A fresh store of the given kind (``in_memory`` or ``delta``).""" if kind == "in_memory": return InMemoryArrowDatabase() return DeltaTableDatabase(base_path=tmp_path / "store") @@ -162,6 +168,7 @@ def _run_job( tmp_path: Path, materialize_results: bool = True, ) -> PipelineJob: + """Build ``case`` in a new job on a fresh store and run it on ``orchestrator``.""" job = PipelineJob( name="itl896", store=_make_store(store_kind, tmp_path / orchestrator) ) @@ -175,6 +182,7 @@ def _run_job( def _op_node(job: PipelineJob) -> OperatorJobNode: + """The job's operator node labelled ``op``.""" node: object = job.nodes["op"] assert isinstance(node, OperatorJobNode) return node @@ -186,6 +194,7 @@ def _table_rows(table: pa.Table) -> list[dict[str, object]]: def _row_hashes(node: OperatorJobNode) -> list[tuple[str, str]]: + """Sorted (tag, data) content hashes of the node's rows from ``iter_data()``.""" return sorted( (tag.content_hash().to_string(), data.content_hash().to_string()) for tag, data in node.iter_data() @@ -198,6 +207,7 @@ def _row_hashes(node: OperatorJobNode) -> list[tuple[str, str]]: def test_operator_node_reads_back_same_rows_after_async_run( case: Case, store_kind: str, materialize_results: bool, tmp_path: Path ) -> None: + """An async run reads back exactly the rows, types and system tags of a sync run.""" sync_node = _op_node( _run_job(case, "sync", store_kind, tmp_path, materialize_results) ) @@ -223,6 +233,7 @@ def test_operator_node_reads_back_same_rows_after_async_run( def test_operator_node_stores_nothing_so_get_all_records_is_none( orchestrator: str, tmp_path: Path ) -> None: + """With the default ``cache_mode=OFF`` nothing is stored, on either orchestrator.""" node = _op_node(_run_job(CASES[0], orchestrator, "in_memory", tmp_path)) assert node.as_table().num_rows == 3 @@ -245,6 +256,7 @@ def test_operator_node_stores_nothing_so_get_all_records_is_none( def _sync_and_async_nodes( case: Case, tmp_path: Path ) -> tuple[OperatorJobNode, OperatorJobNode]: + """The ``op`` node after a sync run and after an async run of ``case``.""" sync_node = _op_node(_run_job(case, "sync", "in_memory", tmp_path)) async_node = _op_node(_run_job(case, "async", "in_memory", tmp_path)) return sync_node, async_node @@ -253,6 +265,7 @@ def _sync_and_async_nodes( def test_operator_node_with_no_rows_reads_back_empty_after_async_run( tmp_path: Path, ) -> None: + """A run with no rows reads back a zero-row table with the sync path's columns.""" sync_node, async_node = _sync_and_async_nodes(NO_MATCH, tmp_path) sync_table = sync_node.as_table() @@ -275,6 +288,7 @@ def test_operator_node_with_no_rows_reads_back_empty_after_async_run( def test_operator_node_with_no_rows_keeps_columns_and_types_after_async_run( tmp_path: Path, ) -> None: + """A zero-row async result should keep the sync path's columns and Arrow types.""" sync_node, async_node = _sync_and_async_nodes(NO_MATCH, tmp_path) assert async_node.as_table(all_info=True).schema == ( @@ -292,6 +306,7 @@ def test_operator_node_with_no_rows_keeps_columns_and_types_after_async_run( def test_merge_join_after_an_operator_reads_back_same_system_tags_after_async_run( tmp_path: Path, ) -> None: + """Async ``MergeJoin`` after an operator should name system tags as sync does.""" sync_node, async_node = _sync_and_async_nodes(MERGE_JOIN_AFTER_OPERATOR, tmp_path) assert async_node.as_table(all_info=True).column_names == ( @@ -313,6 +328,7 @@ def _cleared_table(orchestrator: str, tmp_path: Path) -> tuple[pa.Table, pa.Tabl def test_clear_cache_returns_operator_node_to_not_yet_run(tmp_path: Path) -> None: + """``clear_cache()`` gives a zero-row table on both orchestrators.""" sync_after_run, sync_cleared = _cleared_table("sync", tmp_path / "sync") async_after_run, async_cleared = _cleared_table("async", tmp_path / "async") @@ -333,6 +349,58 @@ def test_clear_cache_returns_operator_node_to_not_yet_run(tmp_path: Path) -> Non def test_clear_cache_keeps_operator_node_arrow_types( orchestrator: str, tmp_path: Path ) -> None: + """The zero-row table after ``clear_cache()`` should keep the run's Arrow types.""" after_run, cleared = _cleared_table(orchestrator, tmp_path) assert cleared.schema == after_run.schema + + +def _rerun_source(ids: list[int], groups: list[str]) -> ArrowTableSource: + """A one-tag-pair source for the rerun tests; ``x`` is ``10 * id``.""" + table = pa.table( + { + "id": pa.array(ids, type=pa.int8()), + "grp": pa.array(groups, type=pa.large_string()), + "x": pa.array([10 * i for i in ids], type=pa.int8()), + } + ) + return ArrowTableSource(table, tag_columns=["id", "grp"], source_id="a") + + +def _table_ids(node: OperatorJobNode) -> list[int]: + """Sorted ``id`` values of the node's table.""" + ids = cast(list[int], node.as_table().column("id").to_pylist()) + return sorted(ids) + + +@pytest.mark.parametrize( + ("second_ids", "second_groups", "expected_ids"), + [ + pytest.param([1, 2], ["g2", "g2"], [], id="second_run_has_no_rows"), + pytest.param([3, 4], ["g1", "g1"], [3, 4], id="second_run_has_other_rows"), + ], +) +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_operator_node_reads_back_only_the_latest_run( + orchestrator: str, + second_ids: list[int], + second_groups: list[str], + expected_ids: list[int], +) -> None: + """A second run replaces the node's rows, even when it produces none.""" + job = PipelineJob(name="itl896_rerun", store=InMemoryArrowDatabase()) + with job: + _ = PolarsFilter(constraints={"grp": "g1"})( + _rerun_source([1, 2], ["g1", "g1"]), label="op" + ) + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert _table_ids(_op_node(job)) == [1, 2] + + job.bind(sources={"a": _rerun_source(second_ids, second_groups)}) + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + + node = _op_node(job) + assert _table_ids(node) == expected_ids + assert sorted(repr(tag.as_dict()["id"]) for tag, _ in node.iter_data()) == [ + repr(i) for i in expected_ids + ] From 7235f8c1c94d7a1712af0d9b58938c48699f88c0 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 05:01:14 -0800 Subject: [PATCH 5/7] docs(operator_node): document REPLAY reads in iter_data and as_table (ITL-896) Address CodeRabbit on PR #302: before a run or after clear_cache(), a REPLAY node with a database reads its stored rows; other modes read nothing. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/nodes/operator_node.py | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 2353f9267..4045027b4 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -1058,10 +1058,12 @@ def run(self) -> None: def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Return an iterator over (tag, data) pairs. - Read-only: never triggers computation. Returns empty before ``run()``, - ``execute()`` or ``async_execute()`` populates the cache, and after - ``clear_cache()``. Call ``node.is_stale`` before iterating if you need - to detect outdated cached data. + Read-only: never triggers computation. Yields the rows of the latest + ``run()``, ``execute()`` or ``async_execute()``. With nothing cached in + memory (before a run, or after ``clear_cache()``), a node in + ``CacheMode.REPLAY`` with a database yields the rows stored there; in + any other mode it yields nothing. Call ``node.is_stale`` before + iterating if you need to detect outdated cached data. """ if self._cached_output_stream is not None: return self._cached_output_stream.iter_data() @@ -1078,9 +1080,11 @@ def as_table( ) -> "pa.Table": """Return the output as a PyArrow table. - Read-only: never triggers computation. Returns a zero-row table - with the node's columns before ``run()``, ``execute()`` or - ``async_execute()`` is called, and after ``clear_cache()``. + Read-only: never triggers computation. Returns the rows of the latest + ``run()``, ``execute()`` or ``async_execute()``. With nothing cached in + memory (before a run, or after ``clear_cache()``), a node in + ``CacheMode.REPLAY`` with a database returns the rows stored there; in + any other mode it returns a zero-row table with the node's columns. """ if self._cached_output_stream is not None: return self._cached_output_stream.as_table( From 91a342cfe02cdf0ab5671f61ec22dbce9644e8e4 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 06:17:48 -0800 Subject: [PATCH 6/7] fix(operator-node): preserve async exceptions and verify v0.3 readback Cover cache modes and both DropNulls forms, retain concurrent errors, and clarify reads of earlier LOG records. Co-Authored-By: GPT-6 --- docs/concepts/function-pods.md | 7 +- src/orcapod/core/nodes/operator_node.py | 26 +-- src/orcapod/pipeline/job.py | 5 +- ...6-11-11-itl-896-async-operator-readback.md | 52 ++++++ .../test_operator_node_async_readback.py | 153 +++++++++++++++++- 5 files changed, 228 insertions(+), 15 deletions(-) diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index e6e8013e6..06a48dec6 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -313,9 +313,10 @@ A node from a job that has run also provides: An operator node (a `Join`, `Batch`, column selection and so on, applied inside the job) is read the same way: after `job.run()` its `as_table()` and `iter_data()` return the rows it produced in the latest run, on both the synchronous and the asynchronous orchestrator. It -keeps that output in memory and, with the default `cache_mode=OFF`, stores nothing, so its -`get_all_records()` returns `None`. Calling `clear_cache()` on it discards the output, and it -then reads as a zero-row table, as before the first run, until the job runs again. +keeps that output in memory and, with the default `cache_mode=OFF`, stores nothing. Its +`get_all_records()` returns `None` when no matching records exist; records from an earlier +LOG run remain readable after switching to OFF. Calling `clear_cache()` on it discards the +output, and it then reads as a zero-row table, as before the first run, until the job runs again. ### When the store cannot be written diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 979fbd179..b8bf9457e 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -1231,7 +1231,8 @@ def get_all_records( Returns the stored output table with column filtering applied per ``ColumnConfig`` conventions. Returns ``None`` when no - database is attached or no records exist. + database is attached or no matching records exist. Switching to OFF + does not remove records stored by an earlier LOG run. """ if self._pipeline_database is None: return None @@ -1384,15 +1385,22 @@ async def forward() -> None: await output.send(item) hashes = [s.pipeline_hash() for s in self._input_streams] - async with asyncio.TaskGroup() as tg: - tg.create_task( - self._operator.async_execute( - inputs, - intermediate.writer, - input_pipeline_hashes=hashes, + operator = cast(StaticOutputOperatorPod, self._operator) + try: + async with asyncio.TaskGroup() as tg: + tg.create_task( + operator.async_execute( + inputs, + intermediate.writer, + input_pipeline_hashes=hashes, + ) ) - ) - tg.create_task(forward()) + tg.create_task(forward()) + except ExceptionGroup as exc: + # Preserve direct operator errors; concurrent failures stay grouped. + if len(exc.exceptions) == 1: + raise exc.exceptions[0] from exc + raise # TaskGroup has completed. Keep the output on the node, as execute() # does, and store it in LOG mode (sync DB write, post-hoc). A run diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 1acc170bc..9fde57aa5 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -796,8 +796,9 @@ def run( ``as_table()`` and ``iter_data()`` in either case, on either orchestrator. An operator node with the default ``cache_mode=OFF`` (the only mode a job sets) keeps its output - in memory and stores nothing, so its ``get_all_records()`` - returns ``None``. + in memory and stores nothing. Its ``get_all_records()`` returns + ``None`` when no matching records exist; records from an earlier + LOG run remain readable after switching to OFF. commit_every_rows: How many rows a function node processes between commits of its results and pipeline records to the store (default 100). Each node also commits when it finishes, before a diff --git a/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md b/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md index 86ab7da95..e715c9e41 100644 --- a/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md +++ b/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md @@ -77,3 +77,55 @@ After `clear_cache()` an operator node returns a zero-row table on both paths to - [ ] ruff check/format, basedpyright exit code, full `uv run pytest tests/`, golden hashes unchanged. - [ ] Merge origin/main, push, PR (Fixes ITL-896), link, CI, CodeRabbit (inline + review bodies). + +## November 12 refresh (Codex takeover) + +- [x] Preserve approved merge 69af724b: pre-forward context validation when LOG has + a database, unconditional row retention, conditional persistence; retain both run + docstring paragraphs. Changelog and OR1 now live in their per-entry files. +- [x] Both fragment checkers pass. Prior worker established 57 readback failures + against main and passing targeted tests after the merge. +- [ ] Add named-column and no-argument DropNulls to CASES (two retained rows). + Compare normal rows and all-info schemas; mark only the known ITL-976 metadata + mismatch and stop for any other difference. +- [ ] Pin OFF, LOG with/without database, and populated REPLAY through public node + run/async_execute and stream reads; assert persisted rows only for LOG/REPLAY. + Keep existing test_stored_operator_node_raises_before_forwarding_rows_async as + the pre-forward LOG refusal regression. +- [ ] Run focused readback/equivalence/stored-context tests and golden tests without + changing fixtures; run full suite sequentially on 3.12, 3.11, 3.13, then ruff, + format and basedpyright (exit zero). +- [ ] Fetch current main before final commit; merge if needed and revalidate. + Update existing PR body, push once, inspect CI and all CodeRabbit review bodies + and GraphQL threads on the final head, route findings through coordinator ask. + +### Approved review follow-up + +Coordinator approved November 12: fix singleton exception wrapping here; defer +data-less stream retention (sync rejects it too); clarify old LOG records remain +readable after switching to OFF. Add an Index fail_on_miss public-execution +regression for no database, OFF and LOG before editing library code. Catch the +TaskGroup's ExceptionGroup and re-raise its sole exception; leave groups with +multiple failures intact. Re-run focused tests and all three full suites after +this change. Report the data-less reproduction in the final worker report. + +### Refresh verification checkpoint + +- Both DropNulls forms pass all eight store/materialization combinations without + a metadata exception. Cache-mode parity passes for all four requested cases. +- Exception regression: 3 failures before the fix, then all pass. A concurrent + operator RuntimeError and output LookupError remain an ExceptionGroup of two. +- Disposable-process test using main's async_execute: no-database exception keeps + its original RuntimeError; OFF/LOG wrap it. Main still loses readback in OFF + and requested LOG without a database (4 failures, 3 passes across new cases). +- Affected readback/channels/stored-context tests: 542 passed, 5 known xfails. +- Focused equivalence and golden tests before the exception-only change: + 1386 passed, 14 known xfails. No golden files differ from main. +- Ruff check and format pass; basedpyright exits 0 (10370 existing warnings). +- Initial full Python 3.12 run: 9438 passed, 101 skipped, 23 xfailed; superseded + by the fresh full run after the exception tests were added. +- Data-less reproduction (deferred by coordinator): one-row ArrowTableSource + with id tag and x data; SelectDataColumns([]) rejects sync with ValueError, + async forwards one row and then raises the same no-data-columns ValueError. +- All previous review bodies/threads audited; older nitpick docstrings already + replied to. Three latest threads will receive approved dispositions after commit. diff --git a/tests/test_pipeline/test_operator_node_async_readback.py b/tests/test_pipeline/test_operator_node_async_readback.py index c3c314053..cee87f120 100644 --- a/tests/test_pipeline/test_operator_node_async_readback.py +++ b/tests/test_pipeline/test_operator_node_async_readback.py @@ -9,6 +9,7 @@ from __future__ import annotations +import asyncio from collections.abc import Callable from dataclasses import dataclass from pathlib import Path @@ -20,8 +21,10 @@ from orcapod.core.operators import ( Batch, DropDataColumns, + DropNulls, DropTagColumns, GroupBy, + Index, Join, MapData, MapTags, @@ -32,14 +35,16 @@ SemiJoin, ) from orcapod import function_pod +from orcapod.channels import Channel from orcapod.core.nodes import OperatorJobNode from orcapod.core.sources import ArrowTableSource from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator from orcapod.pipeline.job import PipelineJob from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator -from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.types import CacheMode ORCHESTRATORS = { "sync": SyncPipelineOrchestrator, @@ -126,6 +131,8 @@ class Case: 3, ), Case("drop_data_columns", lambda a, b, c: DropDataColumns(["s"])(a, label="op"), 3), + Case("drop_nulls_named", lambda a, b, c: DropNulls(["s"])(a, label="op"), 2), + Case("drop_nulls_all", lambda a, b, c: DropNulls()(a, label="op"), 2), Case( "select_tag_columns", lambda a, b, c: SelectTagColumns(["id"])(a, label="op"), @@ -240,6 +247,150 @@ def test_operator_node_stores_nothing_so_get_all_records_is_none( assert node.get_all_records() is None +@pytest.mark.parametrize( + ("cache_mode", "with_database"), + [ + pytest.param(CacheMode.OFF, True, id="off"), + pytest.param(CacheMode.LOG, True, id="log_with_database"), + pytest.param(CacheMode.LOG, False, id="log_without_database"), + pytest.param(CacheMode.REPLAY, True, id="replay"), + ], +) +def test_operator_node_cache_modes_keep_sync_async_readback_equivalent( + cache_mode: CacheMode, with_database: bool +) -> None: + """Cache mode controls persistence; both execution paths retain emitted rows.""" + source = _source_a() + + def make_node() -> OperatorJobNode: + """Build a node, seeding a populated cache for REPLAY.""" + store = InMemoryArrowDatabase() if with_database else None + if cache_mode == CacheMode.REPLAY: + seed = OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[source], + pipeline_database=store, + cache_mode=CacheMode.LOG, + ) + _ = seed.execute(source) + return OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[source], + pipeline_database=store, + cache_mode=cache_mode, + ) + + sync_node, async_node = make_node(), make_node() + sync_rows = sync_node.execute(source) + + async def run() -> list[tuple[TagProtocol, DataProtocol]]: + """Drive the node's public channel interface and collect forwarded rows.""" + inputs: Channel[tuple[TagProtocol, DataProtocol]] = Channel(buffer_size=16) + outputs: Channel[tuple[TagProtocol, DataProtocol]] = Channel(buffer_size=16) + # REPLAY must emit the stored rows even with no upstream rows to compute. + if cache_mode != CacheMode.REPLAY: + for item in source.iter_data(): + await inputs.writer.send(item) + await inputs.writer.close() + await async_node.async_execute([inputs.reader], outputs.writer) + return await outputs.reader.collect() + + async_rows = asyncio.run(run()) + assert len(sync_rows) == len(async_rows) == 3 + sync_table = sync_node.as_table(all_info=True) + async_table = async_node.as_table(all_info=True) + assert async_table.schema == sync_table.schema + assert _table_rows(async_table) == _table_rows(sync_table) + assert _row_hashes(async_node) == _row_hashes(sync_node) + for node in (sync_node, async_node): + records = node.get_all_records() + if with_database and cache_mode in (CacheMode.LOG, CacheMode.REPLAY): + assert records is not None + assert records.num_rows == 3 + else: + assert records is None + + +@pytest.mark.parametrize("cache_mode", [None, CacheMode.OFF, CacheMode.LOG]) +def test_operator_failure_preserves_exception_type_and_message( + cache_mode: CacheMode | None, +) -> None: + """A sole operator error reaches callers unchanged on either execution path.""" + source = ArrowTableSource( + pa.table( + {"x": pa.array([[1]], type=pa.large_list(pa.int8()))}, + schema=pa.schema([pa.field("x", pa.large_list(pa.int8()), nullable=False)]), + ), + source_id="index_failure", + ) + + def make_node() -> OperatorJobNode: + """Create a fresh Index node with the requested persistence mode.""" + return OperatorJobNode( + operator=Index("x", 5, fail_on_miss=True), + input_streams=[source], + pipeline_database=None if cache_mode is None else InMemoryArrowDatabase(), + cache_mode=cache_mode or CacheMode.OFF, + ) + + with pytest.raises(RuntimeError) as sync_error: + _ = make_node().execute(source) + + async def run() -> None: + """Send the source through the node's public async execution interface.""" + inputs: Channel[tuple[TagProtocol, DataProtocol]] = Channel(buffer_size=16) + outputs: Channel[tuple[TagProtocol, DataProtocol]] = Channel(buffer_size=16) + for item in source.iter_data(): + await inputs.writer.send(item) + await inputs.writer.close() + await make_node().async_execute([inputs.reader], outputs.writer) + + with pytest.raises(RuntimeError) as async_error: + asyncio.run(run()) + assert type(async_error.value) is type(sync_error.value) + assert str(async_error.value) == str(sync_error.value) + + +def test_concurrent_operator_and_forwarding_failures_stay_grouped() -> None: + """Unwrapping a sole failure must never discard concurrent channel failures.""" + source = ArrowTableSource( + pa.table( + {"x": pa.array([[1], []], type=pa.large_list(pa.int8()))}, + schema=pa.schema([pa.field("x", pa.large_list(pa.int8()), nullable=False)]), + ), + source_id="concurrent_failure", + ) + node = OperatorJobNode( + operator=Index("x", 0, fail_on_miss=True), input_streams=[source] + ) + + class FailingWriter: + """An output sink that fails when the first valid row is forwarded.""" + + async def send(self, item: tuple[TagProtocol, DataProtocol]) -> None: + """Reject the forwarded row.""" + raise LookupError("output unavailable") + + async def close(self) -> None: + """Allow the node to close its output even after a failure.""" + + async def run() -> None: + """Queue a valid row followed by an invalid row before starting both tasks.""" + inputs: Channel[tuple[TagProtocol, DataProtocol]] = Channel(buffer_size=16) + for item in source.iter_data(): + await inputs.writer.send(item) + await inputs.writer.close() + await node.async_execute([inputs.reader], FailingWriter()) + + with pytest.raises(ExceptionGroup) as info: + asyncio.run(run()) + assert {type(error) for error in info.value.exceptions} == { + RuntimeError, + LookupError, + } + assert len(info.value.exceptions) == 2 + + NO_MATCH = Case( "polars_filter_no_match", lambda a, b, c: PolarsFilter(constraints={"grp": "none"})(a, label="op"), From 42d6d2508743348dad49a519caba4c22781f61ed Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 06:35:27 -0800 Subject: [PATCH 7/7] docs: record ITL-896 refresh validation Co-Authored-By: GPT-6 --- .../2026-11-11-itl-896-async-operator-readback.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md b/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md index e715c9e41..fcaf318f3 100644 --- a/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md +++ b/superpowers/plans/2026-11-11-itl-896-async-operator-readback.md @@ -129,3 +129,16 @@ this change. Report the data-less reproduction in the final worker report. async forwards one row and then raises the same no-data-columns ValueError. - All previous review bodies/threads audited; older nitpick docstrings already replied to. Three latest threads will receive approved dispositions after commit. + +### Final local validation + +- [x] Full suites completed sequentially on Python 3.12, 3.11, 3.13: each + 9442 passed, 101 skipped, 23 xfailed; all exit 0. +- [x] Ruff check/format pass. Basedpyright on Python 3.11 exits 0 with no errors + (10370 existing warnings). Both fragment checkers pass. +- [x] Main remains b46549fd; no golden or CHANGELOG.md differences against main. +- [x] Approved review dispositions implemented, tested and checkpointed in 91a342cf. +- [ ] Push once and verify remote CI, CodeRabbit head coverage and thread replies. + +Audit logs, skip names, review data and the coordinator report are under the +worktree-local gitignored `.untracked/` directory.