diff --git a/.zed/rules b/.zed/rules index 0342f7342..0599d6439 100644 --- a/.zed/rules +++ b/.zed/rules @@ -776,6 +776,16 @@ data. The rules: - FunctionNode Phase 1 returns ALL records in the shared pipeline_path DB table. Phase 2 skips inputs whose hash is already in the DB. - Empty data → ArrowTableSource raises ValueError("Table is empty"). +- A side-effect job node delivers only when it runs (developer, November 13, 2026, ITL-1091): + job.run(), node.run() (the job run's work for that node: with track_completion only + what the invocation log does not record, without it every row again), execute, + async_execute. Every read (iter_data, as_table, dataframes, flow, head, preview, a + downstream node's read) returns the rows its last run in this process passed on, held in + memory (SideEffectJobNode._held_rows), and delivers nothing; with none (no run, a run that + raised, or stale by is_stale) a read returns zero rows. Never route a job node's read to + the standalone delivery path. The standalone SideEffectPodStream still delivers on every + consuming read (iter_data, as_table, dataframes, flow, run); its head() and + preview deliver nothing. - DerivedSource over a job node before run() → presents zero rows with the origin's columns (no computed records; no error), in the plain form and with tags_as_data. - Join requires non-overlapping data columns; raises InputValidationError on collision. diff --git a/CLAUDE.md b/CLAUDE.md index 8164ba5f9..73d048ac6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -818,6 +818,16 @@ data. The rules: DB table (not filtered to current inputs). Phase 2 skips inputs whose hash is already in the DB. - Empty data → `ArrowTableSource` raises `ValueError("Table is empty")`. +- A side-effect job node delivers only when it runs (developer, November 13, 2026, ITL-1091): + `job.run()`, `node.run()` (the job run's work for that node: with `track_completion` only + what the invocation log does not record, without it every row again), `execute`, + `async_execute`. Every read (`iter_data`, `as_table`, dataframes, `flow`, `head`, preview, a + downstream node's read) returns the rows its last run in this process passed on, held in + memory (`SideEffectJobNode._held_rows`), and delivers nothing; with none (no run, a run that + raised, or stale by `is_stale`) a read returns zero rows. Never route a job node's read to + the standalone delivery path. The standalone `SideEffectPodStream` still delivers on every + consuming read (`iter_data`, `as_table`, dataframes, `flow`, `run`); its `head()` and + preview deliver nothing. - `DerivedSource` over a job node before `run()` → presents zero rows with the origin's columns (no computed records; no error), in the plain form and with `tags_as_data`. - Join requires non-overlapping data columns; raises `InputValidationError` on collision. diff --git a/changelog.d/ITL-1091.md b/changelog.d/ITL-1091.md new file mode 100644 index 000000000..b35479e60 --- /dev/null +++ b/changelog.d/ITL-1091.md @@ -0,0 +1,7 @@ +### Breaking changes and migration + +- `run()` on a job's side-effect node no longer delivers every side effect again on every call (ITL-1091). It does the work `job.run()` does for that node: with `track_completion=True` (the default) it delivers only the rows whose delivery the invocation log does not record, and with `track_completion=False` it delivers every row again, as a second `job.run()` does. Code that called `node.run()` or `node.as_table()` on a job's side-effect node to deliver again must call `job.run()` (with `track_completion=False` to repeat deliveries) or use the side-effect pod outside a job, whose stream still delivers on every read. + +### Fixed + +- Reading a side-effect node of a `PipelineJob` after `job.run()` no longer delivers every side effect again (ITL-1091). Before, `iter_data()`, `async_iter_data()`, `as_table()`, the dataframe methods, `flow()` and `run()` of the node, and a downstream node's `run()`, each delivered every row again, without consulting or writing the invocation log, even with `track_completion=True`. The node now holds, in memory, the rows its last run passed on (on both orchestrators; the rows of that run only), and every read returns them and delivers nothing; `head()` and the notebook preview show them. A read with no such rows (no run in this process, the last run raised, or a node upstream changed since) returns no rows and delivers nothing. Rows whose delivery failed under `on_error="log"` are returned as they were passed downstream (left out with `drop_on_failure=True`, kept with `False`). The standalone side-effect stream outside a job still delivers on every read. No hash, stored table or layout changes. diff --git a/design_issues/ITL-1091-SE2.md b/design_issues/ITL-1091-SE2.md new file mode 100644 index 000000000..fbc5ceeb6 --- /dev/null +++ b/design_issues/ITL-1091-SE2.md @@ -0,0 +1,22 @@ +## `src/orcapod/side_effects.py` + +### SE2 — Reading a side-effect job node after `job.run()` delivered every side effect again +**Status:** resolved +**Severity:** high +**Issue:** ITL-1091 + +`SideEffectJobNode` inherited `SideEffectNode.iter_data`, the standalone path: every row was +delivered with no invocation log lookup and no log write. `as_table()`, the dataframe methods, +`flow()`, `async_iter_data()`, `run()` and a downstream node's `run()` went through it, so +reading the node after `job.run()` delivered every side effect again, even with +`track_completion=True`. The node kept nothing after `execute` / `async_execute`; its rows +lived only in the orchestrator's buffers. + +**Fix:** The node holds the rows its last completed run passed on (`_held_rows`, set by +`execute` and `async_execute`, cleared when a run starts, so a run that raises leaves none) +and stamps itself modified. `iter_data` and `head` return them while the node is not stale +(`is_stale`, as `FunctionJobNode`) and nothing otherwise; no read delivers. `run()` calls +`execute(self._input_stream)`: the work `job.run()` does for the node, through the log, and +commits the log before it returns, as a job run does. No +stored table, column or hash changed. The blueprint `SideEffectNode` and the standalone +`SideEffectPodStream` are unchanged. diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 108453b24..0d5e49e01 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -649,6 +649,52 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro This is the contract for now. An opt-in mode that tolerates store failures may be added later. +### Side effects in a job: delivered by a run, never by a read + +A `SideEffectPod` (or `side_effect_pod`, `sink_pod`, `tap_pod`) calls its function for every +row for what the call does outside Orcapod (send an email, write a file) and passes the row +on unchanged. In a job, side effects are delivered only by running: `job.run()`, or +`run()` on the side-effect node. Reading the node never delivers (ITL-1091): `as_table()`, +`iter_data()`, the dataframe methods, `flow()`, `head()`, the notebook preview, and a node +downstream reading its input all return the rows the node's last run passed on, which the +node holds in memory: + +```python +from orcapod.side_effects import SideEffectPod, SideEffectPodConfig + +sent = [] + +def notify(ctx, weight: float, height: float) -> None: + sent.append(weight) # stands for sending a message + +notify_job = PipelineJob(name="notify_demo", database=InMemoryArrowDatabase()) +with notify_job: + mice_source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + SideEffectPod(notify)(mice_source, label="notify") + +notify_job.run() +node = notify_job.nodes["notify"] +print(node.as_table().num_rows, len(list(node.iter_data())), len(sent)) +# 3 3 3 +node.run() # delivers only what the invocation log does not record: nothing here +print(len(sent)) +# 3 +``` + +- With `track_completion=True` (the default), a run delivers only the rows whose delivery the + node's invocation log does not record. With `track_completion=False`, every run, a second + `job.run()` or `node.run()`, delivers every row again; reads still deliver nothing. +- The rows a read returns are those the last run in this process passed on: under + `on_error="log"`, a row whose delivery failed is left out with `drop_on_failure=True` + (the default) and kept with `drop_on_failure=False`, as for the nodes downstream. +- Before any run in this process (a new job over a store that already records deliveries), + after a run that raised, or once a node upstream has changed since the run, a read returns + no rows and `head()` says that `run()` delivers. Run the job to get the rows. +- Outside a job, the stream a side-effect pod returns keeps nothing, so every consuming read + of it (`iter_data()`, `as_table()`, the dataframe methods, `flow()`, `run()`) delivers every + side effect again; its `head()` and notebook preview deliver nothing and show no rows. See + [Streams](streams.md#computing-a-stream-outside-a-job-run). + ### Saving and loading a job `job.save(path)` writes the job to a JSON file: the graph, the store's location and how each diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index 7d0250146..311431c80 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -189,11 +189,11 @@ ten, and its header gives the count. A view holds the default column groups only. For system tags, source info, the data context or content hashes, ask the stream after `run()`. A function pod's or operator's output stream, -a job node, an in-memory source and a `CachedSource` keep the rows `run()` gave: their -`as_table(columns=...)` and `head(columns=...)` read those rows and compute nothing. Other -streams do their `as_table()` work again: a side-effect stream delivers its side effects again -(its `head()` shows no rows), a polling source polls again and a `SpiralDBTableSource` reads -its whole table again. A view never asks its stream for more columns, because for those +a job node (a job's side-effect node included), an in-memory source and a `CachedSource` keep +the rows `run()` gave: their `as_table(columns=...)` and `head(columns=...)` read those rows, +compute nothing and deliver no side effect. Other streams do their `as_table()` work again: a +side-effect stream outside a job delivers its side effects again (its `head()` shows no rows), +a polling source polls again and a `SpiralDBTableSource` reads its whole table again. A view never asks its stream for more columns, because for those streams asking again is not a read. What `run()` does on each kind of object: @@ -206,7 +206,8 @@ What `run()` does on each kind of object: | A `CachedSource` | reads its cache and writes the wrapped source's new rows to it | | A function pod's or operator's output stream | computes what is not computed yet | | A job node | runs the node as a job run would (`as_table()` of a job node only reads), then returns the view | -| A side-effect stream | delivers its side effects for every row, on every call, and returns the rows it passed on; `head()` still shows none | +| A job's side-effect node | runs the node as `job.run()` does: delivers what its invocation log does not record (every row again without `track_completion`), then returns the rows it passed on; every read after it only reads | +| A side-effect stream outside a job | delivers its side effects for every row, on every call, and returns the rows it passed on; `head()` still shows none | | A pipeline blueprint node, an unbound input slot | raises, as `as_table()` does | What each kind of object shows: @@ -217,7 +218,8 @@ What each kind of object shows: | An operator's or function pod's output stream | the rows already computed, or none with status `not_computed` | | A job node | the rows it holds after `job.run()`, else the first of its stored rows, else none (`nothing_stored`) | | A pipeline blueprint node, an unbound input slot | none (`unavailable`) | -| A side-effect stream | none: showing it never delivers its side effects | +| A job's side-effect node | the rows its last run passed on, else none (`not_computed`); it never delivers, and neither do its `as_table()`, `iter_data()` and other reads | +| A side-effect stream outside a job | none: showing it never delivers its side effects | `head(n, columns=..., all_info=...)` takes the same column configuration as `as_table`, sorting included, and applies it to the rows it has before taking the first `n`. The default diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index c426beb65..12782ba7a 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -601,9 +601,10 @@ def run(self) -> StreamView: its ``head``, ``tail`` and ``sort_by_tags`` choose what is shown. For other column groups, use ``as_table(columns=...)`` or ``head(columns=...)`` after ``run()``. On a stream that keeps its rows (a pod's or operator's output stream, - a job node, an in-memory source) they read the rows ``run()`` gave; a stream that - keeps none does its ``as_table()`` work again (a side-effect stream delivers - again, a polling source polls again). + a job node, a job's side-effect node included, an in-memory source) they read the + rows ``run()`` gave and deliver no side effect; a stream that keeps none does its + ``as_table()`` work again (a side-effect stream outside a job delivers again, a + polling source polls again). Returns: A ``StreamView`` of all the rows ``as_table()`` returns, with status diff --git a/src/orcapod/core/streams/view.py b/src/orcapod/core/streams/view.py index 8417e984f..d19412899 100644 --- a/src/orcapod/core/streams/view.py +++ b/src/orcapod/core/streams/view.py @@ -151,9 +151,9 @@ class StreamView: made with: for other column groups (system tags, source info, context, content hashes) ask the stream, ``stream.as_table(columns=...)`` or ``stream.head(columns=...)``. After ``run()`` they read the rows a stream kept (a - pod's or operator's output stream, a job node, an in-memory source); a stream that - keeps none does its ``as_table()`` work again (a side-effect stream delivers again, a - polling source polls again). + pod's or operator's output stream, a job node, a job's side-effect node included, an + in-memory source); a stream that keeps none does its ``as_table()`` work again (a + side-effect stream outside a job delivers again, a polling source polls again). Attributes: table: The rows, with the exact Arrow types the stream gives them. When the diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 33c43cccd..fb9ad3d2e 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -15,11 +15,14 @@ from collections.abc import Callable, Collection, Iterator, Sequence from typing import TYPE_CHECKING, Any, Literal +from typing_extensions import override + from orcapod.core.base import TraceableBase from orcapod.core.nodes.function_node import _build_record_id_preimage from orcapod.core.streams.base import ( StreamBase, check_head_size, + schema_requires_io, ) from orcapod.core.streams.view import ( SIDE_EFFECTS_NOTE, @@ -49,6 +52,7 @@ ) from orcapod.types import ColumnConfig from orcapod.utils import arrow_utils +from orcapod.utils.store_errors import flush_best_effort from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -476,7 +480,10 @@ class SideEffectPod(TraceableBase): ``SideEffectPodStream`` with no invocation logging. In pipeline mode, promoted to ``SideEffectJobNode`` at compile time, - which adds DB-backed invocation logging and completion tracking. + which adds DB-backed invocation logging and completion tracking. A job + delivers only when it runs (``job.run()`` or the node's ``run()``); every + read of the node returns the rows its last run passed on and delivers + nothing (ITL-1091). Args: fn: A callable ``(data, ctx: InvocationContext) -> None``. @@ -846,9 +853,11 @@ def run(self) -> StreamView: """Deliver the side effects for every row and return a ``StreamView`` of the rows. Does exactly what ``as_table()`` does: every call delivers every side effect - again, standalone, with no invocation log, also on a ``SideEffectJobNode`` (a job run logs its deliveries through ``execute``), because the stream keeps no rows. The view holds the rows that call - passed on. ``head()`` and the notebook preview never deliver, so they keep showing - no rows. + again, standalone, with no invocation log, because the node keeps no rows. A + blueprint node's input is an unbound slot, so it raises. ``SideEffectJobNode`` + overrides this: a job's node delivers as ``job.run()`` does and its reads never + deliver. The view holds the rows that call passed on. ``head()`` and the notebook + preview never deliver, so they keep showing no rows. Returns: A ``StreamView`` of the rows this call delivered for. @@ -914,7 +923,10 @@ class SideEffectJobNode(SideEffectNode): as a call-time keyword argument from the orchestrator. Extends ``SideEffectNode`` with DB attachment and orchestrated execution - methods. Analogous to ``FunctionJobNode`` in the function pod hierarchy. + methods. Analogous to ``FunctionJobNode`` in the function pod hierarchy: it + holds the rows its last run passed on (in memory), and every read returns + them without delivering (ITL-1091). Only ``execute``, ``async_execute`` and + ``run()`` deliver. Args: side_effect_pod: The ``SideEffectPod`` this node wraps. @@ -935,6 +947,9 @@ def __init__( self._table_path: tuple[str, ...] | None = None self._unlisted_store_reported = False self._current_log_checked = False + # The rows the last completed run passed on, in the order it passed them; + # ``None`` when no run in this process completed (ITL-1091). + self._held_rows: list[tuple[TagProtocol, DataProtocol]] | None = None def _refuse_log_of_another_data_context(self) -> None: """Refuse an invocation log holding deliveries of another data context. @@ -1029,6 +1044,9 @@ def execute( observer: Optional execution observer (currently unused). run_id: Pipeline run identifier forwarded from the orchestrator. + The node keeps the rows it passed on, which every later read returns without + delivering (ITL-1091). A run that raises leaves the node with no rows. + Returns: List of ``(tag, data)`` tuples — the pass-through rows. @@ -1036,8 +1054,9 @@ def execute( SideEffectLogDataContextError: If the invocation log holds deliveries logged under another data context; nothing is delivered then. """ + self._held_rows = None self._refuse_log_of_another_data_context() - results = [] + results: list[tuple[TagProtocol, DataProtocol]] = [] for tag, data in input_stream.iter_data(): result = _execute_side_effect_row( fn=self._pod._fn, @@ -1055,8 +1074,123 @@ def execute( ) if result is not None: results.append(result) + self._hold(results) return results + def _hold(self, rows: list[tuple[TagProtocol, DataProtocol]]) -> None: + """Keep the rows a completed run passed on and stamp the node as modified. + + Stamped after the upstream nodes, which stamp at the end of their own run, so the + rows are current until an upstream changes again (``is_stale``). + """ + self._held_rows = rows + self._update_modified_time() + + def _current_rows(self) -> list[tuple[TagProtocol, DataProtocol]] | None: + """The rows the last run passed on, or ``None`` if there are none or they are stale. + + Held rows go stale, and are dropped, when an upstream changed after the run: the + judgement ``FunctionJobNode`` uses for the rows it holds. + """ + if self._held_rows is not None and self.is_stale: + self._held_rows = None + return self._held_rows + + # ------------------------------------------------------------------ + # Reads: never deliver + # ------------------------------------------------------------------ + + @override + def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: + """Yield the rows the last run of this node passed on; never delivers. + + Every read of a job node only reads (developer, November 13, 2026, ITL-1091): + ``as_table()``, the dataframe methods, ``flow()`` and ``async_iter_data()`` go + through this method. With no rows from a completed run in this process (the + job has not run, the last run raised, or an upstream changed since), there are + no rows. ``run()`` or ``job.run()`` delivers. + """ + yield from self._current_rows() or () + + @override + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamView: + """Return the first rows the last run passed on; never delivers. + + After a run, the first ``n`` of ``as_table`` with the same column configuration. + Without rows from a run, no rows, with a note that ``run()`` delivers. After a + run that passed on no rows, when the columns asked for are known only by + polling a source, no columns either: ``head`` never polls. See + ``StreamBase.head``. + """ + check_head_size(n) + config = ColumnConfig.handle_config(columns, all_info=all_info) + rows = self._current_rows() + if rows is None: + return self._head_without_rows( + config, ViewStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE + ) + if not rows and schema_requires_io(self, config): + # The table of no rows takes its columns from the schema, which here only a + # poll of the source could give: show the empty content with no columns. + return self._head_without_rows( + config, ViewStatus.ROWS, note="its last run passed on no rows" + ) + return StreamView.of_rows( + self, + self.as_table(columns=config).slice(0, n), + tag_columns=self.keys(columns=config)[0], + total_rows=len(rows), + origin="held", + ) + + @override + def run(self) -> StreamView: + """Run the node as ``job.run()`` runs it and return a ``StreamView`` of its rows. + + Reads the input and delivers through the invocation log, as a job run does: + with ``track_completion`` only the rows the log does not record as delivered are + delivered; without it every row is delivered again, as a second ``job.run()`` + delivers it. With no database attached nothing is logged and every row is + delivered. The node then holds the rows it passed on, which ``as_table()`` and + every other read return without delivering. The invocation log is committed + before ``run()`` returns, as at the end of a job run; after a failed delivery + the log rows of the deliveries that succeeded are committed too, and the + delivery's error is raised. + + Returns: + A ``StreamView`` of the rows this run passed on. + + Raises: + SideEffectLogDataContextError: As for ``execute``. + StoreWriteError: If committing the invocation log fails. + """ + database = self._pipeline_database + try: + _ = self.execute(self._input_stream) + except Exception as run_error: + if database is not None: + flush_best_effort(run_error, database) + raise + if database is not None: + # Committed as at the end of a job run (PipelineJob._finish_run). + try: + database.flush() + except StoreReadError: + # A stored table could not be read: report the read failure itself. + # The run raised, so it leaves the node no rows. + self._held_rows = None + raise + except Exception as exc: + self._held_rows = None + raise StoreWriteError(database.base_path, flush=True) from exc + return self._view_of_rows(self.as_table(), "held") + # ------------------------------------------------------------------ # Async execution # ------------------------------------------------------------------ @@ -1079,6 +1213,10 @@ async def async_execute( Emits non-``None`` results to ``output``. Always closes ``output`` in a ``finally`` block. + The node then holds exactly this run's rows, in the order they were sent + downstream, which every later read returns without delivering (ITL-1091). A run + that raises leaves the node with no rows. + Args: inputs: Single-element sequence containing the input channel. output: Writable channel for pass-through ``(tag, data)`` pairs. @@ -1098,6 +1236,13 @@ async def async_execute( asyncio.Semaphore(max_concurrency) if max_concurrency is not None else None ) + self._held_rows = None + # This run's rows, in the order they were sent downstream. + passed_on: list[tuple[TagProtocol, DataProtocol]] = [] + # Makes recording a row and sending it one step: a send blocked by a full channel + # can be overtaken by a later one, so without it the held order could differ from + # the order downstream received. Never held across the delivery itself. + send_lock = asyncio.Lock() try: self._refuse_log_of_another_data_context() @@ -1118,7 +1263,9 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: table_path=self._table_path, ) if result is not None: - await output.send(result) + async with send_lock: + passed_on.append(result) + await output.send(result) finally: if sem is not None: sem.release() @@ -1128,5 +1275,8 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: if sem is not None: await sem.acquire() tg.create_task(process_one(tag, data)) + # Held and stamped before the output closes, so every downstream node, which + # stamps after it has seen that close, is newer than this one (ITL-1042). + self._hold(passed_on) finally: await output.close() diff --git a/superpowers/plans/2026-11-13-itl-1091-side-effect-job-node-reads.md b/superpowers/plans/2026-11-13-itl-1091-side-effect-job-node-reads.md new file mode 100644 index 000000000..72ad9005a --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1091-side-effect-job-node-reads.md @@ -0,0 +1,97 @@ +# ITL-1091: a read of a side-effect job node delivers nothing — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan +> task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** After `job.run()`, every read of a `SideEffectJobNode` (`iter_data`, `async_iter_data`, +`as_table`, the dataframe methods, `flow`, `head`, the preview, and a downstream node's read) +returns the rows the run passed through and delivers nothing; `node.run()` does the work +`job.run()` does for that node. + +**Architecture:** `SideEffectJobNode` keeps the rows `execute` / `async_execute` passed on +(`_held_rows`, in memory, as `FunctionJobNode` keeps `_cached_output_datas`) and stamps itself +modified at the end of a run. Its `iter_data` yields those rows while it is not stale +(`StreamBase.is_stale`, the judgement function job nodes use) and nothing otherwise. `run()` +calls `execute(self._input_stream)` (log lookup and write as a job run does). `head()` shows the +held rows. The blueprint `SideEffectNode` and the standalone `SideEffectPodStream` are unchanged. + +**Tech Stack:** Python, PyArrow, pytest, uv. + +## Decisions (coordinator, November 13, 2026, developer option 1) + +- (a) Rows held in memory by the run. Not rebuilt from the log (tdb_v2 holds no rows). +- (b1) No held rows (no run in this process, held rows stale, or the last run raised): a read + delivers nothing and returns zero rows with the node's columns; `head()` stays + `NOT_COMPUTED` with `SIDE_EFFECTS_NOTE`. +- (c) `node.run()` = `job.run()`'s work for this node: with `track_completion=True` it delivers + only rows the log does not record; with `False` it delivers every row again. Docstring says so. +- (d) Held rows = `execute`'s return list (today's row set under `on_error="log"` and + `drop_on_failure`); cleared at the start of a run, so a run that raises leaves none. +- (e) `head()`/preview after a run show the held rows (status `ROWS`, origin `held`). +- Conditions: stale test (upstream change after a run → zero rows, no delivery); downstream + function node `down.run()` / `down.as_table()` → 0 new deliveries; async held rows are exactly + this run's rows (ITL-1093 is fixing function/operator nodes after an async run); loaded-job + finding (side-effect nodes vanish from `PipelineJob.load`) goes to worker_done only. +- No change to `_table_of_rows` (PR #375 touches it); no ordering dependency. + +## Diagnosis (b4fde330), cumulative deliveries over 6 rows, same on both orchestrators + +| track_completion | job.run | head | preview | iter_data | async_iter_data | as_table | as_polars_df | as_pandas_df | flow | node.run | job.run #2 | +|---|---|---|---|---|---|---|---|---|---|---|---| +| False | 6 | 6 | 6 | 12 | 18 | 24 | 30 | 36 | 42 | 48 | 54 | +| True | 6 | 6 | 6 | 12 | 18 | 24 | 30 | 36 | 42 | 48 | 48 | + +Downstream `down.run()` after `job.run()` delivered 6 more (track=True). + +## Files + +- Modify: `src/orcapod/side_effects.py` (`SideEffectJobNode`; `SideEffectNode.run` docstring) +- Modify: `src/orcapod/core/streams/base.py` (`StreamBase.run` docstring) +- Modify: `src/orcapod/core/streams/view.py` (`StreamView` docstring) +- Create: `tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py` +- Modify: `tests/test_core/streams/test_run.py` (job-node case of the "delivers as as_table" test) +- Modify: `docs/concepts/streams.md`, `docs/concepts/function-pods.md` +- Modify: `CLAUDE.md`, `.zed/rules` +- Create: `changelog.d/ITL-1091.md`, `design_issues/ITL-1091-SE1.md` + +### Task 1: failing tests + +- [ ] Write `tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py`: public + interface only (`PipelineJob`, `SideEffectPod`, `job.nodes[...]`). Cases: + every read after `job.run()` delivers nothing (param: orchestrator sync/async x + track_completion True/False), returned rows equal the source rows (pass-through) and the + downstream node's input tags; `node.run()` delivers 0 (True) / 6 (False); second `job.run()` + unchanged; head after run shows rows; read before any run (fresh job on populated DB) → 0 + rows, 0 deliveries, head NOT_COMPUTED; on_error="log" drop True/False row sets; on_error raise + → read returns 0 rows; stale after an upstream function node re-runs → 0 rows, 0 deliveries; + downstream `down.run()` / `down.as_table()` → 0 deliveries; async run twice → held rows are 6. +- [ ] Run (py3.12) and see them fail. + +### Task 2: implementation in `SideEffectJobNode` + +- [ ] `_held_rows: list[(tag, data)] | None` in `__init__`. +- [ ] `execute`: clear at start, set to results at the end, `_update_modified_time()`. +- [ ] `async_execute`: clear at start, collect each row sent, set and stamp after the + TaskGroup and before `output.close()` (ITL-1042 ordering). +- [ ] `_current_rows()`: drop held rows when `is_stale`; return them or `None`. +- [ ] `iter_data`: yield `_current_rows() or ()`; never delivers. +- [ ] `head`: held rows → `StreamView.of_rows(... as_table(columns=config).slice(0, n) ..., + origin="held")`; else rowless NOT_COMPUTED + `SIDE_EFFECTS_NOTE`. +- [ ] `run`: `execute(self._input_stream)` then `_view_of_rows(self.as_table(), "held")`. +- [ ] Run tests (3.12 and 3.11 for async), plus `tests/test_core/side_effect_pod`, + `tests/test_core/streams`, side-effect pipeline tests. + +### Task 3: docs, CLAUDE.md/.zed/rules, changelog fragment, design issue entry + +### Task 4: golden suite unchanged; ruff, basedpyright; full suite once before PR (CI runs it) + +### Task 5: draft PR, CI, CodeRabbit rounds, worker_done + +## Progress notes + +- 2026-11-13: diagnosis sent, design approved by coordinator. +- 2026-11-13: implemented (Tasks 1-3 done); draft PR #377 opened (head 1e3d7904); + waiting for CI, then ready-for-review, CodeRabbit rounds, worker_done. +- 2026-11-13: CodeRabbit round 1 (1e3d7904, 922fede6): 4 inline accepted (docs wording, + node.run() flushes the log, send lock in async_execute, head() guard for an empty run whose + schema needs a poll), 1 body style hint declined. Plan approved by coordinator. diff --git a/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py b/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py new file mode 100644 index 000000000..47e803a67 --- /dev/null +++ b/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py @@ -0,0 +1,414 @@ +"""A read of a side-effect job node delivers nothing (ITL-1091). + +After ``job.run()``, every read of a side-effect job node (``iter_data``, ``async_iter_data``, +``as_table``, the dataframe methods, ``flow``, ``head`` and the notebook preview, and a +downstream node's read) returns the rows the run passed through and delivers nothing, on both +orchestrators and with either ``track_completion``. ``node.run()`` does the work ``job.run()`` +does for the node. A node with no rows from a run in this process returns no rows and +delivers nothing. +""" + +from __future__ import annotations + +import asyncio +import logging +from collections.abc import Callable, Collection +from pathlib import Path +from typing import Any, Literal + +import pyarrow as pa +import pytest +from typing_extensions import override + +from orcapod import function_pod +from orcapod.core.sources import DictSource +from orcapod.core.streams import ArrowTableStream +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.errors import StoreWriteError +from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator +from orcapod.pipeline.job import PipelineJob +from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.side_effects import SideEffectJobNode, SideEffectPod, SideEffectPodConfig +from orcapod.streams import ViewStatus +from orcapod.types import ColumnConfig, Schema + +ROWS = 6 +SORTED = {"sort_by_tags": True} + +Orchestrator = Literal["sync", "async"] + + +def _source() -> DictSource: + """Six rows: tag ``k``, data ``x``.""" + return DictSource( + [{"k": i, "x": i} for i in range(ROWS)], tag_columns=["k"], source_id="s" + ) + + +def _job( + *, + track_completion: bool = True, + on_error: Literal["raise", "log"] = "raise", + drop_on_failure: bool = True, + failing: Collection[int] = (), + upstream: bool = False, + downstream: bool = False, + database: ArrowDatabaseProtocol | None = None, +) -> tuple[PipelineJob, list[int]]: + """A job with a side-effect node ``delivered``, and the list of its deliveries. + + Args: + track_completion: As for ``SideEffectPodConfig``. + on_error: As for ``SideEffectPodConfig``. + drop_on_failure: As for ``SideEffectPodConfig``. + failing: Values of ``x`` whose delivery raises. + upstream: Put a function node ``up`` (``x`` passed on unchanged) before the + side-effect node. + downstream: Put a function node ``down`` (``y = x + 100``) after it. + database: The job's database; a new in-memory one by default. + """ + sent: list[int] = [] + + def deliver(ctx: Any, x: int) -> None: + """The side effect: record ``x``, or fail for the values in ``failing``.""" + if x in failing: + raise RuntimeError(f"delivery of {x} failed") + sent.append(x) + + @function_pod(output_keys="x") + def same(x: int) -> int: + """Pass ``x`` on.""" + return x + + @function_pod(output_keys="y") + def plus(x: int) -> int: + """``x + 100``.""" + return x + 100 + + job = PipelineJob( + name="se", + database=database if database is not None else InMemoryArrowDatabase(), + ) + pod = SideEffectPod( + deliver, + config=SideEffectPodConfig( + track_completion=track_completion, + on_error=on_error, + drop_on_failure=drop_on_failure, + ), + ) + with job: + stream: StreamProtocol = _source() + if upstream: + stream = same.pod(stream, label="up") + delivered = pod(stream, label="delivered") + if downstream: + _ = plus.pod(delivered, label="down") + return job, sent + + +def _run(job: PipelineJob, orchestrator: Orchestrator) -> None: + """Run ``job`` on the named orchestrator.""" + if orchestrator == "async": + _ = job.run(orchestrator=AsyncPipelineOrchestrator()) + else: + _ = job.run() + + +def _reads(node: Any) -> dict[str, Callable[[], Any]]: + """Every public read of ``node``, by name.""" + + async def collect() -> list[Any]: + return [row async for row in node.async_iter_data()] + + return { + "head": lambda: node.head(100), + "preview": node._repr_html_, + "iter_data": lambda: list(node.iter_data()), + "__iter__": lambda: list(node), + "async_iter_data": lambda: asyncio.run(collect()), + "as_table": node.as_table, + "as_polars_df": node.as_polars_df, + "as_df": node.as_df, + "as_lazy_frame": lambda: node.as_lazy_frame().collect(), + "as_pandas_df": node.as_pandas_df, + "flow": node.flow, + } + + +def _user_table(table: pa.Table) -> pa.Table: + """``table`` with its user columns only (``k``, ``x``).""" + return table.select(["k", "x"]) + + +# --------------------------------------------------------------------------- +# Every read after a run delivers nothing and returns the rows the run passed through +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +@pytest.mark.parametrize("track_completion", [True, False]) +def test_every_read_after_a_run_delivers_nothing( + orchestrator: Orchestrator, track_completion: bool +) -> None: + """Each read returns the run's rows; the deliveries stay those of the run.""" + job, sent = _job(track_completion=track_completion) + _run(job, orchestrator) + assert len(sent) == ROWS + node = job.nodes["delivered"] + for name, read in _reads(node).items(): + _ = read() + assert len(sent) == ROWS, f"{name} delivered again" + assert len(list(node.iter_data())) == ROWS + assert _user_table(node.as_table(columns=SORTED)).equals( + _user_table(_source().as_table(columns=SORTED)) + ) + head = node.head(100) + assert head.status is ViewStatus.ROWS + assert len(head) == ROWS + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_node_run_delivers_what_the_log_does_not_record( + orchestrator: Orchestrator, +) -> None: + """With ``track_completion``, ``node.run()`` after a run delivers nothing.""" + job, sent = _job(track_completion=True) + _run(job, orchestrator) + view = job.nodes["delivered"].run() + assert len(sent) == ROWS + assert len(view) == ROWS + _run(job, orchestrator) + assert len(sent) == ROWS + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_node_run_without_track_completion_delivers_every_row_as_a_job_run_does( + orchestrator: Orchestrator, +) -> None: + """Without ``track_completion``, ``node.run()`` delivers every row again, as ``job.run()`` does.""" + job, sent = _job(track_completion=False) + _run(job, orchestrator) + view = job.nodes["delivered"].run() + assert len(sent) == 2 * ROWS + assert len(view) == ROWS + _run(job, orchestrator) + assert len(sent) == 3 * ROWS + # The reads after either kind of run deliver nothing. + for name, read in _reads(job.nodes["delivered"]).items(): + _ = read() + assert len(sent) == 3 * ROWS, f"{name} delivered again" + + +def test_node_run_on_a_job_never_run_delivers_and_logs_as_a_job_run() -> None: + """``node.run()`` before any ``job.run()`` delivers once; the job run then delivers nothing.""" + job, sent = _job(track_completion=True) + node = job.nodes["delivered"] + view = node.run() + assert len(sent) == ROWS + assert len(view) == ROWS + assert len(node.as_table()) == ROWS + _ = job.run() + assert len(sent) == ROWS + + +@pytest.mark.parametrize("track_completion", [True, False]) +def test_an_async_run_holds_exactly_its_own_rows(track_completion: bool) -> None: + """After two async runs, the node holds the rows of the last run only.""" + job, sent = _job(track_completion=track_completion) + _run(job, "async") + _run(job, "async") + node = job.nodes["delivered"] + assert len(node.as_table()) == ROWS + assert len(list(node.iter_data())) == ROWS + assert len(sent) == (ROWS if track_completion else 2 * ROWS) + + +# --------------------------------------------------------------------------- +# The rows read equal the rows the run passed downstream +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +@pytest.mark.parametrize("track_completion", [True, False]) +def test_downstream_reads_deliver_nothing( + orchestrator: Orchestrator, track_completion: bool +) -> None: + """A function node after the side-effect node reads it without delivering.""" + job, sent = _job(track_completion=track_completion, downstream=True) + _run(job, orchestrator) + down = job.nodes["down"] + _ = down.as_table() + _ = down.run() + assert len(sent) == ROWS + assert sorted(down.as_table()["k"].to_pylist()) == sorted( + job.nodes["delivered"].as_table()["k"].to_pylist() + ) + assert len(sent) == ROWS + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +@pytest.mark.parametrize("drop_on_failure", [True, False]) +def test_rows_whose_delivery_failed_keep_todays_row_set( + orchestrator: Orchestrator, drop_on_failure: bool, caplog: pytest.LogCaptureFixture +) -> None: + """Under ``on_error="log"`` a read returns the rows the run passed downstream.""" + caplog.set_level(logging.CRITICAL) + job, sent = _job( + on_error="log", + drop_on_failure=drop_on_failure, + failing={2, 3}, + downstream=True, + ) + _run(job, orchestrator) + assert sorted(sent) == [0, 1, 4, 5] + node = job.nodes["delivered"] + passed = sorted(node.as_table()["k"].to_pylist()) + assert passed == ([0, 1, 4, 5] if drop_on_failure else list(range(ROWS))) + assert sorted(job.nodes["down"].as_table()["k"].to_pylist()) == passed + assert sorted(sent) == [0, 1, 4, 5] + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_a_run_that_raises_leaves_no_rows_to_read(orchestrator: Orchestrator) -> None: + """After a run that raised, a read returns no rows and delivers nothing.""" + job, sent = _job(failing={3}) + with pytest.raises(BaseException): # noqa: B017 # sync raises RuntimeError, async an ExceptionGroup + _run(job, orchestrator) + count = len(sent) + node = job.nodes["delivered"] + for name, read in _reads(node).items(): + _ = read() + assert len(sent) == count, f"{name} delivered again" + assert len(node.as_table()) == 0 + assert node.head().status is ViewStatus.NOT_COMPUTED + + +# --------------------------------------------------------------------------- +# No rows from a run: nothing is delivered and no rows are returned +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("track_completion", [True, False]) +def test_a_read_before_any_run_delivers_nothing(track_completion: bool) -> None: + """A fresh job over a store with deliveries recorded returns no rows and delivers nothing.""" + database = InMemoryArrowDatabase() + first, first_sent = _job(track_completion=track_completion, database=database) + _ = first.run() + assert len(first_sent) == ROWS + job, sent = _job(track_completion=track_completion, database=database) + node = job.nodes["delivered"] + for name, read in _reads(node).items(): + _ = read() + assert sent == [], f"{name} delivered" + table = node.as_table() + assert table.num_rows == 0 + assert table.column_names == ["k", "x"] + head = node.head() + assert head.status is ViewStatus.NOT_COMPUTED + assert "run() delivers its side effects" in head.header() + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_rows_held_from_a_run_are_dropped_when_an_upstream_changes( + orchestrator: Orchestrator, +) -> None: + """An upstream node that ran since the run makes the held rows out of date: no rows.""" + job, sent = _job(upstream=True) + _run(job, orchestrator) + node = job.nodes["delivered"] + assert len(node.as_table()) == ROWS + _ = job.nodes["up"].run() + for name, read in _reads(node).items(): + _ = read() + assert len(sent) == ROWS, f"{name} delivered again" + assert len(node.as_table()) == 0 + assert node.head().status is ViewStatus.NOT_COMPUTED + + +# --------------------------------------------------------------------------- +# node.run() commits the invocation log; head() after an empty run never polls +# --------------------------------------------------------------------------- + + +def test_node_run_commits_the_invocation_log(tmp_path: Path) -> None: + """The deliveries ``node.run()`` logged are found by a new job over the same store.""" + job, sent = _job(database=DeltaTableDatabase(tmp_path / "store")) + _ = job.nodes["delivered"].run() + assert len(sent) == ROWS + # A new database object reads only what was committed to the store. + again, again_sent = _job(database=DeltaTableDatabase(tmp_path / "store")) + _ = again.run() + assert again_sent == [] + + +class _SchemaOnlyByPolling(ArrowTableStream): + """A stream of no rows whose schema, once ``armed``, could be read only by polling.""" + + armed: bool = False + + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Say that reading the schema would poll, once armed.""" + return self.armed + + @override + def output_schema( + self, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> tuple[Schema, Schema]: + """Refuse once armed: reading the schema would poll.""" + if self.armed: + raise AssertionError("the schema was read: a poll") + return super().output_schema(columns=columns, all_info=all_info) + + +def test_head_after_a_run_that_passed_on_no_rows_never_polls() -> None: + """With no rows held and columns known only by polling, ``head()`` reads no schema.""" + empty = pa.table( + {"k": pa.array([], type=pa.int64()), "x": pa.array([], type=pa.int64())} + ) + upstream = _SchemaOnlyByPolling(empty, tag_columns=["k"]) + sent: list[int] = [] + + def deliver(ctx: Any, x: int) -> None: + """The side effect: record ``x``.""" + sent.append(x) + + node = SideEffectJobNode( + side_effect_pod=SideEffectPod(deliver), input_stream=upstream + ) + assert len(node.run()) == 0 + upstream.armed = True + head = node.head(all_info=True) + _ = node._repr_html_() # the notebook preview: the public display hook + assert head.status is ViewStatus.ROWS + assert len(head) == 0 + assert "its last run passed on no rows" in head.header() + assert sent == [] + + +class _FailingFlushDatabase(InMemoryArrowDatabase): + """An in-memory database whose commit fails.""" + + @override + def flush(self) -> None: + """Fail, as a store that cannot be written does.""" + raise OSError("the store cannot be written") + + +def test_node_run_whose_log_commit_fails_leaves_no_rows() -> None: + """A ``node.run()`` that raises at its commit leaves the node no rows to read.""" + job, sent = _job() + node = job.nodes["delivered"] + # The job hands its nodes scoped views of its database; attach the failing one + # to the node itself. + node.attach_databases(pipeline_database=_FailingFlushDatabase()) + with pytest.raises(StoreWriteError): + _ = node.run() + assert len(sent) == ROWS + assert list(node.iter_data()) == [] + assert node.head().status is ViewStatus.NOT_COMPUTED + assert len(sent) == ROWS diff --git a/tests/test_core/streams/test_run.py b/tests/test_core/streams/test_run.py index 54da7aecc..5f3796352 100644 --- a/tests/test_core/streams/test_run.py +++ b/tests/test_core/streams/test_run.py @@ -4,8 +4,10 @@ and there is one uniform way to compute any stream: ``run()``. It takes no arguments, does the work ``as_table()`` does (through the same path and cache) and returns a ``StreamView`` of all the computed rows. Job nodes keep their own ``run()`` (it computes, where their -``as_table()`` only reads) and return the view too. A side-effect stream keeps no rows: each -``run()`` delivers again, as each ``as_table()`` does, and its ``head()`` stays rowless. +``as_table()`` only reads) and return the view too. A side-effect stream outside a job keeps +no rows: each ``run()`` delivers again, as each ``as_table()`` does, and its ``head()`` stays +rowless. A job's side-effect node keeps the rows its run passed on, and its reads deliver +nothing (ITL-1091). """ from __future__ import annotations @@ -471,14 +473,31 @@ def test_side_effect_stream_run_delivers_each_time_and_head_stays_rowless() -> N assert len(delivered) == 2 * ROWS -@pytest.mark.parametrize("node_class", [SideEffectNode, SideEffectJobNode]) -def test_side_effect_nodes_run_delivers_as_as_table_does(node_class: type) -> None: - """A side-effect node's ``run()`` delivers standalone, as its ``as_table()`` does.""" +def test_side_effect_node_run_delivers_as_as_table_does() -> None: + """A side-effect node outside a job delivers on ``run()``, as on ``as_table()``.""" delivered, deliver = _deliveries() - node = node_class(side_effect_pod=SideEffectPod(deliver), input_stream=_source()) + node = SideEffectNode( + side_effect_pod=SideEffectPod(deliver), input_stream=_source() + ) + view = node.run() + assert len(delivered) == ROWS + assert len(view) == ROWS + _ = node.as_table() + assert len(delivered) == 2 * ROWS + + +def test_side_effect_job_node_run_delivers_and_its_reads_do_not() -> None: + """A side-effect job node's ``run()`` delivers; its ``as_table()`` reads what it passed on (ITL-1091).""" + delivered, deliver = _deliveries() + node = SideEffectJobNode( + side_effect_pod=SideEffectPod(deliver), input_stream=_source() + ) view = node.run() assert len(delivered) == ROWS assert len(view) == ROWS + assert len(node.as_table()) == ROWS + assert node.head().status is ViewStatus.ROWS + assert len(delivered) == ROWS # ---------------------------------------------------------------------------