Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -776,6 +776,16 @@ data. The rules:
- FunctionNode Phase 1 returns ALL records in the shared pipeline_path DB table.
Phase 2 skips inputs whose hash is already in the DB.
- Empty data → ArrowTableSource raises ValueError("Table is empty").
- A side-effect job node delivers only when it runs (developer, November 13, 2026, ITL-1091):
job.run(), node.run() (the job run's work for that node: with track_completion only
what the invocation log does not record, without it every row again), execute,
async_execute. Every read (iter_data, as_table, dataframes, flow, head, preview, a
downstream node's read) returns the rows its last run in this process passed on, held in
memory (SideEffectJobNode._held_rows), and delivers nothing; with none (no run, a run that
raised, or stale by is_stale) a read returns zero rows. Never route a job node's read to
the standalone delivery path. The standalone SideEffectPodStream still delivers on every
consuming read (iter_data, as_table, dataframes, flow, run); its head() and
preview deliver nothing.
- DerivedSource over a job node before run() → presents zero rows with the origin's columns
(no computed records; no error), in the plain form and with tags_as_data.
- Join requires non-overlapping data columns; raises InputValidationError on collision.
Expand Down
10 changes: 10 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -818,6 +818,16 @@ data. The rules:
DB table (not filtered to current inputs). Phase 2 skips inputs whose hash is already
in the DB.
- Empty data → `ArrowTableSource` raises `ValueError("Table is empty")`.
- A side-effect job node delivers only when it runs (developer, November 13, 2026, ITL-1091):
`job.run()`, `node.run()` (the job run's work for that node: with `track_completion` only
what the invocation log does not record, without it every row again), `execute`,
`async_execute`. Every read (`iter_data`, `as_table`, dataframes, `flow`, `head`, preview, a
downstream node's read) returns the rows its last run in this process passed on, held in
memory (`SideEffectJobNode._held_rows`), and delivers nothing; with none (no run, a run that
raised, or stale by `is_stale`) a read returns zero rows. Never route a job node's read to
the standalone delivery path. The standalone `SideEffectPodStream` still delivers on every
consuming read (`iter_data`, `as_table`, dataframes, `flow`, `run`); its `head()` and
preview deliver nothing.
- `DerivedSource` over a job node before `run()` → presents zero rows with the origin's columns
(no computed records; no error), in the plain form and with `tags_as_data`.
- Join requires non-overlapping data columns; raises `InputValidationError` on collision.
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/ITL-1091.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
### Breaking changes and migration

- `run()` on a job's side-effect node no longer delivers every side effect again on every call (ITL-1091). It does the work `job.run()` does for that node: with `track_completion=True` (the default) it delivers only the rows whose delivery the invocation log does not record, and with `track_completion=False` it delivers every row again, as a second `job.run()` does. Code that called `node.run()` or `node.as_table()` on a job's side-effect node to deliver again must call `job.run()` (with `track_completion=False` to repeat deliveries) or use the side-effect pod outside a job, whose stream still delivers on every read.

### Fixed

- Reading a side-effect node of a `PipelineJob` after `job.run()` no longer delivers every side effect again (ITL-1091). Before, `iter_data()`, `async_iter_data()`, `as_table()`, the dataframe methods, `flow()` and `run()` of the node, and a downstream node's `run()`, each delivered every row again, without consulting or writing the invocation log, even with `track_completion=True`. The node now holds, in memory, the rows its last run passed on (on both orchestrators; the rows of that run only), and every read returns them and delivers nothing; `head()` and the notebook preview show them. A read with no such rows (no run in this process, the last run raised, or a node upstream changed since) returns no rows and delivers nothing. Rows whose delivery failed under `on_error="log"` are returned as they were passed downstream (left out with `drop_on_failure=True`, kept with `False`). The standalone side-effect stream outside a job still delivers on every read. No hash, stored table or layout changes.
22 changes: 22 additions & 0 deletions design_issues/ITL-1091-SE2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
## `src/orcapod/side_effects.py`

### SE2 — Reading a side-effect job node after `job.run()` delivered every side effect again
**Status:** resolved
**Severity:** high
**Issue:** ITL-1091

`SideEffectJobNode` inherited `SideEffectNode.iter_data`, the standalone path: every row was
delivered with no invocation log lookup and no log write. `as_table()`, the dataframe methods,
`flow()`, `async_iter_data()`, `run()` and a downstream node's `run()` went through it, so
reading the node after `job.run()` delivered every side effect again, even with
`track_completion=True`. The node kept nothing after `execute` / `async_execute`; its rows
lived only in the orchestrator's buffers.

**Fix:** The node holds the rows its last completed run passed on (`_held_rows`, set by
`execute` and `async_execute`, cleared when a run starts, so a run that raises leaves none)
and stamps itself modified. `iter_data` and `head` return them while the node is not stale
(`is_stale`, as `FunctionJobNode`) and nothing otherwise; no read delivers. `run()` calls
`execute(self._input_stream)`: the work `job.run()` does for the node, through the log, and
commits the log before it returns, as a job run does. No
stored table, column or hash changed. The blueprint `SideEffectNode` and the standalone
`SideEffectPodStream` are unchanged.
46 changes: 46 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -649,6 +649,52 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro
This is the contract for now. An opt-in mode that tolerates store failures may be added
later.

### Side effects in a job: delivered by a run, never by a read

A `SideEffectPod` (or `side_effect_pod`, `sink_pod`, `tap_pod`) calls its function for every
row for what the call does outside Orcapod (send an email, write a file) and passes the row
on unchanged. In a job, side effects are delivered only by running: `job.run()`, or
`run()` on the side-effect node. Reading the node never delivers (ITL-1091): `as_table()`,
`iter_data()`, the dataframe methods, `flow()`, `head()`, the notebook preview, and a node
downstream reading its input all return the rows the node's last run passed on, which the
node holds in memory:

```python
from orcapod.side_effects import SideEffectPod, SideEffectPodConfig

sent = []

def notify(ctx, weight: float, height: float) -> None:
sent.append(weight) # stands for sending a message

notify_job = PipelineJob(name="notify_demo", database=InMemoryArrowDatabase())
with notify_job:
mice_source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice")
SideEffectPod(notify)(mice_source, label="notify")

notify_job.run()
node = notify_job.nodes["notify"]
print(node.as_table().num_rows, len(list(node.iter_data())), len(sent))
# 3 3 3
node.run() # delivers only what the invocation log does not record: nothing here
print(len(sent))
# 3
```

- With `track_completion=True` (the default), a run delivers only the rows whose delivery the
node's invocation log does not record. With `track_completion=False`, every run, a second
`job.run()` or `node.run()`, delivers every row again; reads still deliver nothing.
- The rows a read returns are those the last run in this process passed on: under
`on_error="log"`, a row whose delivery failed is left out with `drop_on_failure=True`
(the default) and kept with `drop_on_failure=False`, as for the nodes downstream.
- Before any run in this process (a new job over a store that already records deliveries),
after a run that raised, or once a node upstream has changed since the run, a read returns
no rows and `head()` says that `run()` delivers. Run the job to get the rows.
- Outside a job, the stream a side-effect pod returns keeps nothing, so every consuming read
of it (`iter_data()`, `as_table()`, the dataframe methods, `flow()`, `run()`) delivers every
side effect again; its `head()` and notebook preview deliver nothing and show no rows. See
[Streams](streams.md#computing-a-stream-outside-a-job-run).

### Saving and loading a job

`job.save(path)` writes the job to a JSON file: the graph, the store's location and how each
Expand Down
16 changes: 9 additions & 7 deletions docs/concepts/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,11 +189,11 @@ ten, and its header gives the count.

A view holds the default column groups only. For system tags, source info, the data context
or content hashes, ask the stream after `run()`. A function pod's or operator's output stream,
a job node, an in-memory source and a `CachedSource` keep the rows `run()` gave: their
`as_table(columns=...)` and `head(columns=...)` read those rows and compute nothing. Other
streams do their `as_table()` work again: a side-effect stream delivers its side effects again
(its `head()` shows no rows), a polling source polls again and a `SpiralDBTableSource` reads
its whole table again. A view never asks its stream for more columns, because for those
a job node (a job's side-effect node included), an in-memory source and a `CachedSource` keep
the rows `run()` gave: their `as_table(columns=...)` and `head(columns=...)` read those rows,
compute nothing and deliver no side effect. Other streams do their `as_table()` work again: a
side-effect stream outside a job delivers its side effects again (its `head()` shows no rows),
a polling source polls again and a `SpiralDBTableSource` reads its whole table again. A view never asks its stream for more columns, because for those
streams asking again is not a read.

What `run()` does on each kind of object:
Expand All @@ -206,7 +206,8 @@ What `run()` does on each kind of object:
| A `CachedSource` | reads its cache and writes the wrapped source's new rows to it |
| A function pod's or operator's output stream | computes what is not computed yet |
| A job node | runs the node as a job run would (`as_table()` of a job node only reads), then returns the view |
| A side-effect stream | delivers its side effects for every row, on every call, and returns the rows it passed on; `head()` still shows none |
| A job's side-effect node | runs the node as `job.run()` does: delivers what its invocation log does not record (every row again without `track_completion`), then returns the rows it passed on; every read after it only reads |
| A side-effect stream outside a job | delivers its side effects for every row, on every call, and returns the rows it passed on; `head()` still shows none |
| A pipeline blueprint node, an unbound input slot | raises, as `as_table()` does |

What each kind of object shows:
Expand All @@ -217,7 +218,8 @@ What each kind of object shows:
| An operator's or function pod's output stream | the rows already computed, or none with status `not_computed` |
| A job node | the rows it holds after `job.run()`, else the first of its stored rows, else none (`nothing_stored`) |
| A pipeline blueprint node, an unbound input slot | none (`unavailable`) |
| A side-effect stream | none: showing it never delivers its side effects |
| A job's side-effect node | the rows its last run passed on, else none (`not_computed`); it never delivers, and neither do its `as_table()`, `iter_data()` and other reads |
| A side-effect stream outside a job | none: showing it never delivers its side effects |

`head(n, columns=..., all_info=...)` takes the same column configuration as `as_table`,
sorting included, and applies it to the rows it has before taking the first `n`. The default
Expand Down
7 changes: 4 additions & 3 deletions src/orcapod/core/streams/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -601,9 +601,10 @@ def run(self) -> StreamView:
its ``head``, ``tail`` and ``sort_by_tags`` choose what is shown. For other
column groups, use ``as_table(columns=...)`` or ``head(columns=...)`` after
``run()``. On a stream that keeps its rows (a pod's or operator's output stream,
a job node, an in-memory source) they read the rows ``run()`` gave; a stream that
keeps none does its ``as_table()`` work again (a side-effect stream delivers
again, a polling source polls again).
a job node, a job's side-effect node included, an in-memory source) they read the
rows ``run()`` gave and deliver no side effect; a stream that keeps none does its
``as_table()`` work again (a side-effect stream outside a job delivers again, a
polling source polls again).

Returns:
A ``StreamView`` of all the rows ``as_table()`` returns, with status
Expand Down
6 changes: 3 additions & 3 deletions src/orcapod/core/streams/view.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,9 @@ class StreamView:
made with: for other column groups (system tags, source info, context, content
hashes) ask the stream, ``stream.as_table(columns=...)`` or
``stream.head(columns=...)``. After ``run()`` they read the rows a stream kept (a
pod's or operator's output stream, a job node, an in-memory source); a stream that
keeps none does its ``as_table()`` work again (a side-effect stream delivers again, a
polling source polls again).
pod's or operator's output stream, a job node, a job's side-effect node included, an
in-memory source); a stream that keeps none does its ``as_table()`` work again (a
side-effect stream outside a job delivers again, a polling source polls again).

Attributes:
table: The rows, with the exact Arrow types the stream gives them. When the
Expand Down
Loading
Loading