Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
3c8a69c
test(function_node): failing tests for the rows a job node returns af…
Nov 14, 2026
2e9972e
fix(function_node): after a run a job node returns exactly the rows o…
Nov 14, 2026
51cb1b5
docs(plans): ITL-1095 diagnosis and proposal (WIP)
Nov 14, 2026
1d99017
test(nodes): failing tests: a job node reads empty and stale before a…
Nov 14, 2026
b1a8d21
feat(nodes)!: a job node reads empty and stale before a run (ITL-1095)
Nov 14, 2026
7111a20
Merge remote-tracking branch 'origin/main' into eywalker/itl-1093-aft…
Nov 14, 2026
39aa109
test: read job nodes after a run, not before (ITL-1095)
Nov 14, 2026
f1c36f2
test: read stored results of loaded jobs through as_source() (ITL-1095)
Nov 14, 2026
bea357f
fix(function_node): count a run input only when it gave a row; docstr…
Nov 14, 2026
55e01eb
test: update operator, delta and stored-context reads to the current-…
Nov 14, 2026
bfefb42
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
3301092
fix(operator_node): the stored view of a loaded operator raises when …
Nov 14, 2026
c8f66c0
test(loaded_job): read loaded nodes' stored results through as_source…
Nov 14, 2026
da66b83
docs: a job node shows its current run, as_source() what is stored (I…
Nov 14, 2026
7411a89
test(pipeline): failing tests: job.run(load_only=True) (ITL-1095)
Nov 14, 2026
6541425
feat(pipeline): job.run(load_only=True) fills the nodes from what is …
Nov 14, 2026
c90eb96
docs: loading without computing, job.run(load_only=True) (ITL-1095)
Nov 14, 2026
aacd8ce
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
Nov 14, 2026
d389828
test(pipeline): failing tests: node.run() cascade, state rule and loa…
Nov 14, 2026
db1b80a
feat(nodes): node.run() runs the job nodes it depends on, by their st…
Nov 14, 2026
48a2015
fix(pipeline): a run whose end-of-run commit fails leaves no node hol…
Nov 14, 2026
9108423
docs: running one node, node states, and the run() argument rule (ITL…
Nov 14, 2026
a9d07f0
fix(function_node): any raise of a run clears the node's rows, also a…
Nov 14, 2026
2930456
test(function_node): a CACHE_ONLY node's run() raises and points to a…
Nov 14, 2026
03fa3aa
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
Nov 14, 2026
b693b50
test: #378's new raised-run tests read the stored view (ITL-1095); lo…
Nov 14, 2026
3610fd2
feat(pipeline): a loaded job is filled from its store by run(load_onl…
Nov 14, 2026
ee3e485
test(objective): a second node runs before it is read (ITL-1095)
Nov 14, 2026
40c54f0
test(arrays): read the v0.2-store jobs after a load-only run (ITL-1095)
Nov 14, 2026
74a2ddf
fix(pipeline): type-narrowing in the node.run() cascade (ITL-1095)
Nov 14, 2026
32bb766
fix(function_node): an async run whose output close fails defines no …
Nov 14, 2026
0b4c231
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
Nov 14, 2026
4d10024
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
2846dde
feat(side_effects): a side-effect job node follows the job node state…
Nov 14, 2026
c1a97de
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
4281755
test(pipeline): failing tests: job.run() follows the state rule (ITL-…
Nov 14, 2026
b1b9944
feat(pipeline)!: job.run() follows the job node state rule (ITL-1095)
Nov 14, 2026
3cd6bef
fix(nodes): review fixes for the current-run rule and load-only runs …
Nov 14, 2026
075b8e9
fix(pipeline): second review round for the current-run rule (ITL-1095)
Nov 14, 2026
9f145ce
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
fd812c1
test(pipeline): #379's async error-policy tests read a failed run's n…
Nov 14, 2026
f02d72a
fix(side_effects): a side-effect node filled by a load-only run shows…
Nov 14, 2026
8314c93
fix(pipeline): third review round for the current-run rule (ITL-1095)
Nov 14, 2026
b22763f
fix(pipeline): an interrupted end-of-run commit of job.run clears the…
Nov 14, 2026
7de3512
fix(pipeline): a run that leaves a node out still reports pre-v0.3 ta…
Nov 14, 2026
86902ec
chore(repro): reproduction scripts for the ITL-1121 review comments
Nov 14, 2026
ef2bb44
docs(plan): ITL-1121 implementation plan
Nov 14, 2026
b62feae
fix(function_node): stamp a node before on_node_end, so an observer r…
Nov 14, 2026
a4a9504
fix(operator_node): a run whose lazy output fails to compute holds no…
Nov 14, 2026
3404b7e
fix(run_summary): say 'nothing to do' only when the run ran no node (…
Nov 14, 2026
207f4ec
fix(side_effects): a load-only execute whose input raises clears the …
Nov 14, 2026
e3d4984
chore(repro): ruff-clean the ITL-1121 reproduction scripts
Nov 14, 2026
16a6210
fix(node_run): a load-only run refuses an input that delivers when re…
Nov 14, 2026
2c0f1f8
docs(function_pods): a held side-effect node delivers nothing on a se…
Nov 14, 2026
902e4e9
docs(changelog): ITL-1121 fragment and design issue entry
Nov 14, 2026
de87d16
test(itl-1121): type-clean the regression tests and reproduction scripts
Nov 14, 2026
979396f
style: ruff format
Nov 14, 2026
1d66a27
chore(repro): reproduce the PR #388 review's load-only delivery paths…
Nov 14, 2026
75324ab
fix(pipeline): PR #388 review round 1: more load-only delivery paths,…
Nov 14, 2026
6fe3c6e
chore(repro): a held node.run() keeps the previous last_run (observat…
Nov 14, 2026
019618d
Merge remote-tracking branch 'origin/main' into eywalker/itl-1121-job…
Nov 14, 2026
c76dee0
Merge remote-tracking branch 'origin/main' into eywalker/itl-1121-job…
Nov 14, 2026
d06ffcf
feat(node_run): a load-only run refuses an input that computes when r…
Nov 14, 2026
ac63354
docs(plan): ITL-1121 progress notes
Nov 14, 2026
5dd8d6a
docs(function_pods): name ITL-1130, which re-evaluates the load-only …
Nov 14, 2026
c906e56
Merge origin/main (ITL-912, #375) into ITL-1095
Nov 14, 2026
5a960d1
test(pipeline): a job node's current-run view has one Arrow schema in…
Nov 14, 2026
410420e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
8c44199
test(pipeline): docstrings for the current-run view schema tests (ITL…
Nov 14, 2026
1aa7ca2
Merge #380's final head (8c44199b, with #375 and #387) into ITL-1121
Nov 14, 2026
6b5db05
Merge origin/main (e4b4319c, #380 squash-merged) into ITL-1121
Nov 14, 2026
8455c2f
docs(claude): a failed end-of-run commit of node.run() clears only th…
Nov 14, 2026
d35d889
docs(plan): ITL-1121 merge of main after #380 merged
Nov 14, 2026
03a8717
docs(node_run): a job node behind an outside stream is read, not run …
Nov 14, 2026
82a6851
Merge origin/main (653c138e, #390) into ITL-1121
Nov 14, 2026
43bf5fd
test(scripts): type the c11 reproduction's pods for basedpyright (ITL…
Nov 14, 2026
fec5ab8
Merge origin/main (810d1b03, #389) into ITL-1121
Nov 14, 2026
9fc663d
docs(plan): ITL-1121 merges of #390 and #389
Nov 14, 2026
83350de
fix(node_run): the load-only refusal reads a held table and refuses a…
Nov 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -843,7 +843,7 @@ data. The rules:
non-stale run (for a full run: not load-only, nor anything above it) runs nothing;
otherwise its non-current dependencies run first, each once, on the sync orchestrator
(pipeline/node_run.py), and the run commits as job.run() does. A failed end-of-run
commit clears every node of the run. A load-only run's rows have status
commit clears only the nodes that ran; nodes it held keep their earlier run. A load-only run's rows have status
loaded_only. Only job nodes' run() takes an argument (load_only).
A loaded job (PipelineJob.load) runs only with load_only=True.
job.run()/run_async() use the same state rule and plan (node_run.plan_run): a current node is
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -885,7 +885,7 @@ data. The rules:
non-stale run (for a full run: not load-only, nor anything above it) runs nothing;
otherwise its non-current dependencies run first, each once, on the sync orchestrator
(`pipeline/node_run.py`), and the run commits as `job.run()` does. A failed end-of-run
commit clears every node of the run. A load-only run's rows have status
commit clears only the nodes that ran; nodes it held keep their earlier run. A load-only run's rows have status
`loaded_only`. Only job nodes' `run()` takes an argument (`load_only`).
A loaded job (`PipelineJob.load`) runs only with `load_only=True`.
`job.run()`/`run_async()` use the same state rule and plan (`node_run.plan_run`): a current node is
Expand Down
12 changes: 12 additions & 0 deletions changelog.d/ITL-1121.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
### Changed

- A load-only run (`job.run(load_only=True)`, `run_async(load_only=True)`, `node.run(load_only=True)`, a source slot's `run(load_only=True)`) reads only job nodes, sources and streams that neither compute nor deliver (developer, November 14, 2026; ITL-1121). A pod's stream built outside the job is refused with a `ValueError` before anything is read: a function pod's stream (a cached pod's included), which computes when it is read, and a side-effect pod's stream or a `SideEffectNode` outside a job, which deliver their side effects (before, the run read them and delivered every row). An `ArrowTableStream` holds its rows and is read whatever its `upstreams` name. This applies whether a job node reads the stream, a source slot is bound to it or an operator's stream outside the job reads it. Build the pod inside the job, or run without `load_only`; a run without `load_only` is unchanged. ITL-1130 re-evaluates the choice.

### Fixed

- An observer that read a function job node in `on_node_end` emptied it: the node was marked current only after that hook, so the read found it stale and cleared the run's rows (ITL-1121). The node is now current before the hook, on sync, async and load-only runs.
- An operator job node without a database, or in `CacheMode.OFF`, whose output failed to compute in `run()` kept claiming a current run and its reads raised; it now reads as not run, like every node whose run raised.
- The run summary said "nothing to do; every node holds a current run" when the run ran a side-effect node (and delivered) or an operator node below a held function node. It now says so only when the run ran no node; `RunSummary` takes `ran_nothing`.
- A side-effect job node whose load-only `execute()` failed on its input stayed current, and so did the nodes below it; it is now cleared, as on the async path.
- An operator job node without a database, run with `load_only=True`, showed its rows as a full run's (`rows` instead of `loaded_only`) and a later full `run()` took them as its own.
- A side-effect job node over a stream outside a job, with a store attached, kept its rows as a current run when its `run()` was interrupted (`KeyboardInterrupt`) while committing the invocation log, so the next `run()` neither ran nor committed again; it is now cleared.
39 changes: 39 additions & 0 deletions design_issues/ITL-1121-NV1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
## `src/orcapod/core/nodes/function_node.py`

### NV1 — Job node view: a run's rows lost to an observer, failed runs left current, a misleading summary
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1121

A full review of the job node current-run view (ITL-1095) found five defects, each reproduced
through the public interface (`superpowers/scripts/itl_1121/`):

- A function job node was stamped as modified after `on_node_end`, so an observer reading the
node there found it stale and cleared the run's rows; the node read empty after the run (sync,
async and both load-only paths).
- An operator job node without a database, or in `CacheMode.OFF`, computed its lazy output while
building the `run()` view, outside the block that clears a failed run: after a failure it
claimed a current run and its reads raised.
- `RunSummary` said "nothing to do; every node holds a current run" when the run ran only a
side-effect node (which delivered) or an operator node below a held function node: the
headline judged only the function nodes' counts.
- A side-effect job node's sync `execute(load_only=True)` whose input raised kept its modified
time, so it and the nodes below it were not stale.
- A load-only run read a side-effect pod's stream outside a job (directly, through another
stream outside a job, or bound to a source slot), which delivered every row, although a
load-only run delivers nothing; it also read a function pod's stream outside a job, which
computed.
- An operator job node without a database did not record a load-only run as one.
- A standalone side-effect job node interrupted while committing its invocation log kept a
current run, so the next run neither ran nor committed.

**Fix:** stamp before `on_node_end`; build the computed view inside the clearing block;
`RunSummary(ran_nothing=...)` from the run plan; clear the node when the load-only input
raises; the run plan and a source slot's `run()` refuse (`ValueError`, before reading) a
load-only run that would read a `SideEffectPodStream` or a `FunctionPodStream`
(`node_run.refuse_unloadable_inputs`, walking streams outside a job and bound source slots;
developer, November 14, 2026: "Refuse is consistent"); the no-database operator branch
records `load_only`; the side-effect node's log commit clears the node on any interruption. Two review comments did not
reproduce or are tracked elsewhere: a node run outside `job.run()` commits every row, so no
pre-run flush is needed; a function node with no database holding no rows after an async run is
ITL-1097.
16 changes: 13 additions & 3 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,14 @@ incomplete: a group formed after it may lack members, and the result stored for
group by an earlier run may then be loaded for it. `run_async(load_only=True)` does the same
on the asynchronous orchestrator.

A load-only run reads only job nodes, sources and streams that neither compute nor deliver.
A pod's stream built outside the job (a function pod's, which computes when it is read, or a
side-effect pod's, which delivers) is refused with a `ValueError` before anything is read,
whether a node of the job reads it, a source slot is bound to it or an operator's stream
outside the job reads it. Build the pod inside the job, so its node loads its stored results,
or run without `load_only`. (Refusing is the consistent choice for now; ITL-1130 re-evaluates
it.)

### Running again: what a run runs

`job.run()`, `await job.run_async()` and `job.nodes[label].run()` decide what to run from
Expand Down Expand Up @@ -832,14 +840,16 @@ notify_job.run()
node = notify_job.nodes["notify"]
print(node.as_table().num_rows, len(list(node.iter_data())), len(sent))
# 3 3 3
node.run() # delivers only what the invocation log does not record: nothing here
node.run() # holds a current run: runs nothing and delivers nothing
print(len(sent))
# 3
```

- With `track_completion=True` (the default), a run delivers only the rows whose delivery the
node's invocation log does not record. With `track_completion=False`, every run, a second
`job.run()` or `node.run()`, delivers every row again; reads still deliver nothing.
node's invocation log does not record. With `track_completion=False`, a run delivers every
row. A node that holds a current run is not run again, so a second `job.run()` or
`node.run()` delivers nothing; call `clear_cache()` on the node first to deliver again.
Reads never deliver.
- The rows a read returns are those the last run in this process passed on: under
`on_error="log"`, a row whose delivery failed is left out with `drop_on_failure=True`
(the default) and kept with `drop_on_failure=False`, as for the nodes downstream.
Expand Down
30 changes: 20 additions & 10 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -302,14 +302,20 @@ def run_job_node(
``RunSummary`` or ``None``.
caller: The frame of the ``run()`` caller.
"""
from orcapod.pipeline.node_run import depends_on_job_nodes, run_with_dependencies
from orcapod.pipeline.node_run import (
depends_on_job_nodes,
refuse_unloadable_inputs,
run_with_dependencies,
)
from orcapod.pipeline.run_summary import warn_row_failures

if node.holds_current_run(load_only=load_only):
return
if depends_on_job_nodes(node):
summary = run_with_dependencies(node, load_only=load_only)
else:
if load_only:
refuse_unloadable_inputs(node)
summary = run_alone()
if summary is None:
return
Expand Down Expand Up @@ -1767,8 +1773,9 @@ def _execute_load_only(
if entry[1] is not None:
output.append((entry[0], entry[1]))
self._set_run_rows(loaded_ids, load_only=True, not_stored=not_stored)
ctx_obs.on_node_end(node_label, node_hash)
# Current before the observer hears the end, so a read there sees the rows.
self._update_modified_time()
ctx_obs.on_node_end(node_label, node_hash)
except BaseException:
# A run that raised (an observer at the node's end included) defines no rows.
self.clear_cache()
Expand Down Expand Up @@ -1822,8 +1829,9 @@ async def _async_execute_load_only(
if entry[1] is not None:
await output.send((entry[0], entry[1]))
self._set_run_rows(loaded_ids, load_only=True, not_stored=not_stored)
ctx_obs.on_node_end(node_label, node_hash)
# Current before the observer hears the end, so a read there sees the rows.
self._update_modified_time()
ctx_obs.on_node_end(node_label, node_hash)
except BaseException:
# A run that raised or was cancelled defines no rows.
self.clear_cache()
Expand Down Expand Up @@ -2345,6 +2353,10 @@ def execute(
)
# The node's rows are those of this run's inputs, whatever else it caches.
self._set_run_rows(emitted_entry_ids, failed=failed_rows)
# Mark this node as freshly computed before the observer hears the end, so
# a read there (and every later one) serves this run's rows instead of
# finding the node stale and clearing them (ITL-1121).
self._update_modified_time()
ctx_obs.on_node_end(node_label, node_hash)
except Exception as run_error:
self._commit_after_failure(run_error)
Expand All @@ -2355,9 +2367,6 @@ def execute(
self.clear_cache()
raise

# Mark this node as freshly computed so subsequent iter_data() calls
# skip the is_stale check and serve results directly from the in-memory cache.
self._update_modified_time()
return output

def _process_data_internal(
Expand Down Expand Up @@ -4036,13 +4045,14 @@ def create_data_logger(self, tag: Any, data: Any) -> Any:
run_id=run_id,
)

# The run is complete: stamp the node as the sync execute() does, before
# the observer hears the end (a read there sees the rows, ITL-1121) and
# before the finally closes the output, so every downstream node, which
# stamps after it has seen that close, is newer than this one (ITL-1042).
self._update_modified_time()
# Ended for the observer here, also when the call raises: not again below.
node_open = False
ctx_obs.on_node_end(node_label, node_hash)
# The run is complete: stamp the node as the sync execute() does, before
# the finally closes the output, so every downstream node, which stamps
# after it has seen that close, is newer than this one (ITL-1042).
self._update_modified_time()
except asyncio.CancelledError as cancelled:
# Another node's failure cancelled this one: commit what it wrote, as a
# failing node does. The job's flush commits neither a node's ephemeral
Expand Down
9 changes: 9 additions & 0 deletions src/orcapod/core/nodes/operator_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -1617,6 +1617,7 @@ def run(self, *, load_only: bool = False) -> StreamView:
"""
from orcapod.pipeline.node_run import (
depends_on_job_nodes,
refuse_unloadable_inputs,
run_with_dependencies,
)
from orcapod.pipeline.run_summary import warn_row_failures
Expand All @@ -1633,6 +1634,8 @@ def run(self, *, load_only: bool = False) -> StreamView:
return self._run_view(computed=False)
if self._loaded:
raise self._cannot_compute_error()
if load_only:
refuse_unloadable_inputs(self)
if self.is_stale or not self._holds_run(load_only=load_only):
# Stale, or a load-only run's output asked for by a full run: run again.
self.clear_cache()
Expand All @@ -1655,8 +1658,14 @@ def run(self, *, load_only: bool = False) -> StreamView:
self._run_load_only = load_only
else:
self._cached_output_stream = self._compute_output()
# As in the branches with a database (ITL-1121).
self._run_load_only = load_only
self._update_modified_time()
computed = True
if computed:
# Building the view is part of the run: a failure there clears the
# node (ITL-1121).
return self._run_view(computed=True)
except BaseException:
# A run that raised defines no rows.
self.clear_cache()
Expand Down
11 changes: 9 additions & 2 deletions src/orcapod/core/nodes/source_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -594,15 +594,22 @@ def __init__(
def run(self, *, load_only: bool = False) -> StreamView:
"""Read the bound source's rows and return a ``StreamView`` of all of them.

A source is read, never computed, so ``load_only`` changes nothing; a job
node's ``run()`` takes it (ITL-1095).
A source is read, never computed, so ``load_only`` changes nothing, except that
a load-only run refuses a bound stream that computes or delivers when read
(ITL-1121); a job node's ``run()`` takes it (ITL-1095).

Args:
load_only: Accepted for every job node; a source reads its rows either way.

Raises:
UnboundSourceError: When no concrete source is bound.
ValueError: If ``load_only`` and the bound stream computes or delivers
when read (``node_run.refuse_unloadable_inputs``).
"""
if load_only:
from orcapod.pipeline.node_run import refuse_unloadable_inputs

refuse_unloadable_inputs(self)
return super().run()
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

@property
Expand Down
1 change: 1 addition & 0 deletions src/orcapod/pipeline/job.py
Original file line number Diff line number Diff line change
Expand Up @@ -1349,6 +1349,7 @@ def _finish_run(
"/".join(self._name),
run_node_summaries(prepared.exec_dag, counter, held=plan.held_counts),
load_only=load_only,
ran_nothing=not plan.to_run,
)
return self._last_run

Expand Down
Loading
Loading