Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
3c8a69c
test(function_node): failing tests for the rows a job node returns af…
Nov 14, 2026
2e9972e
fix(function_node): after a run a job node returns exactly the rows o…
Nov 14, 2026
51cb1b5
docs(plans): ITL-1095 diagnosis and proposal (WIP)
Nov 14, 2026
1d99017
test(nodes): failing tests: a job node reads empty and stale before a…
Nov 14, 2026
b1a8d21
feat(nodes)!: a job node reads empty and stale before a run (ITL-1095)
Nov 14, 2026
7111a20
Merge remote-tracking branch 'origin/main' into eywalker/itl-1093-aft…
Nov 14, 2026
39aa109
test: read job nodes after a run, not before (ITL-1095)
Nov 14, 2026
f1c36f2
test: read stored results of loaded jobs through as_source() (ITL-1095)
Nov 14, 2026
bea357f
fix(function_node): count a run input only when it gave a row; docstr…
Nov 14, 2026
55e01eb
test: update operator, delta and stored-context reads to the current-…
Nov 14, 2026
bfefb42
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
3301092
fix(operator_node): the stored view of a loaded operator raises when …
Nov 14, 2026
c8f66c0
test(loaded_job): read loaded nodes' stored results through as_source…
Nov 14, 2026
da66b83
docs: a job node shows its current run, as_source() what is stored (I…
Nov 14, 2026
7411a89
test(pipeline): failing tests: job.run(load_only=True) (ITL-1095)
Nov 14, 2026
6541425
feat(pipeline): job.run(load_only=True) fills the nodes from what is …
Nov 14, 2026
c90eb96
docs: loading without computing, job.run(load_only=True) (ITL-1095)
Nov 14, 2026
aacd8ce
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
Nov 14, 2026
d389828
test(pipeline): failing tests: node.run() cascade, state rule and loa…
Nov 14, 2026
db1b80a
feat(nodes): node.run() runs the job nodes it depends on, by their st…
Nov 14, 2026
48a2015
fix(pipeline): a run whose end-of-run commit fails leaves no node hol…
Nov 14, 2026
9108423
docs: running one node, node states, and the run() argument rule (ITL…
Nov 14, 2026
a9d07f0
fix(function_node): any raise of a run clears the node's rows, also a…
Nov 14, 2026
2930456
test(function_node): a CACHE_ONLY node's run() raises and points to a…
Nov 14, 2026
03fa3aa
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
Nov 14, 2026
b693b50
test: #378's new raised-run tests read the stored view (ITL-1095); lo…
Nov 14, 2026
3610fd2
feat(pipeline): a loaded job is filled from its store by run(load_onl…
Nov 14, 2026
ee3e485
test(objective): a second node runs before it is read (ITL-1095)
Nov 14, 2026
40c54f0
test(arrays): read the v0.2-store jobs after a load-only run (ITL-1095)
Nov 14, 2026
74a2ddf
fix(pipeline): type-narrowing in the node.run() cascade (ITL-1095)
Nov 14, 2026
32bb766
fix(function_node): an async run whose output close fails defines no …
Nov 14, 2026
0b4c231
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
Nov 14, 2026
4d10024
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
2846dde
feat(side_effects): a side-effect job node follows the job node state…
Nov 14, 2026
c1a97de
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
a1e1af7
docs(plans): ITL-1094 reproduction and proposal (WIP)
Nov 14, 2026
4281755
test(pipeline): failing tests: job.run() follows the state rule (ITL-…
Nov 14, 2026
62694bc
fix(pipeline): a loaded job restores its side-effect nodes, read-only…
Nov 14, 2026
2dd32ea
fix(pipeline): restore loaded side-effect nodes from their own module…
Nov 14, 2026
e78960c
docs: a loaded job keeps its side-effect nodes; changelog fragment an…
Nov 14, 2026
b1b9944
feat(pipeline)!: job.run() follows the job node state rule (ITL-1095)
Nov 14, 2026
3cd6bef
fix(nodes): review fixes for the current-run rule and load-only runs …
Nov 14, 2026
6f738ed
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
Nov 14, 2026
075b8e9
fix(pipeline): second review round for the current-run rule (ITL-1095)
Nov 14, 2026
9f145ce
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
fd812c1
test(pipeline): #379's async error-policy tests read a failed run's n…
Nov 14, 2026
f02d72a
fix(side_effects): a side-effect node filled by a load-only run shows…
Nov 14, 2026
dbf9c31
Merge #380 head f02d72ae (ITL-1095) into ITL-1094: loaded side-effect…
Nov 14, 2026
8314c93
fix(pipeline): third review round for the current-run rule (ITL-1095)
Nov 14, 2026
fc82fe8
Merge #380 round 3 (8314c934) into ITL-1094 (WIP, local)
Nov 14, 2026
b22763f
fix(pipeline): an interrupted end-of-run commit of job.run clears the…
Nov 14, 2026
7de3512
fix(pipeline): a run that leaves a node out still reports pre-v0.3 ta…
Nov 14, 2026
229f2c2
Merge branch 'eywalker/itl-1095-design-what-a-job-node-shows-the-curr…
Nov 14, 2026
f5a32a2
docs(plans): ITL-1094 merge of #380 rounds 3 and 4; stacked PR (WIP)
Nov 14, 2026
a6e4cf5
fix(pipeline): review round 1 for loaded side-effect nodes (ITL-1094)
Nov 14, 2026
c906e56
Merge origin/main (ITL-912, #375) into ITL-1095
Nov 14, 2026
5a960d1
test(pipeline): a job node's current-run view has one Arrow schema in…
Nov 14, 2026
410420e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
Nov 14, 2026
8c44199
test(pipeline): docstrings for the current-run view schema tests (ITL…
Nov 14, 2026
967f831
Merge #380 (8c44199b, with main 0208dca4) into eywalker/itl-1094-a-lo…
Nov 14, 2026
d1046a6
Merge origin/main (#380 merged) into eywalker/itl-1094-a-loaded-job-h…
Nov 14, 2026
5185317
fix(pipeline): a loaded side-effect node has the saved job's content …
Nov 14, 2026
a48dd6b
docs(plan): ITL-1094 follow-up, CodeRabbit schema finding (WIP notes)
Nov 14, 2026
5d28e26
fix(pipeline): a restored side-effect node passes its upstream's colu…
Nov 14, 2026
e73255b
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
Nov 14, 2026
c284b09
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
Nov 14, 2026
88e8517
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
Nov 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions changelog.d/ITL-1094.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
### Fixed

- **A loaded job keeps its side-effect nodes.** `PipelineJob.load` and `Pipeline.load`
dropped every side-effect node of the saved file: `loaded.nodes["<label>"]` was a
`KeyError`, the node below one had no upstream (a load-only run of it could not start), and
saving the loaded job again wrote a file without the node but with its edges. The node is
now restored read-only, in its place in the graph, with its saved label, pipeline hash,
output schema and the content hash the saved job's node had, as a loaded function node has. The file holds neither the side-effect function nor the
pod's configuration, so a loaded side-effect node never delivers and never writes its
invocation log. A load-only run (`loaded.run(load_only=True)`) passes its rows on to the
nodes below it, which it holds as loaded only; any other run raises. `PipelineJob.save` now
also records each side-effect node's content hash in the existing
`run.node_job_content_hashes` map of the job file, which the loader reads by label; a job
file saved before this change loads its side-effect nodes with the content hash their
descriptor records (the blueprint's). No hash value or stored table changes; files saved
by v0.2.0 (format `0.1.0`) restore their side-effect nodes too. Its schema under any
column configuration is its restored upstream's (system tags included), as for a live
side-effect node, and `as_pipeline()` of the loaded job keeps it. (ITL-1094)
- **A sync load-only run of a loaded job loads the function nodes below a Join.** On the
sync orchestrator, an operator below a node with no pod (a loaded function node, a restored
side-effect node) named its system-tag columns after a root's schema-only hash instead of
the node's pipeline hash, so the function nodes below it found none of their stored
entries (`not stored`), while the async orchestrator loaded them all. The sync path now
names them as the async path and the live job do. No hash value changes. (ITL-1094)
31 changes: 31 additions & 0 deletions design_issues/ITL-1094-LJ2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
## `src/orcapod/pipeline/job.py` — reading a loaded `PipelineJob`

### LJ2 — A loaded job dropped its side-effect nodes
**Status:** resolved
**Severity:** high
**Issue:** ITL-1094

A saved job file records each side-effect node (`node_type` `"side_effect"`: label, content
hash, pipeline hash, output schema, node uri, data context, Orcapod configuration), but
`Pipeline.load` had no branch for that node type and skipped it, and so did
`PipelineJob.load`. The loaded job's `nodes` lacked it (`KeyError`); the node below it had
`(None,)` as its upstreams and no edge in the job's graph, though its own hashes, read from
the file, were unchanged; a load-only run (ITL-1095) of a job with a side-effect node above a
function node raised `FunctionNode expects exactly 1 upstream, got 0`; and saving the loaded
job again wrote a file without the node but with its edges. Files written by v0.2.0 have the
same entries and v0.2.0 dropped them the same way.

**Fix:** `orcapod/pipeline/loaded_side_effects.py` restores the node read-only from its
entry: `LoadedSideEffectNode` (blueprint, `Pipeline.load`) and `LoadedSideEffectJobNode`
(`PipelineJob.load`), with the saved label, hashes, schema, node uri, data context and
config, and the restored upstream node as input. Neither has a pod: the file holds no
side-effect function and no pod configuration, so they never deliver and never write the
invocation log. In a load-only run the job node passes its rows on and holds them as
loaded only, through the live `SideEffectJobNode` load-only path, which uses no pod; before
one it reads no rows. `execute`, `async_execute` and `run()` without `load_only` raise
before anything is read. Saving the loaded job writes
the entry back unchanged. The job node's content hash is the saved job's: `PipelineJob.save`
records it in `run.node_job_content_hashes`, as for function nodes, and a file without that
entry gives the descriptor's (blueprint) hash. Nothing stored is keyed by a side-effect
node's content hash; a run's `pipeline_uri`, which hashes its end nodes, now matches the live
job's. No hash value or stored table changes.
10 changes: 10 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -937,6 +937,16 @@ from that store, and no rows under that key is an empty result, not an error.
exactly as the same node built from your code would; when the store holds no rows under
the node's key (no table, or none of its rows), the answer is an empty result, as for that
node. A loaded operator node itself is empty, like every node that has not run.
- A side-effect node is restored read-only, in its place in the graph, with its label,
pipeline hash (which names its invocation log), output schema and the content hash the saved
job's node had (a file saved before that hash was recorded gives the blueprint's), so the
nodes below it keep their own identity. The file holds neither the
side-effect function nor the pod's configuration (`on_error`, `track_completion`,
`drop_on_failure`, `max_concurrency`), so a loaded side-effect node never delivers and never
writes its invocation log. A load-only run passes its input rows on to the nodes below it
and the node holds them (status `loaded_only`), as any side-effect job node does in a
load-only run; before that its reads return no rows. A run that is not load-only
(`loaded.run()`, `node.run()`) raises before anything runs.
- A loaded node has the pipeline hash, content hash, table path and output schema its run
had (a source node whose data was not saved gets its run's content hash once its source is
bound). The file records which columns may be null (`T | None`), because the Arrow type text
Expand Down
47 changes: 38 additions & 9 deletions src/orcapod/core/operators/static_output_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,20 @@ def _as_upstream_stream(
pa.field(constants.CONTEXT_KEY, pa.large_string(), nullable=False),
pa.array([context.context_key] * table.num_rows, pa.large_string()),
)
if upstream.producer is None:
# A node with no pod (a loaded function node, a restored side-effect
# node, ITL-1094) cannot give the stream a producer to hash, so the
# stream reports the node's pipeline hash, as the async path does;
# otherwise it would fall back to a root's schema-only hash and an
# operator would name its system tags after that. A source's pipeline
# hash is that schema-only hash already.
return _RowsOfStream(
table,
tag_columns=tag_keys,
pipeline_hash=upstream.pipeline_hash(),
upstreams=upstream.upstreams,
data_context=context,
)
# No ``source_info`` argument: the table's own ``_source_*`` columns
# pass through, one value per row.
return ArrowTableStream(
Expand Down Expand Up @@ -648,14 +662,22 @@ async def async_execute(


class _RowsOfStream(ArrowTableStream):
"""Rows collected from a channel, standing in for the stream they came from.

The barrier-mode ``async_execute`` rebuilds each input from its rows. A
rebuilt ``ArrowTableStream`` has no producer, so its own pipeline hash is
the schema-only hash of a root; this stream reports the pipeline hash of
the stream the rows came from instead, as the operator node passes it
(``input_pipeline_hashes``). Its content hash is that of its table, as for
any ``ArrowTableStream`` without a producer.
"""Rows standing in for the stream they came from, with its pipeline hash.

The barrier-mode ``async_execute`` rebuilds each input from its rows, and the
sync path rebuilds the rows of an upstream with no producer
(``_as_upstream_stream``). A rebuilt ``ArrowTableStream`` has no producer, so
its own pipeline hash is the schema-only hash of a root; this stream reports
the pipeline hash of the stream the rows came from instead (the operator
node's ``input_pipeline_hashes``, or the upstream's own). Its content hash is
that of its table, as for any ``ArrowTableStream`` without a producer.

Args:
table: The rows.
tag_columns: The user tag columns.
pipeline_hash: The pipeline hash of the stream the rows came from.
upstreams: The upstreams of the stream the rows came from, if any.
data_context: The rows' data context; ``None`` for the default.
"""

def __init__(
Expand All @@ -664,8 +686,15 @@ def __init__(
*,
tag_columns: Collection[str],
pipeline_hash: ContentHash,
upstreams: tuple[StreamProtocol, ...] = (),
data_context: DataContext | None = None,
) -> None:
super().__init__(table, tag_columns=tag_columns)
super().__init__(
table,
tag_columns=tag_columns,
upstreams=upstreams,
data_context=data_context,
)
self._input_pipeline_hash: ContentHash = pipeline_hash

def pipeline_hash(
Expand Down
15 changes: 13 additions & 2 deletions src/orcapod/pipeline/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -503,7 +503,9 @@ def to_invocations(self) -> InvocationGraph:
a mapping of hash → source node (which acts as its own stream).

Raises:
RuntimeError: If the pipeline has not been compiled.
RuntimeError: If the pipeline has not been compiled, or if a node was
loaded from a saved file without its function pod, operator or
side-effect pod.
"""
import networkx as _nx

Expand Down Expand Up @@ -540,8 +542,17 @@ def to_invocations(self) -> InvocationGraph:
label=node._label,
)
elif isinstance(node, SideEffectNode):
side_effect_pod = node.producer
if side_effect_pod is None:
raise RuntimeError(
f"to_invocations() cannot serialise SideEffectNode "
f"{node_hash!r}: it was loaded with no live side-effect pod "
"(read-only / stub mode). Use PipelineJob.load() to work with "
"loaded pipelines, or build the pipeline from its code before "
"calling to_invocations() / from_pipeline()."
)
inv_by_node_hash[node_hash] = SideEffectInvocation(
pod=node._pod,
pod=side_effect_pod,
input_streams=(node.upstreams[0],),
label=node._label,
)
Expand Down
21 changes: 21 additions & 0 deletions src/orcapod/pipeline/graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,27 @@ def load(cls, path: str | Path) -> "Pipeline":
)
reconstructed[node_hash] = node

elif node_type == "side_effect":
from orcapod.pipeline.loaded_side_effects import LoadedSideEffectNode

# Restored read-only: the file holds no side-effect function (ITL-1094).
up_hashes = upstream_map.get(node_hash, [])
upstream_node = reconstructed.get(up_hashes[0]) if up_hashes else None
if upstream_node is None:
# Its input is a node this version does not restore: leave it out,
# as that node; the rest of the file still loads.
logger.warning(
"Skipping the saved side-effect node %r: its upstream node was "
"not restored from the file.",
descriptor.get("label"),
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
continue
# GraphNode predates side-effect nodes; compile() stores them in the
# same maps through the untyped side_effect_node_class.
reconstructed[node_hash] = LoadedSideEffectNode( # pyright: ignore[reportArgumentType]
descriptor, input_stream=upstream_node
)

# Build Pipeline instance
pipeline = cls(name=name, auto_compile=False)
pipeline._unrecorded_output_nullability = unrecorded_output_nullability
Expand Down
57 changes: 44 additions & 13 deletions src/orcapod/pipeline/job.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@
import sys
from collections.abc import Iterator
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from typing import TYPE_CHECKING, Any, cast

import orcapod
from orcapod.core.nodes import JobNode
from orcapod.core.nodes.function_node import FunctionJobNode
from orcapod.core.nodes.operator_node import OperatorJobNode
from orcapod.core.nodes.source_node import SourceJobNode
from orcapod.core.streams.base import StreamBase
from orcapod.side_effects import SideEffectJobNode
from orcapod.side_effects import SideEffectJobNode, SideEffectNode
from orcapod.core.tracker import AutoRegisteringContextBasedTracker
from orcapod.contexts.stored import (
report_unclaimed_tables,
Expand Down Expand Up @@ -609,6 +609,10 @@ def as_pipeline(self) -> "Pipeline":
from orcapod.core.nodes.function_node import FunctionNode
from orcapod.core.nodes.operator_node import OperatorNode
from orcapod.pipeline.graph import Pipeline
from orcapod.pipeline.loaded_side_effects import (
LoadedSideEffectJobNode,
LoadedSideEffectNode,
)
from orcapod.side_effects import SideEffectNode

if not self._compiled:
Expand Down Expand Up @@ -659,6 +663,15 @@ def as_pipeline(self) -> "Pipeline":
tracker_manager=job_node.tracker_manager,
)

elif isinstance(job_node, LoadedSideEffectJobNode):
# No pod to wrap: the blueprint node is restored from the saved
# entry, with the saved blueprint identity (ITL-1094).
upstream_bp_hash = job_id_to_bp_hash[id(job_node._input_stream)]
node_map[node_hash] = LoadedSideEffectNode(
job_node._descriptor,
cast("cp.StreamProtocol", node_map[upstream_bp_hash]),
)

elif isinstance(job_node, SideEffectJobNode):
upstream_bp_hash = job_id_to_bp_hash[id(job_node._input_stream)]
node_map[node_hash] = SideEffectNode(
Expand Down Expand Up @@ -1142,8 +1155,9 @@ def _prepare_run(
excluded_hashes: set[str] = set()
for node_hash in hash_dag.topological_sort():
if node_hash not in self._persistent_node_map:
# A node a loaded job did not restore (ITL-1094): below an unbound
# source it is left out too, so the nodes below it are as well.
# A node a loaded job did not restore (a kind PipelineJob.load does
# not know): below an unbound source it is left out too, so the nodes
# below it are as well.
if any(p in excluded_hashes for p in hash_dag.predecessors(node_hash)):
excluded_hashes.add(node_hash)
continue
Expand All @@ -1153,9 +1167,9 @@ def _prepare_run(
elif any(p in excluded_hashes for p in hash_dag.predecessors(node_hash)):
excluded_hashes.add(node_hash)

# A loaded job whose graph lost a node (PipelineJob.load does not restore
# side-effect nodes yet, ITL-1094) cannot feed the nodes below it: every saved
# input of a node must be a restored node.
# A loaded job whose graph lost a node (the file records a node of a kind
# PipelineJob.load does not restore) cannot feed the nodes below it: every
# saved input of a node must be a restored node.
if self._loaded:
for node_hash, node in self._persistent_node_map.items():
if node_hash in excluded_hashes:
Expand All @@ -1172,10 +1186,10 @@ def _prepare_run(
if missing:
raise RuntimeError(
f"Node {node.label!r} of the loaded job has lost "
f"{len(missing)} of its input nodes: PipelineJob.load does not "
"restore side-effect nodes yet (ITL-1094), so the nodes below "
"one cannot run, not even load-only. Read their stored results "
"with node.as_source()."
f"{len(missing)} of its input nodes: the saved file records a "
"node of a kind PipelineJob.load does not restore, so the nodes "
"below it cannot run, not even load-only. Read their stored "
"results with node.as_source()."
)

# --- Build execution OrcaDAG from existing node objects (no cloning) ---
Expand Down Expand Up @@ -1427,8 +1441,8 @@ def save(self, path: "str | Path") -> None:
else:
status = "complete"

# Collect the live (data-inclusive) content hashes of function and operator
# job nodes.
# Collect the live (data-inclusive) content hashes of function, operator and
# side-effect job nodes.
# The pipeline blueprint stores schema-only blueprint hashes; but the DB
# records are written with the live hash (which includes the upstream data
# content). Saving these ensures that PipelineJob.load() can reconstruct
Expand All @@ -1454,6 +1468,10 @@ def save(self, path: "str | Path") -> None:
run_content_hash = node._run_content_hash()
if run_content_hash is not None:
node_job_content_hashes[node.label] = run_content_hash.to_string()
elif isinstance(node, SideEffectJobNode):
# Its descriptor holds the blueprint's hash; a loaded side-effect node
# takes the job's from here, as a function node does (ITL-1094).
node_job_content_hashes[node.label] = node.content_hash().to_string()

output: dict[str, Any] = {
"orcapod_pipeline_job_version": PIPELINE_JOB_FORMAT_VERSION,
Expand Down Expand Up @@ -1644,6 +1662,7 @@ def load(
from orcapod.core.nodes.function_node import FunctionNode as _FunctionNode
from orcapod.core.nodes.operator_node import OperatorNode as _OperatorNode
from orcapod.core.nodes.source_node import SourceNode as _SourceNode
from orcapod.pipeline.loaded_side_effects import LoadedSideEffectJobNode

job._sources = dict(sources)

Expand Down Expand Up @@ -1716,6 +1735,18 @@ def load(
)
job._persistent_node_map[node_hash] = ojn

elif isinstance(bp_node, SideEffectNode):
# Read-only: it never delivers (ITL-1094). Its upstream is the restored
# job node, so the nodes below it keep their position in the graph.
preds = list(job._hash_graph.predecessors(node_hash))
# JobNode predates side-effect nodes; compile() stores them in this map
# too, through the untyped side_effect_node_class.
job._persistent_node_map[node_hash] = LoadedSideEffectJobNode( # pyright: ignore[reportArgumentType]
getattr(bp_node, "_descriptor", None) or {},
input_stream=job._persistent_node_map[preds[0]],
job_content_hash=node_job_content_hashes.get(bp_node.label),
)

# Update _nodes to point to job nodes (FunctionJobNode/OperatorJobNode stubs)
# rather than blueprint nodes, so that job.nodes["label"].get_all_records()
# works on a loaded job.
Expand Down
Loading
Loading