Repository navigation
fix(pipeline): a loaded job keeps its side-effect nodes, read-only (ITL-1094) #381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 67 commits into
main
from
eywalker/itl-1094-a-loaded-job-has-no-side-effect-nodes
Nov 15, 2026
Merged
Changes from all commits
Commits
Show all changes
67 commits
Select commit
Hold shift + click to select a range
3c8a69c
test(function_node): failing tests for the rows a job node returns af…
2e9972e
fix(function_node): after a run a job node returns exactly the rows o…
51cb1b5
docs(plans): ITL-1095 diagnosis and proposal (WIP)
1d99017
test(nodes): failing tests: a job node reads empty and stale before a…
b1a8d21
feat(nodes)!: a job node reads empty and stale before a run (ITL-1095)
7111a20
Merge remote-tracking branch 'origin/main' into eywalker/itl-1093-aft…
39aa109
test: read job nodes after a run, not before (ITL-1095)
f1c36f2
test: read stored results of loaded jobs through as_source() (ITL-1095)
bea357f
fix(function_node): count a run input only when it gave a row; docstr…
55e01eb
test: update operator, delta and stored-context reads to the current-…
bfefb42
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
3301092
fix(operator_node): the stored view of a loaded operator raises when …
c8f66c0
test(loaded_job): read loaded nodes' stored results through as_source…
da66b83
docs: a job node shows its current run, as_source() what is stored (I…
7411a89
test(pipeline): failing tests: job.run(load_only=True) (ITL-1095)
6541425
feat(pipeline): job.run(load_only=True) fills the nodes from what is …
c90eb96
docs: loading without computing, job.run(load_only=True) (ITL-1095)
aacd8ce
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
d389828
test(pipeline): failing tests: node.run() cascade, state rule and loa…
db1b80a
feat(nodes): node.run() runs the job nodes it depends on, by their st…
48a2015
fix(pipeline): a run whose end-of-run commit fails leaves no node hol…
9108423
docs: running one node, node states, and the run() argument rule (ITL…
a9d07f0
fix(function_node): any raise of a run clears the node's rows, also a…
2930456
test(function_node): a CACHE_ONLY node's run() raises and points to a…
03fa3aa
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
b693b50
test: #378's new raised-run tests read the stored view (ITL-1095); lo…
3610fd2
feat(pipeline): a loaded job is filled from its store by run(load_onl…
ee3e485
test(objective): a second node runs before it is read (ITL-1095)
40c54f0
test(arrays): read the v0.2-store jobs after a load-only run (ITL-1095)
74a2ddf
fix(pipeline): type-narrowing in the node.run() cascade (ITL-1095)
32bb766
fix(function_node): an async run whose output close fails defines no …
0b4c231
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
4d10024
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
2846dde
feat(side_effects): a side-effect job node follows the job node state…
c1a97de
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
a1e1af7
docs(plans): ITL-1094 reproduction and proposal (WIP)
4281755
test(pipeline): failing tests: job.run() follows the state rule (ITL-…
62694bc
fix(pipeline): a loaded job restores its side-effect nodes, read-only…
2dd32ea
fix(pipeline): restore loaded side-effect nodes from their own module…
e78960c
docs: a loaded job keeps its side-effect nodes; changelog fragment an…
b1b9944
feat(pipeline)!: job.run() follows the job node state rule (ITL-1095)
3cd6bef
fix(nodes): review fixes for the current-run rule and load-only runs …
6f738ed
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
075b8e9
fix(pipeline): second review round for the current-run rule (ITL-1095)
9f145ce
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
fd812c1
test(pipeline): #379's async error-policy tests read a failed run's n…
f02d72a
fix(side_effects): a side-effect node filled by a load-only run shows…
dbf9c31
Merge #380 head f02d72ae (ITL-1095) into ITL-1094: loaded side-effect…
8314c93
fix(pipeline): third review round for the current-run rule (ITL-1095)
fc82fe8
Merge #380 round 3 (8314c934) into ITL-1094 (WIP, local)
b22763f
fix(pipeline): an interrupted end-of-run commit of job.run clears the…
7de3512
fix(pipeline): a run that leaves a node out still reports pre-v0.3 ta…
229f2c2
Merge branch 'eywalker/itl-1095-design-what-a-job-node-shows-the-curr…
f5a32a2
docs(plans): ITL-1094 merge of #380 rounds 3 and 4; stacked PR (WIP)
a6e4cf5
fix(pipeline): review round 1 for loaded side-effect nodes (ITL-1094)
c906e56
Merge origin/main (ITL-912, #375) into ITL-1095
5a960d1
test(pipeline): a job node's current-run view has one Arrow schema in…
410420e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
8c44199
test(pipeline): docstrings for the current-run view schema tests (ITL…
967f831
Merge #380 (8c44199b, with main 0208dca4) into eywalker/itl-1094-a-lo…
d1046a6
Merge origin/main (#380 merged) into eywalker/itl-1094-a-loaded-job-h…
5185317
fix(pipeline): a loaded side-effect node has the saved job's content …
a48dd6b
docs(plan): ITL-1094 follow-up, CodeRabbit schema finding (WIP notes)
5d28e26
fix(pipeline): a restored side-effect node passes its upstream's colu…
e73255b
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
c284b09
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
88e8517
Merge remote-tracking branch 'origin/main' into eywalker/itl-1094-a-l…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| ### Fixed | ||
|
|
||
| - **A loaded job keeps its side-effect nodes.** `PipelineJob.load` and `Pipeline.load` | ||
| dropped every side-effect node of the saved file: `loaded.nodes["<label>"]` was a | ||
| `KeyError`, the node below one had no upstream (a load-only run of it could not start), and | ||
| saving the loaded job again wrote a file without the node but with its edges. The node is | ||
| now restored read-only, in its place in the graph, with its saved label, pipeline hash, | ||
| output schema and the content hash the saved job's node had, as a loaded function node has. The file holds neither the side-effect function nor the | ||
| pod's configuration, so a loaded side-effect node never delivers and never writes its | ||
| invocation log. A load-only run (`loaded.run(load_only=True)`) passes its rows on to the | ||
| nodes below it, which it holds as loaded only; any other run raises. `PipelineJob.save` now | ||
| also records each side-effect node's content hash in the existing | ||
| `run.node_job_content_hashes` map of the job file, which the loader reads by label; a job | ||
| file saved before this change loads its side-effect nodes with the content hash their | ||
| descriptor records (the blueprint's). No hash value or stored table changes; files saved | ||
| by v0.2.0 (format `0.1.0`) restore their side-effect nodes too. Its schema under any | ||
| column configuration is its restored upstream's (system tags included), as for a live | ||
| side-effect node, and `as_pipeline()` of the loaded job keeps it. (ITL-1094) | ||
| - **A sync load-only run of a loaded job loads the function nodes below a Join.** On the | ||
| sync orchestrator, an operator below a node with no pod (a loaded function node, a restored | ||
| side-effect node) named its system-tag columns after a root's schema-only hash instead of | ||
| the node's pipeline hash, so the function nodes below it found none of their stored | ||
| entries (`not stored`), while the async orchestrator loaded them all. The sync path now | ||
| names them as the async path and the live job do. No hash value changes. (ITL-1094) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| ## `src/orcapod/pipeline/job.py` — reading a loaded `PipelineJob` | ||
|
|
||
| ### LJ2 — A loaded job dropped its side-effect nodes | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-1094 | ||
|
|
||
| A saved job file records each side-effect node (`node_type` `"side_effect"`: label, content | ||
| hash, pipeline hash, output schema, node uri, data context, Orcapod configuration), but | ||
| `Pipeline.load` had no branch for that node type and skipped it, and so did | ||
| `PipelineJob.load`. The loaded job's `nodes` lacked it (`KeyError`); the node below it had | ||
| `(None,)` as its upstreams and no edge in the job's graph, though its own hashes, read from | ||
| the file, were unchanged; a load-only run (ITL-1095) of a job with a side-effect node above a | ||
| function node raised `FunctionNode expects exactly 1 upstream, got 0`; and saving the loaded | ||
| job again wrote a file without the node but with its edges. Files written by v0.2.0 have the | ||
| same entries and v0.2.0 dropped them the same way. | ||
|
|
||
| **Fix:** `orcapod/pipeline/loaded_side_effects.py` restores the node read-only from its | ||
| entry: `LoadedSideEffectNode` (blueprint, `Pipeline.load`) and `LoadedSideEffectJobNode` | ||
| (`PipelineJob.load`), with the saved label, hashes, schema, node uri, data context and | ||
| config, and the restored upstream node as input. Neither has a pod: the file holds no | ||
| side-effect function and no pod configuration, so they never deliver and never write the | ||
| invocation log. In a load-only run the job node passes its rows on and holds them as | ||
| loaded only, through the live `SideEffectJobNode` load-only path, which uses no pod; before | ||
| one it reads no rows. `execute`, `async_execute` and `run()` without `load_only` raise | ||
| before anything is read. Saving the loaded job writes | ||
| the entry back unchanged. The job node's content hash is the saved job's: `PipelineJob.save` | ||
| records it in `run.node_job_content_hashes`, as for function nodes, and a file without that | ||
| entry gives the descriptor's (blueprint) hash. Nothing stored is keyed by a side-effect | ||
| node's content hash; a run's `pipeline_uri`, which hashes its end nodes, now matches the live | ||
| job's. No hash value or stored table changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.