Repository navigation
feat(nodes)!: a job node shows its current run; job.run(load_only=True) (ITL-1095) #380
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 49 commits into
main
from
eywalker/itl-1095-design-what-a-job-node-shows-the-current
Nov 14, 2026
Merged
Changes from all commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
3c8a69c
test(function_node): failing tests for the rows a job node returns af…
2e9972e
fix(function_node): after a run a job node returns exactly the rows o…
51cb1b5
docs(plans): ITL-1095 diagnosis and proposal (WIP)
1d99017
test(nodes): failing tests: a job node reads empty and stale before a…
b1a8d21
feat(nodes)!: a job node reads empty and stale before a run (ITL-1095)
7111a20
Merge remote-tracking branch 'origin/main' into eywalker/itl-1093-aft…
39aa109
test: read job nodes after a run, not before (ITL-1095)
f1c36f2
test: read stored results of loaded jobs through as_source() (ITL-1095)
bea357f
fix(function_node): count a run input only when it gave a row; docstr…
55e01eb
test: update operator, delta and stored-context reads to the current-…
bfefb42
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
3301092
fix(operator_node): the stored view of a loaded operator raises when …
c8f66c0
test(loaded_job): read loaded nodes' stored results through as_source…
da66b83
docs: a job node shows its current run, as_source() what is stored (I…
7411a89
test(pipeline): failing tests: job.run(load_only=True) (ITL-1095)
6541425
feat(pipeline): job.run(load_only=True) fills the nodes from what is …
c90eb96
docs: loading without computing, job.run(load_only=True) (ITL-1095)
aacd8ce
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
d389828
test(pipeline): failing tests: node.run() cascade, state rule and loa…
db1b80a
feat(nodes): node.run() runs the job nodes it depends on, by their st…
48a2015
fix(pipeline): a run whose end-of-run commit fails leaves no node hol…
9108423
docs: running one node, node states, and the run() argument rule (ITL…
a9d07f0
fix(function_node): any raise of a run clears the node's rows, also a…
2930456
test(function_node): a CACHE_ONLY node's run() raises and points to a…
03fa3aa
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
b693b50
test: #378's new raised-run tests read the stored view (ITL-1095); lo…
3610fd2
feat(pipeline): a loaded job is filled from its store by run(load_onl…
ee3e485
test(objective): a second node runs before it is read (ITL-1095)
40c54f0
test(arrays): read the v0.2-store jobs after a load-only run (ITL-1095)
74a2ddf
fix(pipeline): type-narrowing in the node.run() cascade (ITL-1095)
32bb766
fix(function_node): an async run whose output close fails defines no …
0b4c231
Merge remote-tracking branch 'origin/eywalker/itl-1093-after-an-async…
4d10024
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
2846dde
feat(side_effects): a side-effect job node follows the job node state…
c1a97de
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
4281755
test(pipeline): failing tests: job.run() follows the state rule (ITL-…
b1b9944
feat(pipeline)!: job.run() follows the job node state rule (ITL-1095)
3cd6bef
fix(nodes): review fixes for the current-run rule and load-only runs …
075b8e9
fix(pipeline): second review round for the current-run rule (ITL-1095)
9f145ce
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
fd812c1
test(pipeline): #379's async error-policy tests read a failed run's n…
f02d72a
fix(side_effects): a side-effect node filled by a load-only run shows…
8314c93
fix(pipeline): third review round for the current-run rule (ITL-1095)
b22763f
fix(pipeline): an interrupted end-of-run commit of job.run clears the…
7de3512
fix(pipeline): a run that leaves a node out still reports pre-v0.3 ta…
c906e56
Merge origin/main (ITL-912, #375) into ITL-1095
5a960d1
test(pipeline): a job node's current-run view has one Arrow schema in…
410420e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1095-des…
8c44199
test(pipeline): docstrings for the current-run view schema tests (ITL…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| ### Breaking changes and migration | ||
|
|
||
| - A job node shows the current run; `node.as_source()` shows everything stored for it (developer, November 13, 2026; ITL-1095). Before a run in this process, a function or operator job node is empty: `as_table()`, `iter_data()`, the dataframe methods, `flow()`, `head()` and the notebook preview return zero rows with the node's columns, and `head()` (status `not_computed`) says the node has not run and points to `run()` and to `node.as_source()`. Before, a function node built over a store that already held results, or loaded with `PipelineJob.load`, returned every row stored under its key, and an operator node in `CacheMode.REPLAY` returned every logged run. The same holds after `clear_cache()`, after a run that raised, and while the node is stale. Migration: read stored results with `node.as_source()` (or `get_all_records()`), or run the job first; a run over a store that holds every result computes nothing. | ||
| - A function or operator job node that has not run in this process is stale (`is_stale` is `True`; it has no modified time), and a job node is stale also when any function or operator job node it depends on is. Binding a source to a job's source slot (`job.bind(sources=...)`) marks the slot modified, so every node fed by it is stale and empty until it runs again. | ||
| - A loaded operator node no longer raises from `as_table()` or `iter_data()`: it is empty, like every node that has not run. Its stored view (`as_source()`, `get_all_records()`) raises instead when the job file does not record its run's content hash and the store holds nothing under the hash the loaded job computes (ITL-1043). | ||
| - A job node's `run()` is state-driven and runs what the node depends on (developer, November 13, 2026; ITL-1095). A node that holds a completed run and is not stale runs nothing: `run()` returns the rows it holds. Otherwise every job node it depends on that holds no current run runs first, each once, as `job.run()` would run that part of the job, then the node; the run's writes are committed when it ends (before, `node.run()` read an upstream node's stored rows, or nothing, and left its last pipeline record and a `LOG` operator's output uncommitted). `run()` of a node whose function pod is a placeholder, or that was loaded without one, raises unless `load_only=True`; a `CACHE_ONLY` node's `run()` raises (its input is unavailable): read its stored results with `as_source()`. A job node's `run()` takes one keyword-only argument, `load_only`; every other stream's `run()` still takes none. | ||
| - A side-effect job node follows the same rule: it is stale before a run, and while it holds a current run neither its `run()`, nor a downstream node's `run()`, nor `job.run()` delivers anything; to deliver again without `track_completion`, call `clear_cache()` on it first. A load-only run passes its rows on and delivers nothing. | ||
| - `job.run()` and `job.run_async()` follow the same rule as `node.run()` (developer, November 14, 2026): a node that holds a current run is not run again, and its rows feed the nodes after it. Running an unchanged job a second time now computes nothing, reads and writes nothing in the store, does not read its sources and delivers no side effect, even without `track_completion` (before, every `job.run()` re-read every source and re-processed every node, and delivered every untracked side effect again). Nodes run when they have not run in this process, were cleared, are below a cleared node or a node that ran again, or below a source bound again or whose own modified time moved. After this change a re-run does not notice new data in a source whose object is unchanged (a file or table changed in place, a polling source with new rows; a start-of-run check for new data is ITL-1102). A source that holds the data it read when built (in-memory, CSV, Delta, database tables, a pinned `SpiralDBTableSource`) needs a new source object, bound with `job.bind(sources=...)`; for a `PollingSource`, `clear_cache()` on each node that reads it makes the next run poll it. The summary reports a node that was not run as held: "rowwise: 6 rows: held from the previous run" (`NodeRunSummary.held`). If the commit at the end of a run fails, the nodes it ran are cleared and the nodes it held keep their rows. | ||
| - If a run raises or is interrupted, or the commit at the end of `job.run()` or `node.run()` fails, every node the run ran is cleared (not run, stale) before the error propagates; nodes it held keep their rows. A run that left failed rows (`error_policy="continue"`) is never current: the next run computes only the failed rows and reads the rest from the store. | ||
|
|
||
| ### Added | ||
|
|
||
| - `job.run(load_only=True)` and `await job.run_async(load_only=True)` fill a job's nodes with what is already stored for this run's inputs and compute nothing (ITL-1095): sources are read, operators run on the rows that reach them (a `CacheMode.LOG` cache writes nothing), each function node takes the stored results of its inputs and skips the inputs without one, a side-effect node passes its rows on without delivering anything, and nothing is written to the store. `RunSummary.load_only`, `NodeRunSummary.not_stored` (inputs skipped) and `NodeRunSummary.inputs_incomplete` (the function nodes above a node that skipped inputs) report it; loaded rows count in `reused`, and the summary's text says "3 loaded, 1 not stored (skipped)". | ||
|
|
||
| - `node.run(load_only=True)`, and `node.last_run`, the `RunSummary` of a node's last `run()`. `ViewStatus.LOADED_ONLY`: the `head()` and `run()` view of a node filled by a load-only run, captioned "N loaded, M inputs without a stored result" and naming the function nodes above it that skipped inputs. `holds_current_run(load_only=...)` on function and operator job nodes says whether `run()` would run them. | ||
|
|
||
| - A job loaded with `PipelineJob.load` can be filled from its store: `loaded.run(load_only=True)` (and `node.run(load_only=True)` of a loaded node) reads the job's sources (rebuilt from files, or bound with `loaded.bind(sources=...)`), runs the rebuilt operators and gives each function node the results its store holds for those inputs; nodes below an unbound source stay empty. `loaded.run()` without `load_only` still raises, and says how to load. | ||
|
|
||
| ### Fixed | ||
|
|
||
| - On the asynchronous orchestrator, an operator node in `CacheMode.REPLAY` stamps itself after its inputs close, so the rows it replayed are not judged out of date by the time the run ends. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Limit commit-failure cleanup to nodes that ran.
If dependency A is held and dependent B runs, this sentence instructs clearing both nodes after a failed commit. Lines 825-826 instead require clearing only B. The supplied
run_with_dependenciesalso passesplan.to_runtocommit_run.Change this sentence to “A failed end-of-run commit clears only the nodes that ran.” This avoids instructing agents to discard valid held results.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid, but declined in this PR. The sentence is wrong:
node_run.run_with_dependenciescallscommit_run(stores, plan.to_run), so a failed end-of-run commit clears only the nodes that ran, as the sentence five lines below says. It is not part of this round's change, though: the text was already in 7de3512, before the merges of main reviewed here, and three stacked PRs build on this branch. The fix ("A failed end-of-run commit clears only the nodes that ran." inCLAUDE.mdand.zed/rules) goes to ITL-1121, PR #388.Posted by a Claude Code agent on behalf of @eywalker.