Repository navigation
fix(nodes, function_pod): no node or pod stream is stale after a completed run or read (ITL-1042, ITL-1052) #355
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 16 commits into
main
from
eywalker/itl-1042-after-an-async-job-run-an-operator-nodes
Nov 13, 2026
Merged
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
012c5a1
test(nodes): node staleness and head after sync and async runs (ITL-1…
6dc70a5
fix(function_node): stamp a function node's modified time when its as…
c7cfc3c
docs(streams): a rowless head shows the declared columns; nodes after…
0fb6c4a
test(function_node): a function node is stamped before its async outp…
b5fe007
Merge remote-tracking branch 'origin/main' into eywalker/itl-1042-aft…
4535769
test(function_pod): a chain of pod streams is not read again after a …
dc2cdb3
test(function_pod): type-clean executor in the pod stream staleness t…
dcb8ae8
test(function_pod): an upstream reset during a read leaves the reader…
c30950a
docs(plan): ITL-1052 cause, rule and design (ITL-1042)
0ea878c
test(function_pod): mid-read upstream reset for cached chains and thr…
f7b5d53
Merge remote-tracking branch 'origin/main' into eywalker/itl-1042-aft…
328761c
fix(function_pod): a pod stream stamps itself when its input is read …
e002f99
docs: one modified-time rule for nodes and pod streams (ITL-1042, ITL…
b84c966
Merge remote-tracking branch 'origin/main' into eywalker/itl-1042-aft…
2f32e7e
fix(function_pod): note the input state before the concurrent computa…
963310f
Merge remote-tracking branch 'origin/main' into eywalker/itl-1042-aft…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| ### Fixed | ||
|
|
||
| #### A node's preview shows its rows after an async run (ITL-1042) | ||
|
|
||
| After a job ran on `AsyncPipelineOrchestrator`, a function node downstream of an operator | ||
| node (for example the automatic column selection a pod call adds) was taken to be out of | ||
| date. Its first `as_table()`, `iter_data()` or `head()` read the store again instead of the | ||
| results in memory, and from then on every operator node downstream of it showed | ||
| `head()` as "not computed: inputs changed since the last computation", with no rows, although | ||
| `as_table()` and `iter_data()` returned every row. After a run on either orchestrator no node | ||
| is now out of date: `head()` shows the rows `as_table()` returns, and none of them reads the | ||
| store. Nothing was ever computed twice. (ITL-1042) | ||
|
|
||
| #### A chain of function pod streams is not computed again on its second read (ITL-1052) | ||
|
|
||
| In a chain such as `g.pod(f.pod(source))`, the last stream was taken to be out of date right | ||
| after it was read to the end, so its next `iter_data()`, `as_table()` or `head()` called `g` | ||
| again on every row (or, for a cached pod, looked every row up in its store again). A pod | ||
| stream now records that its results are complete when it has read its input to the end, so a | ||
| second read computes nothing. A change upstream (`clear_cache()`), also one made while the | ||
| chain is being read, still makes every stream after it out of date, and its next read | ||
| computes it again. (ITL-1052) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| ## `src/orcapod/core/nodes/function_node.py` | ||
|
|
||
| ### NS1 — After an async run a function node looked stale, so downstream heads said "not computed" | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-1042 | ||
|
|
||
| A node is stale when an upstream (or its pod) has a newer modified time than its own. The sync | ||
| `FunctionJobNode.execute` stamps the node when its run completes; `async_execute` never did. | ||
| An `OperatorJobNode` stamps itself when its async run finishes, so a function node downstream | ||
| of an operator node (most often the automatic `SelectDataColumns` that `pod(stream)` adds when | ||
| the stream has columns the function does not take, ITL-807) was stale right after an async | ||
| run. Its first `iter_data()` / `as_table()` then threw away the in-memory results, re-read the | ||
| node's whole stored table (two `get_all_records` calls) and stamped the node. That made every | ||
| operator node downstream of it stale, and their `head()` reported `not_computed` ("inputs | ||
| changed since the last computation") with no rows, although `as_table()` and `iter_data()` | ||
| still returned every row. `head()` of the function node itself read the store instead of | ||
| memory. No function was called again and no operator recomputed. | ||
|
|
||
| **Rule:** a job node stamps its modified time once per completed run, after it has read all | ||
| its inputs and before it closes its output, on both paths; a failed run does not stamp. Every | ||
| node then ends a run at least as new as its upstreams, so nothing is stale after a completed | ||
| run, and `head()`, `as_table()` and `iter_data()` serve the run's results from memory. The | ||
| same rule governs function pod streams outside a job (F18, ITL-1052). | ||
|
|
||
| **Fix:** `async_execute` stamps the node after its run completes and before its `finally` | ||
| closes the output. On the no-database path the pod closes the writer it is given, so it gets a | ||
| writer whose close is left to the node. No hash, stored value or public API changes. | ||
| Regressions: `tests/test_pipeline/test_node_staleness_after_run.py` (sync and async, a forced | ||
| reverse completion order, in-memory and Delta stores; staleness, heads, store reads, function | ||
| calls, a second run). | ||
|
|
||
| Not changed: a `SideEffectJobNode` never stamps, on either path, so it reads as stale after a | ||
| run; it keeps no rows and nothing reads its staleness. A head with no rows still shows the | ||
| declared columns only (no `_source_*`, `_context_key` or content-hash column; decision of | ||
| November 12, 2026, recorded on ITL-985 and ITL-1002). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| ## `src/orcapod/core/function_pod.py` | ||
|
|
||
| ### F18 — A chained function pod stream was stale right after its first full read | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-1052 | ||
|
|
||
| `FunctionPodStream` stamped its modified time when it opened its input | ||
| (`_ensure_iterator`). The input of a chained pod stream is a lazy generator, so in | ||
| `s2 = g.pod(f.pod(src))` the upstream opened, and stamped, only after `s2` had: `s2` ended its | ||
| first full read older than its upstream, and its next read cleared its cache and called `g` | ||
| again on every row (6 calls, then 9). Operator streams compute eagerly when they are opened, | ||
| so a chain through an operator was not affected; a pod reading a pod directly was. | ||
|
|
||
| **Rule** (the same as for job nodes, NS1 / ITL-1042): a stream stamps its modified time when | ||
| its results are complete, that is when its input is read to the end, before its reader sees | ||
| the end; never when a read starts. `clear_cache()` stamps (a reset is a change). A failed or | ||
| partial read does not stamp. | ||
|
|
||
| **Fix:** the stamp moved from `_ensure_iterator` to the end of the input, on the sequential | ||
| and the concurrent read paths. A reset of the input during a read must not be reported as | ||
| current (the result may mix two states of the input), so: | ||
|
|
||
| - each pod stream keeps a reset generation, bumped by `clear_cache()`; | ||
| - when the first row arrives (on the concurrent path, once the input has delivered its rows, | ||
| before they are computed), the stream notes its input's state: the generation of a pod | ||
| stream input (whose modified time legitimately moves during the read, when it finishes its | ||
| own read), the modified time of any other input; | ||
| - at the end of the input it stamps only if that state is unchanged and the input has a | ||
| modified time; otherwise it clears its own modified time, which makes it stale; | ||
| - a stale pod stream reports `last_modified` as `None` (unknown), so every stream reading it, | ||
| however far down the chain, is stale too. `is_stale` compares only direct upstreams, and a | ||
| stale pod stream used to keep its old time: before this fix a reset two links up went | ||
| unnoticed by the last stream, hidden by the false staleness above. | ||
|
|
||
| The generation and the noted state are in-memory bookkeeping only: they enter no hash, no | ||
| saved file and no store. Regressions: `tests/test_core/function_pod/test_pod_stream_staleness.py` | ||
| (chains of two and three, sequential and concurrent, the cached pod stream, a chain through a | ||
| join; second reads call no function and read no store; a reset upstream recomputes; a reset | ||
| during a read, or made by the function while the stream computes, leaves the reader stale and | ||
| its next read is of one state). | ||
|
|
||
| Not changed: `FunctionPodStream.as_table` still returns its kept table when the stream is stale | ||
| (ITL-1045). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.