Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions changelog.d/ITL-1042.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
### Fixed

#### A node's preview shows its rows after an async run (ITL-1042)

After a job ran on `AsyncPipelineOrchestrator`, a function node downstream of an operator
node (for example the automatic column selection a pod call adds) was taken to be out of
date. Its first `as_table()`, `iter_data()` or `head()` read the store again instead of the
results in memory, and from then on every operator node downstream of it showed
`head()` as "not computed: inputs changed since the last computation", with no rows, although
`as_table()` and `iter_data()` returned every row. After a run on either orchestrator no node
is now out of date: `head()` shows the rows `as_table()` returns, and none of them reads the
store. Nothing was ever computed twice. (ITL-1042)

#### A chain of function pod streams is not computed again on its second read (ITL-1052)

In a chain such as `g.pod(f.pod(source))`, the last stream was taken to be out of date right
after it was read to the end, so its next `iter_data()`, `as_table()` or `head()` called `g`
again on every row (or, for a cached pod, looked every row up in its store again). A pod
stream now records that its results are complete when it has read its input to the end, so a
second read computes nothing. A change upstream (`clear_cache()`), also one made while the
chain is being read, still makes every stream after it out of date, and its next read
computes it again. (ITL-1052)
36 changes: 36 additions & 0 deletions design_issues/ITL-1042-NS1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## `src/orcapod/core/nodes/function_node.py`

### NS1 — After an async run a function node looked stale, so downstream heads said "not computed"
**Status:** resolved
**Severity:** high
**Issue:** ITL-1042

A node is stale when an upstream (or its pod) has a newer modified time than its own. The sync
`FunctionJobNode.execute` stamps the node when its run completes; `async_execute` never did.
An `OperatorJobNode` stamps itself when its async run finishes, so a function node downstream
of an operator node (most often the automatic `SelectDataColumns` that `pod(stream)` adds when
the stream has columns the function does not take, ITL-807) was stale right after an async
run. Its first `iter_data()` / `as_table()` then threw away the in-memory results, re-read the
node's whole stored table (two `get_all_records` calls) and stamped the node. That made every
operator node downstream of it stale, and their `head()` reported `not_computed` ("inputs
changed since the last computation") with no rows, although `as_table()` and `iter_data()`
still returned every row. `head()` of the function node itself read the store instead of
memory. No function was called again and no operator recomputed.

**Rule:** a job node stamps its modified time once per completed run, after it has read all
its inputs and before it closes its output, on both paths; a failed run does not stamp. Every
node then ends a run at least as new as its upstreams, so nothing is stale after a completed
run, and `head()`, `as_table()` and `iter_data()` serve the run's results from memory. The
same rule governs function pod streams outside a job (F18, ITL-1052).

**Fix:** `async_execute` stamps the node after its run completes and before its `finally`
closes the output. On the no-database path the pod closes the writer it is given, so it gets a
writer whose close is left to the node. No hash, stored value or public API changes.
Regressions: `tests/test_pipeline/test_node_staleness_after_run.py` (sync and async, a forced
reverse completion order, in-memory and Delta stores; staleness, heads, store reads, function
calls, a second run).

Not changed: a `SideEffectJobNode` never stamps, on either path, so it reads as stale after a
run; it keeps no rows and nothing reads its staleness. A head with no rows still shows the
declared columns only (no `_source_*`, `_context_key` or content-hash column; decision of
November 12, 2026, recorded on ITL-985 and ITL-1002).
44 changes: 44 additions & 0 deletions design_issues/ITL-1052-F18.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
## `src/orcapod/core/function_pod.py`

### F18 — A chained function pod stream was stale right after its first full read
**Status:** resolved
**Severity:** high
**Issue:** ITL-1052

`FunctionPodStream` stamped its modified time when it opened its input
(`_ensure_iterator`). The input of a chained pod stream is a lazy generator, so in
`s2 = g.pod(f.pod(src))` the upstream opened, and stamped, only after `s2` had: `s2` ended its
first full read older than its upstream, and its next read cleared its cache and called `g`
again on every row (6 calls, then 9). Operator streams compute eagerly when they are opened,
so a chain through an operator was not affected; a pod reading a pod directly was.

**Rule** (the same as for job nodes, NS1 / ITL-1042): a stream stamps its modified time when
its results are complete, that is when its input is read to the end, before its reader sees
the end; never when a read starts. `clear_cache()` stamps (a reset is a change). A failed or
partial read does not stamp.

**Fix:** the stamp moved from `_ensure_iterator` to the end of the input, on the sequential
and the concurrent read paths. A reset of the input during a read must not be reported as
current (the result may mix two states of the input), so:

- each pod stream keeps a reset generation, bumped by `clear_cache()`;
- when the first row arrives (on the concurrent path, once the input has delivered its rows,
before they are computed), the stream notes its input's state: the generation of a pod
stream input (whose modified time legitimately moves during the read, when it finishes its
own read), the modified time of any other input;
- at the end of the input it stamps only if that state is unchanged and the input has a
modified time; otherwise it clears its own modified time, which makes it stale;
- a stale pod stream reports `last_modified` as `None` (unknown), so every stream reading it,
however far down the chain, is stale too. `is_stale` compares only direct upstreams, and a
stale pod stream used to keep its old time: before this fix a reset two links up went
unnoticed by the last stream, hidden by the false staleness above.

The generation and the noted state are in-memory bookkeeping only: they enter no hash, no
saved file and no store. Regressions: `tests/test_core/function_pod/test_pod_stream_staleness.py`
(chains of two and three, sequential and concurrent, the cached pod stream, a chain through a
join; second reads call no function and read no store; a reset upstream recomputes; a reset
during a read, or made by the function while the stream computes, leaves the reader stale and
its next read is of one state).

Not changed: `FunctionPodStream.as_table` still returns its kept table when the stream is stale
(ITL-1045).
12 changes: 11 additions & 1 deletion docs/concepts/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,17 @@ What each kind of object shows:
A head of an operator is never the operator applied to the heads of its inputs: a group over
the first ten rows has the wrong members. So an uncomputed stream shows its columns and
"not computed" until `as_table()`, `iter_data()` or a job run computes it. An empty result
shows status `rows` with zero rows, so it is never mistaken for "not computed".
shows status `rows` with zero rows, so it is never mistaken for "not computed". A head without
rows because nothing is computed or stored yet shows the stream's declared columns: the
source-info (`_source_*`), `_context_key` and content-hash columns, which are known only from
rows, can be missing from it even when `columns` or `all_info` asks for them.

A stream or node records that its results are current when it has read its input to the end.
So after `job.run()` on either orchestrator, sync or async, every node shows the rows of that
run: `head()`, `as_table()` and `iter_data()` serve them from memory, without reading the
store (a function node whose run left no rows in memory reads its stored rows). Likewise, once a chain of streams such as `g.pod(f.pod(source))` has been read to the
end, reading it again computes nothing. A change upstream (`clear_cache()`) makes every stream
after it out of date, and its next read computes it again.

In the preview, large cells are summarised (an array as `ndarray float64 (2, 3)`, bytes as
`bytes (300 B)`, long text and long lists shortened); `head().table` keeps the full values.
Expand Down
74 changes: 72 additions & 2 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -983,6 +983,14 @@ def __init__(
self._cached_output_table: pa.Table | None = None
self._cached_content_hash_column: pa.Array | None = None

# Staleness bookkeeping, in memory only (never hashed, saved or stored). The
# stream stamps its modified time when its input is read to the end, if its
# input did not change while it was read (ITL-1042, ITL-1052): every reset
# bumps the generation, and the state of the input taken when the first row
# arrived is compared at the end.
self._reset_generation = 0
self._input_state: tuple[object, ...] | None = None

@property
def producer(self) -> PodProtocol:
return self._function_pod
Expand Down Expand Up @@ -1040,7 +1048,6 @@ def _ensure_iterator(self) -> None:
if self._needs_iterator:
self._cached_input_iterator = enumerate(self._input_stream.iter_data())
self._needs_iterator = False
self._update_modified_time()
elif self._input_interrupted:
self._cached_input_iterator = enumerate(self._input_stream.iter_data())
self._input_interrupted = False
Expand All @@ -1050,8 +1057,63 @@ def _input_raised(self) -> None:
self._cached_input_iterator = None
self._input_interrupted = True

@property
def last_modified(self) -> datetime | None:
"""When this stream's results were completed; ``None`` while they are not current.

``None`` (modification time unknown) while the stream is stale, that is while its
input or its pod changed after its results were completed, or while those results
may mix two states of its input. A stream reading this one then sees an unknown
time and is stale as well, however far down the chain it is.
"""
own_time = self._modified_time
if own_time is None:
return None
for upstream_time in (
self._input_stream.last_modified,
self._function_pod.last_modified,
):
if upstream_time is None or upstream_time > own_time:
return None
return own_time

def _current_input_state(self) -> tuple[object, ...]:
"""What tells a change of the input: its reset generation, else its modified time.

A pod stream's modified time moves when it finishes its own read, which happens
while this stream reads it; only its resets are changes. Any other input
(operator stream, source, node) is computed or fixed by the time it yields.
"""
upstream = self._input_stream
if isinstance(upstream, FunctionPodStream):
return (upstream._reset_generation,)
return (upstream.last_modified,)

def _note_input_state(self) -> None:
"""Remember the input's state when its first row (or its end) arrives."""
if self._input_state is None:
self._input_state = self._current_input_state()

def _input_read_to_end(self) -> None:
"""Stamp the completed result, or mark it stale if the input changed during the read.

A stream stamps its modified time when its results are complete, before its
reader sees the end, so it is newer than its input and older than its readers.
If the input was reset while it was read, or is itself not current (no modified
time), the result may mix two states of the input: the modified time is cleared
instead, which makes this stream, and every stream reading it, stale.
"""
self._note_input_state()
changed = self._input_state != self._current_input_state()
if changed or self._input_stream.last_modified is None:
self._set_modified_time(None)
else:
self._update_modified_time()

def clear_cache(self) -> None:
"""Discard all in-memory cached state."""
self._reset_generation += 1
self._input_state = None
self._cached_input_iterator = None
self._needs_iterator = True
self._input_interrupted = False
Expand Down Expand Up @@ -1113,14 +1175,17 @@ def _iter_data_sequential(
try:
input_index, (tag, data) = next(input_iter)
except StopIteration:
# Mark completion by releasing the iterator
# Mark completion by releasing the iterator, and stamp the result
# before the reader sees the end.
self._cached_input_iterator = None
self._input_read_to_end()
return
except BaseException:
# The input itself raised; its iterator is closed. Open it again on
# the next read instead of taking it as exhausted.
self._input_raised()
raise
self._note_input_state()
if input_index in self._cached_output_datas:
# An input opened again yields the rows this stream already cached.
continue
Expand Down Expand Up @@ -1160,6 +1225,9 @@ def _iter_data_concurrent(
# and reads them anew (an upstream stream yields its cached rows).
self._input_raised()
raise
# The input has delivered its rows (after its own refresh, if it had one): note
# its state now, before computing, so a reset during the computation is seen.
self._note_input_state()
# Submit uncached data concurrently via async_process_data. Every row that
# returns is cached; a row whose function raised stays in the input, so the
# next read computes it again, and the first error is raised after the others
Expand Down Expand Up @@ -1209,6 +1277,8 @@ async def _gather() -> list[
if i not in self._cached_output_datas
]
self._cached_input_iterator = iter(remaining) if remaining else None
# Every row is computed: stamp the result before the reader sees any row.
self._input_read_to_end()
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

# Yield every row in input order, those an earlier iteration computed included.
for i in range(len(self._cached_output_datas)):
Expand Down
34 changes: 31 additions & 3 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,26 @@ def _executor_supports_concurrent(
return executor is not None and executor.supports_concurrent_execution


class _CloseDeferredWriter:
"""A writer that forwards rows to ``output`` and leaves closing it to its owner.

A node hands it to a pod that closes the writer it is given, when the node must
still do something (stamp its modified time) before its downstream nodes see the
end. The node closes ``output`` itself afterwards.
"""

def __init__(self, output: WritableChannel[tuple[TagProtocol, DataProtocol]]):
"""Wrap ``output``."""
self._output = output

async def send(self, item: tuple[TagProtocol, DataProtocol]) -> None:
"""Send ``item`` to the wrapped output."""
await self._output.send(item)

async def close(self) -> None:
"""Do nothing: the owner closes the wrapped output."""


# ---------------------------------------------------------------------------
# FunctionNodeBase — shared base (no DB)
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -3498,13 +3518,21 @@ def create_data_logger(self, tag: Any, data: Any) -> Any:
# Commit the last rows inside a job (ITL-919); a no-op otherwise.
self._commit_at_node_end()
else:
# No-DB path: delegate dispatch entirely to the pod.
# The pod's async_execute() closes `output` in its own finally.
# No-DB path: delegate dispatch entirely to the pod. The pod closes
# the writer it is given; that close waits for the finally below, so
# the node is stamped before its downstream nodes see the end.
await self._function_pod.async_execute(
[input_channel], output, observer=ctx_obs, run_id=run_id
[input_channel],
_CloseDeferredWriter(output),
observer=ctx_obs,
run_id=run_id,
)

ctx_obs.on_node_end(node_label, node_hash)
# The run is complete: stamp the node as the sync execute() does, before
# the finally closes the output, so every downstream node, which stamps
# after it has seen that close, is newer than this one (ITL-1042).
self._update_modified_time()
except Exception as run_error:
self._commit_after_failure(run_error)
raise
Expand Down
5 changes: 4 additions & 1 deletion src/orcapod/core/streams/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -520,7 +520,10 @@ def head(
This default shows no rows: a stream class that can serve rows without computing
overrides it. The zero-row table then has the stream's columns, typed from its
declared (Python) schema, so a column's Arrow width can differ from the one the
computed stream will have (``int`` shows as ``int64``).
computed stream will have (``int`` shows as ``int64``). That zero-row table has
the declared columns only: never a ``_source_*``, ``_context_key`` or
content-hash column, whatever ``columns`` asks for, because whether such a column
exists and, for ``_source_*``, its type are known only from rows.

Args:
n: The largest number of rows to return.
Expand Down
Loading
Loading