diff --git a/.zed/rules b/.zed/rules index b801c0552..9d77d5a14 100644 --- a/.zed/rules +++ b/.zed/rules @@ -498,14 +498,14 @@ conversion. Two specializations: Stream (core/streams/arrow_table_stream.py) — immutable (Tag, Data) sequence. Key methods: output_schema(), keys(), iter_data(), as_table(). -Every stream, source and node has run() with no arguments (developer, November 13, 2026, -ITL-1081): it does the work as_table() does (a job node runs as in a job) and returns a +Every stream, source and node has run() with no positional arguments (developer, November 13, +2026, ITL-1081): it does the work as_table() does (a job node runs as in a job) and returns a StreamView of every row; the view's head/tail/sort_by_tags choose what is shown. head() and the notebook preview never compute (ITL-991, ITL-912): they never compute, poll or deliver a row (they read only what is held or stored, or a bounded table read), never call user code (a function, a predicate, a side effect) and never write. With no rows to show they learn the column types by applying operators to zero-row tables, which processes no data; an operator whose output schema is its input's -(OUTPUT_SCHEMA_IS_INPUT_SCHEMA, PolarsFilter) is not applied at all. A user-defined operator's static_process is applied to zero-row tables and must not have effects. Never give run() -arguments. +(OUTPUT_SCHEMA_IS_INPUT_SCHEMA, PolarsFilter) is not applied at all. A user-defined operator's static_process is applied to zero-row tables and must not have effects. A job node's run() takes one keyword-only +argument, load_only (ITL-1095); never give any other run() an argument. Source (core/sources/) — produces a stream from external data. ArrowTableSource is the core implementation; CSV/Delta/DataFrame/Dict/List sources all delegate to it internally. Each @@ -800,23 +800,44 @@ data. The rules: by the user is never changed. SourceStreamBuilder.build has no bare-hash default: it needs a source_id or a source_kind. - ArrowTableStream requires at least one data column; raises ValueError otherwise. -- After a run, a FunctionJobNode returns exactly the rows of that run's inputs, sync or - async (ITL-1093; a run that raised defines no rows). Read without a completed run in this - process (a fresh or loaded job), it returns every record stored under its content hash for - the current data context, not only its current inputs. A run skips inputs whose entry is - already stored. +- A job node shows its current run; node.as_source() shows everything stored for it + (developer, November 13, 2026, ITL-1095). After a run, a FunctionJobNode returns exactly + the rows of that run's inputs, sync or async (ITL-1093). Before a run in this process (a + fresh or loaded job), after a run that raised, after clear_cache(), and while stale, every + function and operator job node reads empty (zero typed rows; head() is not_computed and + points to run() and as_source()), in every cache mode: no read touches the store. A + never-run or cleared node has no modified time, so it is stale; a job node is stale also + when any function or operator job node it depends on is (core/nodes/staleness.py), and a + SourceJobNode is modified when it is bound or its bound source is. A run skips inputs + whose entry is already stored. A loaded operator's stored view raises when its run's rows + cannot be named (ITL-1043). job.run(load_only=True) computes nothing and writes nothing: function nodes + load their own stored entries for this run's inputs and skip (and count) the others, + operators run without storing, side-effect nodes pass rows on undelivered. + A job node's run(*, load_only=False) is state-driven: a node that holds a completed, + non-stale run (for a full run: not load-only, nor anything above it) runs nothing; + otherwise its non-current dependencies run first, each once, on the sync orchestrator + (pipeline/node_run.py), and the run commits as job.run() does. A failed end-of-run + commit clears every node of the run. A load-only run's rows have status + loaded_only. Only job nodes' run() takes an argument (load_only). + A loaded job (PipelineJob.load) runs only with load_only=True. + job.run()/run_async() use the same state rule and plan (node_run.plan_run): a current node is + held (not run; its rows feed its successors, both orchestrators take held rows), sources + are read only for nodes that run, a failed end-of-run commit clears only the nodes that ran, + and the summary reports held nodes (NodeRunSummary.held). REPLAY replays every logged run + of its key (ITL-1098). - Empty data → ArrowTableSource raises ValueError("Table is empty"). - A side-effect job node delivers only when it runs (developer, November 13, 2026, ITL-1091): - job.run(), node.run() (the job run's work for that node: with track_completion only - what the invocation log does not record, without it every row again), execute, - async_execute. Every read (iter_data, as_table, dataframes, flow, head, preview, a + job.run(), node.run() (state-driven, ITL-1095: a node holding a current run delivers + nothing, so clear_cache() it first to deliver again; otherwise the job run's work for that + node: with track_completion only what the invocation log does not record, without it every + row), execute, async_execute; a load-only run delivers nothing. Every read (iter_data, as_table, dataframes, flow, head, preview, a downstream node's read) returns the rows its last run in this process passed on, held in memory (SideEffectJobNode._held_rows), and delivers nothing; with none (no run, a run that raised, or stale by is_stale) a read returns zero rows. Never route a job node's read to the standalone delivery path. The standalone SideEffectPodStream still delivers on every consuming read (iter_data, as_table, dataframes, flow, run); its head() and preview deliver nothing. -- DerivedSource over a job node before run() → presents zero rows with the origin's columns +- DerivedSource over a job node with nothing stored → presents zero rows with the origin's columns (no computed records; no error), in the plain form and with tags_as_data. - Join requires non-overlapping data columns; raises InputValidationError on collision. - MergeJoin requires colliding data columns to have identical Python types; merges into sorted diff --git a/CLAUDE.md b/CLAUDE.md index c2c43bbf3..fc7d28484 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -513,14 +513,14 @@ conversion. Two specializations: **Stream** (`core/streams/arrow_table_stream.py`) — immutable (Tag, Data) sequence. Key methods: `output_schema()`, `keys()`, `iter_data()`, `as_table()`. -Every stream, source and node has `run()` with no arguments (developer, November 13, 2026, -ITL-1081): it does the work `as_table()` does (a job node runs as in a job) and returns a +Every stream, source and node has `run()` with no positional arguments (developer, November 13, +2026, ITL-1081): it does the work `as_table()` does (a job node runs as in a job) and returns a `StreamView` of every row; the view's `head`/`tail`/`sort_by_tags` choose what is shown. `head()` and the notebook preview never compute (ITL-991, ITL-912): they never compute, poll or deliver a row (they read only what is held or stored, or a bounded table read), never call user code (a function, a predicate, a side effect) and never write. With no rows to show they learn the column types by applying operators to zero-row tables, which processes no data; an operator whose output schema is its input's -(`OUTPUT_SCHEMA_IS_INPUT_SCHEMA`, `PolarsFilter`) is not applied at all. A user-defined operator's `static_process` is applied to zero-row tables and must not have effects. Never give `run()` -arguments. +(`OUTPUT_SCHEMA_IS_INPUT_SCHEMA`, `PolarsFilter`) is not applied at all. A user-defined operator's `static_process` is applied to zero-row tables and must not have effects. A job node's `run()` takes one keyword-only +argument, `load_only` (ITL-1095); never give any other `run()` an argument. **Source** (`core/sources/`) — produces a stream from external data. `ArrowTableSource` is the core implementation; CSV/Delta/DataFrame/Dict/List sources all delegate to it internally. Each @@ -842,23 +842,44 @@ data. The rules: by the user is never changed. `SourceStreamBuilder.build` has no bare-hash default: it needs a `source_id` or a `source_kind`. - `ArrowTableStream` requires at least one data column; raises `ValueError` otherwise. -- After a run, a `FunctionJobNode` returns exactly the rows of that run's inputs, sync or - async (ITL-1093; a run that raised defines no rows). Read without a completed run in this - process (a fresh or loaded job), it returns every record stored under its content hash for - the current data context, not only its current inputs. A run skips inputs whose entry is - already stored. +- A job node shows its current run; `node.as_source()` shows everything stored for it + (developer, November 13, 2026, ITL-1095). After a run, a `FunctionJobNode` returns exactly + the rows of that run's inputs, sync or async (ITL-1093). Before a run in this process (a + fresh or loaded job), after a run that raised, after `clear_cache()`, and while stale, every + function and operator job node reads empty (zero typed rows; `head()` is `not_computed` and + points to `run()` and `as_source()`), in every cache mode: no read touches the store. A + never-run or cleared node has no modified time, so it is stale; a job node is stale also + when any function or operator job node it depends on is (`core/nodes/staleness.py`), and a + `SourceJobNode` is modified when it is bound or its bound source is. A run skips inputs + whose entry is already stored. A loaded operator's stored view raises when its run's rows + cannot be named (ITL-1043). `job.run(load_only=True)` computes nothing and writes nothing: function nodes + load their own stored entries for this run's inputs and skip (and count) the others, + operators run without storing, side-effect nodes pass rows on undelivered. + A job node's `run(*, load_only=False)` is state-driven: a node that holds a completed, + non-stale run (for a full run: not load-only, nor anything above it) runs nothing; + otherwise its non-current dependencies run first, each once, on the sync orchestrator + (`pipeline/node_run.py`), and the run commits as `job.run()` does. A failed end-of-run + commit clears every node of the run. A load-only run's rows have status + `loaded_only`. Only job nodes' `run()` takes an argument (`load_only`). + A loaded job (`PipelineJob.load`) runs only with `load_only=True`. + `job.run()`/`run_async()` use the same state rule and plan (`node_run.plan_run`): a current node is + held (not run; its rows feed its successors, both orchestrators take held rows), sources + are read only for nodes that run, a failed end-of-run commit clears only the nodes that ran, + and the summary reports held nodes (`NodeRunSummary.held`). REPLAY replays every logged run + of its key (ITL-1098). - Empty data → `ArrowTableSource` raises `ValueError("Table is empty")`. - A side-effect job node delivers only when it runs (developer, November 13, 2026, ITL-1091): - `job.run()`, `node.run()` (the job run's work for that node: with `track_completion` only - what the invocation log does not record, without it every row again), `execute`, - `async_execute`. Every read (`iter_data`, `as_table`, dataframes, `flow`, `head`, preview, a + `job.run()`, `node.run()` (state-driven, ITL-1095: a node holding a current run delivers + nothing, so `clear_cache()` it first to deliver again; otherwise the job run's work for that + node: with `track_completion` only what the invocation log does not record, without it every + row), `execute`, `async_execute`; a load-only run delivers nothing. Every read (`iter_data`, `as_table`, dataframes, `flow`, `head`, preview, a downstream node's read) returns the rows its last run in this process passed on, held in memory (`SideEffectJobNode._held_rows`), and delivers nothing; with none (no run, a run that raised, or stale by `is_stale`) a read returns zero rows. Never route a job node's read to the standalone delivery path. The standalone `SideEffectPodStream` still delivers on every consuming read (`iter_data`, `as_table`, dataframes, `flow`, `run`); its `head()` and preview deliver nothing. -- `DerivedSource` over a job node before `run()` → presents zero rows with the origin's columns +- `DerivedSource` over a job node with nothing stored → presents zero rows with the origin's columns (no computed records; no error), in the plain form and with `tags_as_data`. - Join requires non-overlapping data columns; raises `InputValidationError` on collision. - MergeJoin requires colliding data columns to have identical Python types; merges into sorted diff --git a/changelog.d/ITL-1095.md b/changelog.d/ITL-1095.md new file mode 100644 index 000000000..3d8413b61 --- /dev/null +++ b/changelog.d/ITL-1095.md @@ -0,0 +1,21 @@ +### Breaking changes and migration + +- A job node shows the current run; `node.as_source()` shows everything stored for it (developer, November 13, 2026; ITL-1095). Before a run in this process, a function or operator job node is empty: `as_table()`, `iter_data()`, the dataframe methods, `flow()`, `head()` and the notebook preview return zero rows with the node's columns, and `head()` (status `not_computed`) says the node has not run and points to `run()` and to `node.as_source()`. Before, a function node built over a store that already held results, or loaded with `PipelineJob.load`, returned every row stored under its key, and an operator node in `CacheMode.REPLAY` returned every logged run. The same holds after `clear_cache()`, after a run that raised, and while the node is stale. Migration: read stored results with `node.as_source()` (or `get_all_records()`), or run the job first; a run over a store that holds every result computes nothing. +- A function or operator job node that has not run in this process is stale (`is_stale` is `True`; it has no modified time), and a job node is stale also when any function or operator job node it depends on is. Binding a source to a job's source slot (`job.bind(sources=...)`) marks the slot modified, so every node fed by it is stale and empty until it runs again. +- A loaded operator node no longer raises from `as_table()` or `iter_data()`: it is empty, like every node that has not run. Its stored view (`as_source()`, `get_all_records()`) raises instead when the job file does not record its run's content hash and the store holds nothing under the hash the loaded job computes (ITL-1043). +- A job node's `run()` is state-driven and runs what the node depends on (developer, November 13, 2026; ITL-1095). A node that holds a completed run and is not stale runs nothing: `run()` returns the rows it holds. Otherwise every job node it depends on that holds no current run runs first, each once, as `job.run()` would run that part of the job, then the node; the run's writes are committed when it ends (before, `node.run()` read an upstream node's stored rows, or nothing, and left its last pipeline record and a `LOG` operator's output uncommitted). `run()` of a node whose function pod is a placeholder, or that was loaded without one, raises unless `load_only=True`; a `CACHE_ONLY` node's `run()` raises (its input is unavailable): read its stored results with `as_source()`. A job node's `run()` takes one keyword-only argument, `load_only`; every other stream's `run()` still takes none. +- A side-effect job node follows the same rule: it is stale before a run, and while it holds a current run neither its `run()`, nor a downstream node's `run()`, nor `job.run()` delivers anything; to deliver again without `track_completion`, call `clear_cache()` on it first. A load-only run passes its rows on and delivers nothing. +- `job.run()` and `job.run_async()` follow the same rule as `node.run()` (developer, November 14, 2026): a node that holds a current run is not run again, and its rows feed the nodes after it. Running an unchanged job a second time now computes nothing, reads and writes nothing in the store, does not read its sources and delivers no side effect, even without `track_completion` (before, every `job.run()` re-read every source and re-processed every node, and delivered every untracked side effect again). Nodes run when they have not run in this process, were cleared, are below a cleared node or a node that ran again, or below a source bound again or whose own modified time moved. After this change a re-run does not notice new data in a source whose object is unchanged (a file or table changed in place, a polling source with new rows; a start-of-run check for new data is ITL-1102). A source that holds the data it read when built (in-memory, CSV, Delta, database tables, a pinned `SpiralDBTableSource`) needs a new source object, bound with `job.bind(sources=...)`; for a `PollingSource`, `clear_cache()` on each node that reads it makes the next run poll it. The summary reports a node that was not run as held: "rowwise: 6 rows: held from the previous run" (`NodeRunSummary.held`). If the commit at the end of a run fails, the nodes it ran are cleared and the nodes it held keep their rows. +- If a run raises or is interrupted, or the commit at the end of `job.run()` or `node.run()` fails, every node the run ran is cleared (not run, stale) before the error propagates; nodes it held keep their rows. A run that left failed rows (`error_policy="continue"`) is never current: the next run computes only the failed rows and reads the rest from the store. + +### Added + +- `job.run(load_only=True)` and `await job.run_async(load_only=True)` fill a job's nodes with what is already stored for this run's inputs and compute nothing (ITL-1095): sources are read, operators run on the rows that reach them (a `CacheMode.LOG` cache writes nothing), each function node takes the stored results of its inputs and skips the inputs without one, a side-effect node passes its rows on without delivering anything, and nothing is written to the store. `RunSummary.load_only`, `NodeRunSummary.not_stored` (inputs skipped) and `NodeRunSummary.inputs_incomplete` (the function nodes above a node that skipped inputs) report it; loaded rows count in `reused`, and the summary's text says "3 loaded, 1 not stored (skipped)". + +- `node.run(load_only=True)`, and `node.last_run`, the `RunSummary` of a node's last `run()`. `ViewStatus.LOADED_ONLY`: the `head()` and `run()` view of a node filled by a load-only run, captioned "N loaded, M inputs without a stored result" and naming the function nodes above it that skipped inputs. `holds_current_run(load_only=...)` on function and operator job nodes says whether `run()` would run them. + +- A job loaded with `PipelineJob.load` can be filled from its store: `loaded.run(load_only=True)` (and `node.run(load_only=True)` of a loaded node) reads the job's sources (rebuilt from files, or bound with `loaded.bind(sources=...)`), runs the rebuilt operators and gives each function node the results its store holds for those inputs; nodes below an unbound source stay empty. `loaded.run()` without `load_only` still raises, and says how to load. + +### Fixed + +- On the asynchronous orchestrator, an operator node in `CacheMode.REPLAY` stamps itself after its inputs close, so the rows it replayed are not judged out of date by the time the run ends. diff --git a/design_issues/ITL-1095-CR1.md b/design_issues/ITL-1095-CR1.md new file mode 100644 index 000000000..ff78f3191 --- /dev/null +++ b/design_issues/ITL-1095-CR1.md @@ -0,0 +1,28 @@ +## `src/orcapod/core/nodes/function_node.py` + +### CR1 — A job node read without a run showed every stored row, not the current run +**Status:** in progress +**Severity:** high +**Issue:** ITL-1095 + +A job node is read through the same methods whether or not it has run in this process. After +a run it held the rows of that run's inputs (ITL-1093), but read without one it fell back to the +store: a `FunctionJobNode` hot-loaded every record stored under its content hash (the results of +every earlier run over the same `source_id`, including inputs no longer in the source), a loaded +job's function nodes did the same, and an `OperatorJobNode` in `CacheMode.REPLAY` returned every +logged run. Which view a read gave depended on whether a run had happened in the process, and +nothing said so. A fresh or cleared node also reported `is_stale == False`, and staleness looked +one level up only, so a source rebound two nodes up left the nodes below it "current". + +**Fix:** (developer, November 13, 2026) a job node shows its current run and `node.as_source()` +shows everything stored. Function and operator job nodes read empty (typed zero rows, `head()` +`not_computed` pointing to `run()` and `as_source()`) until a run in this process, after +`clear_cache()`, after a run that raised and while stale; no read touches the store. A never-run +or cleared node has no modified time (stale); `core/nodes/staleness.py` makes a job node stale +when any function or operator job node above it is; a `SourceJobNode` is modified when bound or +when its bound source is. `job.run(load_only=True)` fills the nodes from what is stored and +computes and writes nothing. A job node's `run(*, load_only=False)` is state-driven: a current +node runs nothing; otherwise its non-current dependencies run first, once each, on the sync +orchestrator restricted to them (`pipeline/node_run.py`), and the run commits as `job.run()` +does; a failed end-of-run commit clears every node of the run. A load-only run's rows have +status `loaded_only` and are never current for a full run. diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index c48580d1e..556ef5b56 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -332,12 +332,20 @@ database. Each recorded step becomes a node, available afterwards as `job.nodes[ listed under its `source_id`). A node is a stream, so `as_table()`, `iter_data()` and `output_schema()` all work on it. -After `job.run()`, a function node returns exactly the rows of that run's inputs, on either -orchestrator. The database may hold more under the same node: results of an earlier run -over other rows of the source stay stored and are reused when those rows come back, but -they are not rows of this run. A run that raised defines no rows. A node read with no -finished run in this process (a job built but not run, or a loaded job) returns every row -stored under its key. +A job node shows the current run; `node.as_source()` shows everything stored for it +(developer, November 13, 2026). After `job.run()`, a function node returns exactly the rows +of that run's inputs, on either orchestrator. The database may hold more under the same +node: results of an earlier run over other rows of the source stay stored and are reused +when those rows come back, but they are not rows of this run. Before a run in this process +(a job built over a store that already holds results, or a loaded job) every node is empty: +`as_table()`, `iter_data()`, `head()` and the preview show no rows, and the preview says the +node has not run and points to `run()` and to `as_source()`. A run fills it, also when every +result is read from the store and nothing is computed. A run that raised defines no rows, +and so does `clear_cache()`; a node whose input changed since its run (an upstream node ran +again, or a source was bound again) is stale and empty until it runs again. The stored view, +`node.as_source()`, reads every record stored under the node's key, whether or not the node +has run: the node's stored history, which keeps an entry's earlier result next to its +recomputed one, where a run loads only the latest. `InMemoryArrowDatabase` lasts only as long as the Python process. Use `DeltaTableDatabase("some/directory")` from `orcapod.databases` to keep results on disk @@ -533,6 +541,92 @@ print(summary.ok, summary.failed_rows) # False 3 ``` +### Loading without computing: `run(load_only=True)` + +A job node shows the rows of its current run, so a job built over a store that already holds +its results is empty until it runs. A run that finds every result stored computes nothing, +but it does compute the results that are missing. `job.run(load_only=True)` fills the nodes +with what is already stored for this run's inputs and never computes: sources are read, +operators run on the rows that reach them, each function node takes the stored results of +its inputs and skips the inputs that have none, no side effect is delivered and nothing is +written to the store. The summary counts what each node loaded and skipped: + +```python +more_mice = mice + [{"subject_id": "mouse_04", "weight": 30.0, "height": 0.12}] + +load_job = PipelineJob(name="bmi_demo", database=db) +with load_job: + source = DictSource(data=more_mice, tag_columns=["subject_id"], source_id="mice") + compute_bmi.pod(source, label="bmi") + +print(load_job.nodes["bmi"].as_table().num_rows) # not run in this process: empty +# 0 +calls.clear() +summary = load_job.run(load_only=True) +print(summary) +# Load-only run of job 'bmi_demo': 1 row without a stored result skipped. +# bmi: 4 rows: 3 loaded, 1 not stored (skipped) +print(load_job.nodes["bmi"].as_table().num_rows, "rows; function calls:", len(calls)) +# 3 rows; function calls: 0 +``` + +`summary.nodes["bmi"].not_stored` is the number of skipped inputs, and +`summary.nodes[label].inputs_incomplete` names the function nodes above a node that skipped +inputs. A function node passes on only the rows it loaded, so the rows below it are +incomplete: a group formed after it may lack members, and the result stored for a smaller +group by an earlier run may then be loaded for it. `run_async(load_only=True)` does the same +on the asynchronous orchestrator. + +### Running again: what a run runs + +`job.run()`, `await job.run_async()` and `job.nodes[label].run()` decide what to run from +each node's state (developer, November 13 and 14, 2026). `node.run()` runs that node +together with the job nodes it depends on and returns a view of its rows; `job.run()` does +the same for every node of the job. + +| State of a node | What it shows | A run | A load-only run | +|---|---|---|---| +| Not run in this process (a fresh or loaded job), cleared, or stale | nothing: `head()` says "not run" (`not_computed`) | runs it | loads it | +| Filled by a load-only run | the rows it loaded: status `loaded_only`, "N loaded, M inputs without a stored result" | runs it, computing what it skipped | holds it | +| Filled by a run | that run's rows (`rows`) | holds it | holds it | + +A node that is held is not run: its rows feed the nodes after it, and nothing is computed, +read from the store, written or delivered for it. So running an unchanged job again does +nothing, also on the asynchronous orchestrator, and the summary says so per node: +"rowwise: 6 rows: held from the previous run" (`summary.nodes["rowwise"].held`). A load-only +run's rows are never held by a full run, nor are the rows of any node below them. A source +is read only when a node that runs reads it. + +A node whose last run left failed rows (with the default `error_policy="continue"`) is not +held either: the next run computes only the failed rows and reads the rest from the store, +so running the job again after fixing a transient failure retries exactly those rows. + +A node is stale, and runs, when: + +- it has not run in this process, its last run raised, or the commit at the end of its run + failed; +- `clear_cache()` was called on it, or on any job node above it; +- a job node above it ran again; +- its source was bound again (`job.bind(sources=...)`), or the bound source's own modified + time moved (a `CachedSource` that read new rows, a `PollingSource` that polled). + +Staleness comes from modified times only, so a re-run does not notice new data in a source +whose object is unchanged (a start-of-run check for new data is planned, ITL-1102). Most +sources hold the data they read when they were built: an in-memory source (a `DictSource` +copies the list it is given), a CSV, Delta or database table source, and a +`SpiralDBTableSource` pinned to the table version it found. For those, build a new source +object and bind it (`job.bind(sources=...)`): every node that reads it becomes stale and +runs; clearing the nodes below would only read the same data again. A `PollingSource` polls +when it is read, so `clear_cache()` on each node that reads it directly (one per branch) +makes the next run poll it and process its new rows. To deliver a side effect again (with +`track_completion=False`), call `clear_cache()` on its node first. + +`node.last_run` holds the counts of a `node.run()`, as `job.last_run` does for `job.run()`, +and a run's writes are committed when it ends; when everything is held there is nothing to +commit. If that commit fails, the nodes the run ran are cleared (they read as not run); the +nodes it held keep their rows. A held node fires no observer hook and cannot fail, so the +orchestrator's `error_policy` has nothing to apply to it. + ### Stopping at the first failed row Pass an orchestrator with `error_policy="fail_fast"` to stop the run at the first row whose @@ -568,9 +662,9 @@ has finished. When rows of several nodes fail before the run stops, one error is the others are noted on it (`error.__notes__`): a store error first (see "When the store cannot be written" below), then a `TypeIdentityError`, and otherwise the error of the node that comes first in the pipeline. A node cancelled by the failure commits the results it -wrote before it stops. A run that raised defines no rows: afterwards the failing node, and -every node the run cancelled, reads as a node with no finished run, every row stored under -its key. +wrote before it stops. A run that raised defines no rows: afterwards every node the run ran, +the failing node and every node the run cancelled included, reads as a node that has not +run (empty; its stored rows are read with `as_source()`), and the next run runs it again. What is stored after such a run: @@ -650,6 +744,9 @@ keeps that output in memory and, with the default `cache_mode=OFF`, stores nothi `get_all_records()` returns `None` when no matching records exist; records from an earlier LOG run remain readable after switching to OFF. Calling `clear_cache()` on it discards the output, and it then reads as a zero-row table, as before the first run, until the job runs again. +An operator node attached in `CacheMode.REPLAY` does not compute: its run replays the rows +stored under its key by `CacheMode.LOG` runs, every logged run of that key, not only the +latest; before its run it shows nothing, like every node (a narrower replay is ITL-1098). ### When the store cannot be written @@ -784,37 +881,52 @@ with job: job.run() job.save(workdir / "bmi_job.json") -loaded = PipelineJob.load(workdir / "bmi_job.json") -print(loaded.nodes["bmi"].as_table().sort_by("subject_id").to_pandas()) +loaded = PipelineJob.load(workdir / "bmi_job.json") # its nodes are empty until it runs +print(loaded.nodes["bmi"].as_table().num_rows) # not run in this process: empty +# 0 +print(loaded.nodes["bmi"].as_source().as_table().sort_by("subject_id").to_pandas()) # subject_id bmi # 0 mouse_01 1756.944444 # 1 mouse_02 2210.000000 # 2 mouse_03 2297.520661 ``` -A loaded job is for reading stored results. It cannot be run again: function pods are not -saved with the job, so `loaded.run()` raises. To compute again, build the job from your code -as before. +A loaded job is for reading stored results. Its nodes have not run in this process, so +they are empty; each node's stored view, `node.as_source()`, reads what the store holds. A +loaded job cannot compute: function pods are not saved with it, so `loaded.run()` raises. +`loaded.run(load_only=True)` fills its nodes from the store instead, as for any job (see +*Loading without computing* above): the sources are read (an in-memory source must be bound +first, since its data is not saved), the rebuilt operators run, and each function node takes +the results stored for those inputs. Nodes below a source that is not available stay empty. +`node.run(load_only=True)` does the same for one node and what it depends on. + +```python +loaded.bind(sources={"mice": DictSource(data=mice, tag_columns=["subject_id"], source_id="mice")}) +summary = loaded.run(load_only=True) +print(loaded.nodes["bmi"].as_table().num_rows, summary.nodes["bmi"].not_stored) +# 3 0 +``` + +To compute again, build the job from your code as before. -The store is the source of truth: a loaded job shows what the store holds for it. The job -file names each run's rows by the content hash the run stored them under, so a loaded node -reads exactly what the same node built from your code would read from that store, and no -rows under that key is an empty result, not an error. +The store is the source of truth: a loaded job's stored views show what the store holds for +it. The job file names each run's rows by the content hash the run stored them under, so a +loaded node's stored view reads exactly what the same node built from your code would read +from that store, and no rows under that key is an empty result, not an error. -- A function node's `as_table()`, `iter_data()` and `get_all_records()` return the rows - stored under the saved run's key, with the types they were stored with: those of every - run of that node over the same sources, not only the saved run's inputs. Rows come in the order the store - returns them; sort them if order matters. +- A function node's `as_source()` and `get_all_records()` return the rows stored under the + saved run's key, with the types they were stored with: those of every run of that node + over the same sources, not only the saved run's inputs. Rows come in the order the store + returns them; sort them if order matters. The node itself (`as_table()`, `iter_data()`, + `head()`) is empty: it has not run in this process. - Operators (a join, a filter) store outputs only when caching is enabled, under `odb_v1`. The saved file does not record an operator's cache mode (follow-up ITL-977), so attach a loaded operator node in `CacheMode.REPLAY` to read the rows its run stored: `loaded.nodes["joined"].attach_databases(pipeline_database=store.at(""), - cache_mode=CacheMode.REPLAY)`. A loaded operator node reads exactly as the same node built - from your code and attached in `CacheMode.REPLAY`; when the store holds no rows under the - node's key (no table, or none of its rows), the answer is an empty result, as for that - node. A loaded operator not attached in `REPLAY` has no readable cache and raises from - `as_table()` and `iter_data()`; read its input nodes instead (an in-memory input source - must be bound first, see below), or build the job from your code. + cache_mode=CacheMode.REPLAY)`, then read its stored view, `as_source()`, which reads + exactly as the same node built from your code would; when the store holds no rows under + the node's key (no table, or none of its rows), the answer is an empty result, as for that + node. A loaded operator node itself is empty, like every node that has not run. - A loaded node has the pipeline hash, content hash, table path and output schema its run had (a source node whose data was not saved gets its run's content hash once its source is bound). The file records which columns may be null (`T | None`), because the Arrow type text @@ -823,8 +935,8 @@ rows under that key is an empty result, not an error. file's format version: a file without them (format `0.1.0`, or a `0.2.0` file written before they existed) still loads, but lossy: its schemas read every column as non-null, and a loaded operator node cannot name its run's rows. When the store holds none under the - content hash the loaded job computes, `as_table()` and `iter_data()` raise, and `head()` - shows no rows and says why, rather than presenting stored rows as none. An empty record is + content hash the loaded job computes, its stored view (`as_source()`, `get_all_records()`) + raises and says why, rather than presenting stored rows as none. An empty record is not a missing one: it says that no column may be null. Saving such a loaded job again does not record what its file did not. Upgrading such files is left to a migration tool after v0.3. @@ -837,9 +949,9 @@ rows under that key is an empty result, not an error. `PipelineJob.load(path, database=...)` if the store has moved. - The job file may also hold an optional, advisory section, `run.expected_rows`: for each node, how many rows the store held under its run's key when the job was saved (written - only when that is certain: after a run that finished). Reading a loaded node always - returns what the store holds; when the store holds another number of rows, the read - issues one `StoreContentWarning` per node, naming the node, both numbers and the store: + only when that is certain: after a run that finished). Reading a loaded node's stored + view always returns what the store holds; when the store holds another number of rows, + the read issues one `StoreContentWarning` per node, naming the node, both numbers and the store: the store may not be the one the run wrote, or it may have changed since. Deleting the section from the file changes nothing but this check. diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index 83d4a8319..b24d1fc6f 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -167,8 +167,8 @@ names them. To see more rows, use `as_df()` with a `limit=` or `where=` source. **Restrictions are not part of identity.** `where`, `limit` and `asof` choose rows of the same table, so two sources on one table have the same `source_id` and `content_hash` whatever their restrictions, and a row's result is computed once and shared. One -consequence: with a shared store, reading a job's node back (`get_all_records()`, a loaded -job, and `as_table()` after an async run) returns every row that any job with the same +consequence: with a shared store, reading a job node's stored records back +(`get_all_records()`, `node.as_source()`) returns every row that any job with the same pipeline has read from that table, not only the rows of the latest restriction. The rows sent downstream during a run are only the rows that run read. diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index e6ff8356a..317fc5f11 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -215,8 +215,8 @@ What `run()` does on each kind of object: | A `PollingSource` | polls, as `as_table()` does | | A `CachedSource` | reads its cache and writes the wrapped source's new rows to it | | A function pod's or operator's output stream | computes what is not computed yet | -| A job node | runs the node as a job run would (`as_table()` of a job node only reads), then returns the view | -| A job's side-effect node | runs the node as `job.run()` does: delivers what its invocation log does not record (every row again without `track_completion`), then returns the rows it passed on; every read after it only reads | +| A job node | runs the node as a job run would, after every job node it depends on that holds no current run (each once), commits, and returns the view of that run's rows; a node that holds a current run (completed in this process and not stale) runs nothing. `run(load_only=True)` loads stored results only. `as_table()` of a job node only reads | +| A job's side-effect node | runs as a job node does (row above): delivers what its invocation log does not record (every row again without `track_completion`), then returns the rows it passed on; a node that holds a current run delivers nothing (`clear_cache()` it first to deliver again); every read only reads | | A side-effect stream outside a job | delivers its side effects for every row, on every call, and returns the rows it passed on; `head()` still shows none | | A pipeline blueprint node, an unbound input slot | raises, as `as_table()` does | @@ -226,7 +226,7 @@ What each kind of object shows: |---|---| | A source | its first rows (a bounded read for a source that reads its table lazily) | | An operator's or function pod's output stream | the rows already computed, or none with status `not_computed` | -| A job node | the rows it holds after `job.run()`, else the first of its stored rows, else none (`nothing_stored`) | +| A job node | the rows of its current run (after `job.run()` or `node.run()`); before a run in this process, or once stale, none (`not_computed`), pointing to `run()` and to `node.as_source()`, the view of every stored result | | A pipeline blueprint node, an unbound input slot | none (`unavailable`) | | A job's side-effect node | the rows its last run passed on, else none (`not_computed`); it never delivers, and neither do its `as_table()`, `iter_data()` and other reads | | A side-effect stream outside a job | none: showing it never delivers its side effects | @@ -252,7 +252,7 @@ print(unordered.head(2, all_info=True).as_table().column("subject_id").to_pylist # ['mouse_01', 'mouse_02'] ``` -A stream that has all of its rows (a job node's stored rows, a computed stream, an in-memory +A stream that has all of its rows (a job node after its run, a computed stream, an in-memory source) then gives exactly the first `n` rows of `as_table` with the same configuration, whatever order the rows were computed in. A stream that has only some of its rows sorts those: a source that reads a bounded number of rows (such as a `SpiralDBTableSource`) shows @@ -272,7 +272,8 @@ rows, can be missing from it even when `columns` or `all_info` asks for them. A stream or node records that its results are current when it has read its input to the end. So after `job.run()` on either orchestrator, sync or async, every node shows the rows of that run: `head()`, `as_table()` and `iter_data()` serve them from memory, without reading the -store (a function node whose run left no rows in memory reads its stored rows). Likewise, once a chain of streams such as `g.pod(f.pod(source))` has been read to the +store. Before its run in this process a job node holds nothing and shows no rows, whatever +its store holds; `node.as_source()` reads the store. Likewise, once a chain of streams such as `g.pod(f.pod(source))` has been read to the end, reading it again computes nothing. A change upstream (`clear_cache()`) makes every stream after it out of date, and its next read computes it again. Two reads of one stream open at the same time, or a read that picks up one left partway, share diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index fda08a165..7c695088a 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -18,7 +18,9 @@ import asyncio import contextlib +import dataclasses import logging +import sys import uuid from abc import abstractmethod from collections import Counter @@ -38,10 +40,13 @@ ) from orcapod.core.streams.view import ( BLUEPRINT_NOTE, + JOB_NODE_NOT_RUN_NOTE, + STALE_NOTE, ViewStatus, StreamView, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER +from orcapod.core.nodes.staleness import RunHoldingNode from orcapod.core.datagrams.tag_data import Data, EmptyData, Tag from orcapod.errors import ( EphemeralResultMissingError, @@ -275,6 +280,44 @@ async def close(self) -> None: # --------------------------------------------------------------------------- +def run_job_node( + node: Any, + *, + load_only: bool, + run_alone: Callable[[], Any], + caller: Any, +) -> None: + """Run a job node as its ``run()`` does (developer, November 13, 2026; ITL-1095). + + Nothing runs when the node holds a current run (``holds_current_run``). Otherwise + a node that depends on job nodes runs with them (``run_with_dependencies``) and one + built over streams outside a job runs alone (``run_alone``). The run's summary, if + it gives one, is kept as the node's ``last_run``, and failed rows warn at the + caller's line, as ``job.run()`` warns. + + Args: + node: The job node (a ``RunHoldingNode``). + load_only: Whether to load stored results only. + run_alone: Runs the node over its input when no job node feeds it; returns its + ``RunSummary`` or ``None``. + caller: The frame of the ``run()`` caller. + """ + from orcapod.pipeline.node_run import depends_on_job_nodes, run_with_dependencies + from orcapod.pipeline.run_summary import warn_row_failures + + if node.holds_current_run(load_only=load_only): + return + if depends_on_job_nodes(node): + summary = run_with_dependencies(node, load_only=load_only) + else: + summary = run_alone() + if summary is None: + return + node._last_run = summary + if not summary.ok: + warn_row_failures(summary, caller) + + class FunctionNodeBase(StreamBase): """Shared base for ``FunctionNode`` and ``FunctionJobNode``. @@ -585,7 +628,7 @@ def as_table( # hashes are then built from the same rows, and they are the current ones # (ITL-1045). ``iter_data`` clears a stale cache too, but only after the # cached table would already have been read. - if self.is_stale: + if self._read_finds_stale(): self.clear_cache() if self._cached_output_table is None: self._cached_output_table = self._table_from_pairs(list(self.iter_data())) @@ -604,6 +647,10 @@ def all_content_hashes() -> "pa.Array[Any]": self._cached_output_table, column_config, all_content_hashes ) + def _read_finds_stale(self) -> bool: + """Whether a read finds the rows this node holds out of date.""" + return self.is_stale + def _zero_row_output_table(self) -> "pa.Table | None": """This node's rows' table with no row; see ``function_zero_row_table``.""" from orcapod.core.function_pod import function_zero_row_table @@ -1074,7 +1121,7 @@ class _JoinedRecords(NamedTuple): empty_taginfo_rows: dict[bytes, dict] -class FunctionJobNode(FunctionNodeBase): +class FunctionJobNode(RunHoldingNode, FunctionNodeBase): """DB-backed execution node for function pod invocations. Created by ``PipelineJob`` at run time; never recorded inside a plain @@ -1101,6 +1148,13 @@ class FunctionJobNode(FunctionNodeBase): # its number of inputs; None when no run completed (ITL-1093). A class default so # nodes rebuilt from a saved descriptor start without a run. _run_entry_counts: Counter[bytes] | None = None + # Whether the last completed run was load-only, and how many of its inputs had no + # stored result (ITL-1095). Class defaults for nodes rebuilt from a descriptor. + _run_load_only: bool = False + _run_not_stored: int = 0 + # Rows of the last completed run whose function raised (error_policy "continue"): + # such a run is never current, so the next run retries them (ITL-1095). + _run_failed: int = 0 def __init__( self, @@ -1159,6 +1213,8 @@ def __init__( result_database=result_database, ephemeral_database=ephemeral_database, ) + # A node that has not run in this process holds nothing and is stale (ITL-1095). + self._set_modified_time(None) # ------------------------------------------------------------------ # node_config property @@ -1555,14 +1611,19 @@ def from_descriptor( def clear_cache(self) -> None: """Clear in-memory output caches, content hash cache, and node identity path cache. - The node then holds no run: it reads as a node with no completed run in this - process (every row stored under its content hash). + The node then holds no run: it is stale and reads empty, as before its first run + in this process (ITL-1095). Its stored results stay stored (``as_source()``). """ super().clear_cache() self._cached_output_datas.clear() self._cached_output_table = None self._cached_content_hash_column = None self._run_entry_counts = None + self._run_load_only = False + self._run_not_stored = 0 + self._run_failed = 0 + # No content established: the node, and every node downstream, is stale. + self._set_modified_time(None) def _held_pairs(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Yield the (tag, data) rows this node holds; every read of them goes through here. @@ -1571,27 +1632,210 @@ def _held_pairs(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: (two equal inputs give two rows, as downstream received them). The in-memory cache may hold more (results of other inputs, read from the store), which stay cached for reuse but are not rows of this run (ITL-1093). Without a completed - run in this process, every held row. Rows come in the order the cache holds - them. An input without output (an inactive function) is not a row. + run in this process there are no rows, whatever the cache or the store holds + (ITL-1095). Rows come in the order the cache holds them. An input without output + (an inactive function) is not a row. """ - held = self._cached_output_datas counts = self._run_entry_counts if counts is None: - pairs: Iterable[tuple[TagProtocol, DataProtocol | None]] = held.values() - else: - pairs = ( - pair for eid, pair in held.items() for _ in range(counts.get(eid, 0)) - ) + return + pairs: Iterable[tuple[TagProtocol, DataProtocol | None]] = ( + pair + for eid, pair in self._cached_output_datas.items() + for _ in range(counts.get(eid, 0)) + ) for tag, data in pairs: if data is not None: yield tag, data - def _set_run_rows(self, entry_ids: list[bytes]) -> None: - """Make the inputs of a completed run, by base entry id, the rows this node holds.""" + def _set_run_rows( + self, + entry_ids: list[bytes], + *, + load_only: bool = False, + not_stored: int = 0, + failed: int = 0, + ) -> None: + """Make the inputs of a completed run, by base entry id, the rows this node holds. + + Args: + entry_ids: The base entry ids of the run's inputs whose rows the node holds. + load_only: Whether the run was load-only (ITL-1095). + not_stored: How many inputs of a load-only run had no stored result. + failed: How many rows of the run failed (error policy ``"continue"``). + """ self._run_entry_counts = Counter(entry_ids) + self._run_load_only = load_only + self._run_not_stored = not_stored + self._run_failed = failed self._cached_output_table = None self._cached_content_hash_column = None + def _read_finds_stale(self) -> bool: + """Whether a read finds the rows held out of date; never for ``CACHE_ONLY``, + whose input is unavailable (``head`` and ``iter_data`` judge it the same way).""" + from orcapod.pipeline.serialization import LoadStatus + + return self.load_status != LoadStatus.CACHE_ONLY and self.is_stale + + def _holds_run(self, *, load_only: bool) -> bool: + """Whether the node holds a completed run that a run need not redo. + + A load-only run's rows are good only for another load-only run. A run that left + failed rows is never good: the next run retries them, reading the rest from the + store (ITL-1095). + """ + return ( + self._run_entry_counts is not None + and self._run_failed == 0 + and (load_only or not self._run_load_only) + ) + + @property + def last_run_not_stored(self) -> int: + """Inputs the node's last completed run skipped because nothing was stored. + + Non-zero only after a load-only run (``run(load_only=True)``); 0 before a run. + """ + return self._run_not_stored + + def _stored_entry( + self, entry_id: bytes + ) -> tuple[TagProtocol, DataProtocol | None] | None: + """The result held for ``entry_id`` after a lookup, or ``None`` if none is stored. + + A placeholder for a result kept in an ephemeral store that is gone is not a + stored result. + """ + entry = self._cached_output_datas.get(entry_id) + if entry is None or isinstance(entry[1], EmptyData): + return None + return entry + + def _execute_load_only( + self, + input_stream: StreamProtocol, + observer: ExecutionObserverProtocol | None, + ) -> list[tuple[TagProtocol, DataProtocol]]: + """Fill the node with the stored results of ``input_stream``'s rows; compute nothing. + + The load-only form of ``execute`` (ITL-1095): each input's entry is looked up + among the node's own stored entries (its pipeline records under its content + hash, the lookup a run makes before computing). An input with a stored result + is a loaded row (``on_data_start`` and ``on_data_end(cached=True)``); one + without is skipped and counted (``last_run_not_stored``), with no observer + call. No function is called and nothing is written. + + Returns: + The loaded (tag, output data) rows, excluding ``None`` outputs. + + Raises: + RuntimeError: If no store is attached (``UNAVAILABLE``). + InputValidationError: If the input stream is of another data context. + """ + from orcapod.pipeline.observer import NoOpObserver + from orcapod.pipeline.serialization import LoadStatus + + node_label = self.label + node_hash = self.content_hash().to_string() + obs = observer if observer is not None else NoOpObserver() + ctx_obs = obs.contextualize(*self.node_identity_path) + output: list[tuple[TagProtocol, DataProtocol]] = [] + loaded_ids: list[bytes] = [] + not_stored = 0 + try: + if self.load_status == LoadStatus.UNAVAILABLE: + raise self._no_store_error() + require_same_data_context( + input_stream.data_context_key, self._stored_data_context_key + ) + tag_schema = input_stream.output_schema(columns={"system_tags": True})[0] + ctx_obs.on_node_start(node_label, node_hash, tag_schema=tag_schema) + inputs = [ + (tag, data, self.compute_base_entry_id(tag, data)) + for tag, data in input_stream.iter_data() + ] + _ = self.get_cached_results(base_entry_ids=[eid for _, _, eid in inputs]) + for tag, data, entry_id in inputs: + entry = self._stored_entry(entry_id) + if entry is None: + not_stored += 1 + continue + ctx_obs.on_data_start(node_label, tag, data) + ctx_obs.on_data_end(node_label, tag, data, entry[1], cached=True) + loaded_ids.append(entry_id) + if entry[1] is not None: + output.append((entry[0], entry[1])) + self._set_run_rows(loaded_ids, load_only=True, not_stored=not_stored) + ctx_obs.on_node_end(node_label, node_hash) + self._update_modified_time() + except BaseException: + # A run that raised (an observer at the node's end included) defines no rows. + self.clear_cache() + raise + return output + + async def _async_execute_load_only( + self, + input_channel: ReadableChannel[tuple[TagProtocol, DataProtocol]], + output: WritableChannel[tuple[TagProtocol, DataProtocol]], + observer: ExecutionObserverProtocol | None, + ) -> None: + """The async form of ``_execute_load_only``: same rows, counts and observer calls. + + The node reads its whole input before it looks the entries up, then sends the + loaded rows; it stamps itself before its output closes (ITL-1042), and closes + the output whatever happens. + """ + from orcapod.pipeline.observer import NoOpObserver + from orcapod.pipeline.serialization import LoadStatus + + try: + if self.load_status == LoadStatus.UNAVAILABLE: + raise self._no_store_error() + require_same_data_context( + self._input_stream.data_context_key, self._stored_data_context_key + ) + node_label = self.label + node_hash = self.content_hash().to_string() + obs = observer if observer is not None else NoOpObserver() + ctx_obs = obs.contextualize(*self.node_identity_path) + tag_schema = self._input_stream.output_schema( + columns={"system_tags": True} + )[0] + ctx_obs.on_node_start(node_label, node_hash, tag_schema=tag_schema) + inputs = [ + (tag, data, self.compute_base_entry_id(tag, data)) + async for tag, data in input_channel + ] + _ = self.get_cached_results(base_entry_ids=[eid for _, _, eid in inputs]) + loaded_ids: list[bytes] = [] + not_stored = 0 + for tag, data, entry_id in inputs: + entry = self._stored_entry(entry_id) + if entry is None: + not_stored += 1 + continue + ctx_obs.on_data_start(node_label, tag, data) + ctx_obs.on_data_end(node_label, tag, data, entry[1], cached=True) + loaded_ids.append(entry_id) + if entry[1] is not None: + await output.send((entry[0], entry[1])) + self._set_run_rows(loaded_ids, load_only=True, not_stored=not_stored) + ctx_obs.on_node_end(node_label, node_hash) + self._update_modified_time() + except BaseException: + # A run that raised or was cancelled defines no rows. + self.clear_cache() + raise + finally: + try: + await output.close() + except BaseException: + # A close that raises or is cancelled fails the run: no rows. + self.clear_cache() + raise + def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: """Assign or remove the ephemeral result store. @@ -1648,8 +1892,8 @@ def _cannot_compute_error(self) -> RuntimeError: """ return RuntimeError( f"FunctionJobNode {self.label!r} was loaded by PipelineJob.load and has no " - "function pod, so it cannot compute; read its stored results with " - "as_table() or iter_data()." + "function pod, so it cannot compute; fill it from the store with " + "run(load_only=True), or read its stored results with as_source()." ) def _stored_record_count(self) -> int: @@ -1946,6 +2190,7 @@ def execute( observer: ExecutionObserverProtocol | None = None, error_policy: Literal["continue", "fail_fast"] = "continue", run_id: str | None = None, + load_only: bool = False, ) -> list[tuple[TagProtocol, DataProtocol]]: """Execute all data from a stream: compute, persist, and cache. @@ -1965,6 +2210,11 @@ def execute( ``"fail_fast"`` re-raises on the first failure. run_id: Optional pipeline run identifier threaded through to ``InvocationContext.pipeline_run_id``. + load_only: Fill the node with the results already stored for the + stream's rows and compute nothing (ITL-1095): no function call, no + write; inputs without a stored result are skipped and counted + (``last_run_not_stored``). A loaded node (no function pod) can run + this way. Returns: Materialized list of (tag, output_data) pairs, excluding @@ -1979,6 +2229,8 @@ def execute( """ from orcapod.pipeline.observer import NoOpObserver + if load_only: + return self._execute_load_only(input_stream, observer) if self._function_pod is None: raise self._cannot_compute_error() @@ -2023,6 +2275,7 @@ def execute( # Base entry ids of the inputs that gave a row (a cache hit or a # successful computation), one per input: the node's rows afterwards. emitted_entry_ids: list[bytes] = [] + failed_rows = 0 for tag, data, base_entry_id in upstream_entries: ctx_obs.on_data_start(node_label, tag, data) @@ -2076,6 +2329,7 @@ def execute( if error_policy == "fail_fast": ctx_obs.on_node_end(node_label, node_hash) raise + failed_rows += 1 continue ctx_obs.on_data_end(node_label, tag, data, result, cached=False) @@ -2090,7 +2344,7 @@ def execute( self._commit_at_node_end, ctx_obs, node_label, node_hash ) # The node's rows are those of this run's inputs, whatever else it caches. - self._set_run_rows(emitted_entry_ids) + self._set_run_rows(emitted_entry_ids, failed=failed_rows) ctx_obs.on_node_end(node_label, node_hash) except Exception as run_error: self._commit_after_failure(run_error) @@ -3169,14 +3423,27 @@ async def _async_execute_cache_only( self._cached_output_table = None self._cached_content_hash_column = None - for tag, data in self._cached_output_datas.values(): - if data is not None: + sent: list[bytes] = [] + for entry_id, (tag, data) in list(self._cached_output_datas.items()): + if data is not None and not isinstance(data, EmptyData): ctx_obs.on_data_start(node_label, tag, data) ctx_obs.on_data_end(node_label, tag, data, data, cached=True) await output.send((tag, data)) + sent.append(entry_id) + # The node holds the rows it sent: a completed load of what is stored + # (ITL-1095), stamped before the output closes (ITL-1042). + self._set_run_rows(sent, load_only=True) + self._update_modified_time() ctx_obs.on_node_end(node_label, node_hash) + except BaseException: + self.clear_cache() + raise finally: - await output.close() + try: + await output.close() + except BaseException: + self.clear_cache() + raise # ------------------------------------------------------------------ # Head @@ -3189,18 +3456,15 @@ def head( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> StreamView: - """Return the first rows the node holds or has stored; never runs the function. + """Return the first rows of the node's current run; never runs the function. After a run (or ``execute``) the rows are the node's results in memory for that - run's inputs, the first of what ``as_table`` returns. Otherwise they are the first of its stored - rows: the node's whole stored slice is read and nothing is kept on the node. - With nothing stored, there are no rows. The node has all of its rows, so with - ``sort_by_tags`` (or ``all_info``) the head is exactly - ``as_table(...).slice(0, n)``, whatever order the rows were computed in. See - ``StreamBase.head``. - - Warns: - StoreContentWarning: As for ``as_table``. + run's inputs, the first of what ``as_table`` returns. A node that has not run in + this process, or whose input changed since (stale), shows no rows and says so, + pointing to ``run()`` and to ``as_source()`` for its stored results (ITL-1095); + the store is not read. The node has all of its rows, so with ``sort_by_tags`` + (or ``all_info``) the head is exactly ``as_table(...).slice(0, n)``, whatever + order the rows were computed in. See ``StreamBase.head``. """ from orcapod.pipeline.serialization import LoadStatus @@ -3211,51 +3475,23 @@ def head( return self._head_without_rows( config, ViewStatus.UNAVAILABLE, note="no store attached" ) - current = status == LoadStatus.CACHE_ONLY or not self.is_stale - if current and ( - self._run_entry_counts is not None or self._cached_output_datas - ): - return self._head_of_pairs( - list(self._held_pairs()), n, config, origin="held" + if self._run_entry_counts is None: + return self._head_without_rows( + config, ViewStatus.NOT_COMPUTED, note=JOB_NODE_NOT_RUN_NOTE ) - return self._head_from_store(n, config) - - def _head_from_store(self, n: int, config: ColumnConfig) -> StreamView: - """Return the first ``n`` of this node's stored rows. - - Reads the node's whole stored slice (pipeline table joined with the result - table; no bounded read exists yet) and keeps nothing on the node. - """ - if self._function_pod is None: - # A loaded node: its table is built from the stored Arrow records. - self._check_expected_rows() - fetched = self._fetch_joined_records() - stored = None if fetched is None else self._stored_rows_stream(fetched) - stored_ids = set(stored[0]) if stored is not None else set() - missing = 0 - if fetched is not None: - missing = sum( - 1 for k in fetched.empty_data_tokens if k not in stored_ids - ) - note = ( - f"{missing:,} {'result' if missing == 1 else 'results'} not shown: kept " - "in an ephemeral store, which is not saved with the job" - if missing - else None + if status != LoadStatus.CACHE_ONLY and self.is_stale: + return self._head_without_rows( + config, ViewStatus.NOT_COMPUTED, note=STALE_NOTE ) - if stored is not None: - return ( - stored[1] - .head(n, columns=config) - .for_stream(self, origin="stored", note=note) - ) - return self._head_without_rows(config, ViewStatus.NOTHING_STORED, note=note) - if self._cached_function_pod is None: - return self._head_without_rows(config, ViewStatus.NOT_COMPUTED) - entries = list(self._load_cached_entries().values()) - if not any(data is not None for _, data in entries): - return self._head_without_rows(config, ViewStatus.NOTHING_STORED) - return self._head_of_pairs(entries, n, config, origin="stored") + head = self._head_of_pairs(list(self._held_pairs()), n, config, origin="held") + if not self._run_load_only: + return head + # A load-only run's rows have their own status: they may lack the inputs it + # skipped (ITL-1095). + note = self._load_only_note() + if head.note: + note += f"; {head.note}" + return dataclasses.replace(head, status=ViewStatus.LOADED_ONLY, note=note) # ------------------------------------------------------------------ # Iteration @@ -3267,54 +3503,33 @@ def as_table( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> "pa.Table": - """Return this node's results as a PyArrow table. + """Return the rows of this node's current run as a PyArrow table. - A node loaded by ``PipelineJob.load`` has no function pod. Its table is - built from the Arrow records its saved run stored, so every column keeps - its stored type. Row order follows the store. With nothing stored, the - table has zero rows and the node's columns. + Read-only: never computes and never reads the store. Before a run in this + process, or once the node is stale, the table has zero rows and the node's + columns (ITL-1095); ``as_source()`` reads what the store holds. Raises: - RuntimeError: If the node was loaded and no store is attached, or some of - its results were kept in an ephemeral store, which is not saved with - the job. - - Warns: - StoreContentWarning: Once per store, if the node was loaded from a job file - that records another number of pipeline records for its run than the - store holds. The read returns what the store holds. + RuntimeError: If the node was loaded and no store is attached. """ - if self._function_pod is None: - self._require_readable_stub() - fetched = self._fetch_joined_records() - if fetched is None: - return super().as_table(columns=columns, all_info=all_info) - stored = self._stored_rows_stream(fetched) - stored_ids = set(stored[0]) if stored is not None else set() - ephemeral = [k for k in fetched.empty_data_tokens if k not in stored_ids] - if ephemeral: - raise RuntimeError( - f"FunctionJobNode {self.label!r}: the saved run kept " - f"{len(ephemeral)} of its results in an ephemeral store, which " - "is not saved with the job, so they cannot be read into a " - "table. iter_data() yields them as EmptyData placeholders." - ) - if stored is not None: - return stored[1].as_table(columns=columns, all_info=all_info) + from orcapod.pipeline.serialization import LoadStatus + + if self.load_status == LoadStatus.UNAVAILABLE: + raise self._no_store_error() return super().as_table(columns=columns, all_info=all_info) def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: - """Yield this node's (tag, data) rows. + """Yield the (tag, data) rows of this node's current run. - Strictly read-only — never triggers computation. Callers must call - ``run()`` or ``execute()`` first if they want results computed. + Strictly read-only: never computes and never reads the store. Callers must + call ``run()`` (or run the job) first. After a completed run (``execute``, ``async_execute``, ``run()`` or a job run) the rows are exactly those of that run's inputs, on either orchestrator, even - though results of other inputs stay cached and stored (ITL-1093). A run that - raised defines no rows. Without a completed run in this process, on the first - call with an empty in-memory store and a DB attached, hot-loads every record - stored under the node's content hash (one-shot, no recompute) and yields them. + though results of other inputs stay cached and stored (ITL-1093). A node that + has not run in this process, whose run raised, or that is stale (its input + changed since its run) yields nothing (ITL-1095); ``as_source()`` reads every + row stored for it. Raises: RuntimeError: If ``load_status`` is UNAVAILABLE. @@ -3324,74 +3539,151 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: status = self.load_status if status == LoadStatus.UNAVAILABLE: raise self._no_store_error() - - if status == LoadStatus.CACHE_ONLY: - # Upstream unavailable; serve entirely from DB. - if not self._cached_output_datas: - loaded = self._load_cached_entries() - self._cached_output_datas.update(loaded) - if loaded: - self._cached_output_table = None - self._cached_content_hash_column = None - yield from self._held_pairs() - return - - # FULL / READ_ONLY — in-memory store may be populated from computation - # (via execute/run) or hot-loaded from DB. - if self.is_stale: + # A CACHE_ONLY node has no live input to be stale against. + if status != LoadStatus.CACHE_ONLY and self.is_stale: self.clear_cache() - - if self._function_pod is None: - # A loaded node: its rows can come only from the store. - self._check_expected_rows() - if not self._cached_output_datas and self._cached_function_pod is not None: - # Hot-load from DB on the first call when store is empty. - loaded = self._load_cached_entries() - self._cached_output_datas.update(loaded) - if loaded: - self._cached_output_table = None - self._cached_content_hash_column = None - yield from self._held_pairs() - def run(self) -> StreamView: - """Eagerly compute all input data, filling pipeline and result databases, and - return a ``StreamView`` of the node's rows. + def run(self, *, load_only: bool = False) -> StreamView: + """Run the node as a job run would, and return a ``StreamView`` of its rows. + + State-driven (developer, November 13, 2026; ITL-1095): a node that holds the rows + of a completed run and is not stale is not run again; the view shows what it + holds, and nothing is computed, read or written. Otherwise the node runs, after + every job node it depends on that holds no current run (each once, in + dependency order, as ``job.run()`` would run them; ``last_run`` reports the + counts), and the run's writes are committed. A node built over streams outside a + job reads its input as it is. - Computes as a job run computes this node, unlike ``as_table()``, which only - reads. A node whose upstream is unavailable (``CACHE_ONLY``) or whose function - pod is a placeholder (``READ_ONLY``) computes nothing: the view holds its stored - rows. + Args: + load_only: Fill the node, and the nodes it runs first, with what is already + stored and compute nothing (as ``job.run(load_only=True)``). A node + filled this way shows status ``loaded_only`` and is not current for a + later full ``run()``, which computes what it skipped. Returns: - A ``StreamView`` of every row ``as_table()`` then returns. It says "rows", - as ``head()`` then does: the node holds them in memory, whether it computed - them or read them from its store. + A ``StreamView`` of every row ``as_table()`` then returns: status ``rows``, + or ``loaded_only`` after a load-only run. Raises: - RuntimeError: If the node was loaded without a function pod, or - ``load_status`` is UNAVAILABLE (no pod, no DB). + RuntimeError: If no store is attached (``UNAVAILABLE``); if the node cannot + compute (loaded without a function pod, or a placeholder pod) and + ``load_only`` is false; or if its input is unavailable + (``CACHE_ONLY``): read its stored rows with ``as_source()``. """ from orcapod.pipeline.serialization import LoadStatus - if self._function_pod is None: - raise self._cannot_compute_error() - self._begin_run() - if self._load_status == LoadStatus.UNAVAILABLE: + status = self._load_status + if status == LoadStatus.UNAVAILABLE: raise self._no_store_error() - if self._load_status in (LoadStatus.CACHE_ONLY, LoadStatus.READ_ONLY): - # CACHE_ONLY: upstream unavailable; computation requires a live input stream. - # READ_ONLY: function pod is a proxy placeholder — cannot compute. - # The view serves the existing DB results, as iter_data() does. - return self._view_of_rows(self.as_table(), "held") + if status == LoadStatus.CACHE_ONLY: + raise RuntimeError( + f"FunctionJobNode {self.label!r}: its input is unavailable, so it has " + "no inputs to run or look up; read its stored results with as_source()." + ) + if not load_only and ( + self._function_pod is None or status == LoadStatus.READ_ONLY + ): + raise self._cannot_compute_error() + run_job_node( + self, + load_only=load_only, + run_alone=lambda: self._run_alone(load_only=load_only), + caller=sys._getframe(1), # pyright: ignore[reportPrivateUsage] # the caller's line is where a failure warning is shown + ) + return self._run_view() + + def _run_alone(self, *, load_only: bool) -> Any: + """Run this node over its input stream, which is not a job node; commit. + + Returns: + The run's ``RunSummary``. + """ + from dataclasses import replace + + from orcapod.pipeline.node_run import commit_run + from orcapod.pipeline.run_summary import RunCountingObserver, RunSummary + from orcapod.utils.store_errors import flush_best_effort + if self.is_stale: # Discard any stale in-memory entries before a fresh computation run # so that rerunning does not mix old cached entries with new results. self.clear_cache() - self.execute(self._input_stream) - # The rows are the ones held in memory, as head() names them afterwards: some - # may have been read from the result store rather than computed. - return self._view_of_rows(self.as_table(), "held") + counter = RunCountingObserver() + stores = [ + store + for store in ( + self._pipeline_database, + getattr(self._cached_function_pod, "result_database", None), + ) + if store is not None + ] + try: + _ = self.execute(self._input_stream, observer=counter, load_only=load_only) + except Exception as run_error: + if not load_only: + # Commit what succeeded; a load-only run writes nothing (ITL-1095). + flush_best_effort(run_error, *stores) + self.clear_cache() + raise + if not load_only: + commit_run(stores, [self]) + node_summary = replace( + counter.node_summary(self.label), not_stored=self.last_run_not_stored + ) + base_path = getattr(self._pipeline_database, "base_path", ()) + return RunSummary( + "/".join(base_path), [node_summary], load_only=load_only, target=self.label + ) + + def _incomplete_inputs(self) -> list[str]: + """Labels of the function nodes above this one whose load-only run skipped inputs.""" + labels: list[str] = [] + seen: list[Any] = [] + pending: list[Any] = list(self.upstreams) + while pending: + node = pending.pop() + if any(node is other for other in seen): + continue + seen.append(node) + if isinstance(node, FunctionJobNode) and node._run_load_only: + if node._run_not_stored: + labels.append(node.label) + pending.extend(getattr(node, "upstreams", ())) + return sorted(labels) + + def _load_only_note(self) -> str: + """The caption of a node filled by a load-only run (ITL-1095).""" + note = ( + f"loaded only: {self._run_loaded_count():,} loaded, " + f"{self._run_not_stored:,} inputs without a stored result" + ) + incomplete = self._incomplete_inputs() + if incomplete: + note += f"; inputs incomplete: {', '.join(incomplete)}" + return note + + def _run_loaded_count(self) -> int: + """How many inputs of the last run had a stored result (loaded).""" + counts = self._run_entry_counts + return 0 if counts is None else sum(counts.values()) + + def _run_view(self) -> StreamView: + """The ``run()`` view of the node's rows, with the status of its last run.""" + table = self.as_table() + if not self._run_load_only: + # The rows are the ones held in memory, as head() names them afterwards: + # some may have been read from the result store rather than computed. + return self._view_of_rows(table, "held") + return StreamView.of_rows( + self, + table, + tag_columns=self.keys()[0], + total_rows=table.num_rows, + status=ViewStatus.LOADED_ONLY, + note=self._load_only_note(), + display_all_rows=False, + ) # ------------------------------------------------------------------ # Async channel execution @@ -3405,6 +3697,7 @@ async def async_execute( observer: ExecutionObserverProtocol | None = None, error_policy: Literal["continue", "fail_fast"] = "continue", run_id: str | None = None, + load_only: bool = False, ) -> None: """Streaming async execution for FunctionJobNode. @@ -3432,6 +3725,8 @@ async def async_execute( necessarily the first in input order. run_id: Optional pipeline run identifier threaded through to ``InvocationContext.pipeline_run_id``. + load_only: As for ``execute``: fill the node with the stored results of + its input rows and compute nothing (ITL-1095). Raises: Exception: Under ``"fail_fast"``, the exception of the first row whose @@ -3443,6 +3738,9 @@ async def async_execute( """ from orcapod.pipeline.serialization import LoadStatus + if load_only: + await self._async_execute_load_only(input_channel, output, observer) + return try: self._begin_run() except StoreError: @@ -3532,6 +3830,8 @@ async def async_execute( # Base entry ids of this run's inputs that gave a row (a cache hit or a # forwarded computed row), one per input (ITL-1093). run_entry_ids: list[bytes] = [] + # Rows whose function raised (error policy "continue"). + failed_rows: list[int] = [0] async def route_inputs() -> None: """Stage 1: send cache hits to output; stamp misses for computation.""" @@ -3690,6 +3990,8 @@ def on_data_crash( # The pod re-raises a failed store read or write after # this call; name the node it belongs to. error.add_context(node_label=node_label) + else: + failed_rows[0] += 1 ctx_obs.on_data_crash(node_label, clean, data, error) # Clean up the dangling input_store entry so it doesn't # accumulate memory during long-running calls with errors. @@ -3721,7 +4023,7 @@ def create_data_logger(self, tag: Any, data: Any) -> Any: self._commit_at_node_end() # The node's rows are those of this run's inputs, as the sync path's: # the cache also holds every stored result loaded above. - self._set_run_rows(run_entry_ids) + self._set_run_rows(run_entry_ids, failed=failed_rows[0]) else: # No-DB path: delegate dispatch entirely to the pod. The pod closes # the writer it is given; that close waits for the finally below, so diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index e92574e01..ca0cd5cb0 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -17,6 +17,8 @@ from __future__ import annotations import asyncio +import dataclasses +import sys import logging from abc import abstractmethod from collections.abc import Collection, Iterator, Mapping, Sequence @@ -34,6 +36,8 @@ short_stream_repr, ) from orcapod.core.streams.view import ( + JOB_NODE_NOT_RUN_NOTE, + LOADED_ONLY_OPERATOR_NOTE, STALE_NOTE, BLUEPRINT_NOTE, ViewStatus, @@ -41,6 +45,7 @@ head_of, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER +from orcapod.core.nodes.staleness import RunHoldingNode from orcapod.errors import ( PipelineJobRequiredError, StoreReadError, @@ -679,7 +684,7 @@ def as_node(self) -> OperatorNode: # --------------------------------------------------------------------------- -class OperatorJobNode(OperatorNodeBase): +class OperatorJobNode(RunHoldingNode, OperatorNodeBase): """DB-backed execution node for operator pod invocations. Stream node representing an operator invocation with optional DB persistence. @@ -780,6 +785,8 @@ def __init__( pipeline_database=pipeline_database, cache_mode=cache_mode, ) + # A node that has not run in this process holds nothing and is stale (ITL-1095). + self._set_modified_time(None) # ------------------------------------------------------------------ # as_node — return the lightweight OperatorNode equivalent @@ -809,10 +816,17 @@ def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: # ------------------------------------------------------------------ def clear_cache(self) -> None: - """Clear output caches and node identity path cache.""" + """Clear output caches and node identity path cache. + + The node then holds no run: it is stale and reads empty, as before its first run + in this process (ITL-1095). Rows it stored stay stored (``as_source()``). + """ super().clear_cache() # clears _node_identity_path_cache + _update_modified_time self._cached_output_stream = None self._cached_output_table = None + self._run_load_only = False + # No content established: the node, and every node downstream, is stale. + self._set_modified_time(None) # ------------------------------------------------------------------ # attach_databases @@ -1066,7 +1080,9 @@ def _cannot_compute_error(self) -> RuntimeError: """Describe why this loaded node cannot execute an operator.""" return RuntimeError( f"OperatorJobNode {self.label!r} was loaded by PipelineJob.load and " - "cannot compute; create a fresh PipelineJob to run the pipeline again." + "cannot compute; fill it from the store with run(load_only=True), read " + "its stored results with as_source(), or create a fresh PipelineJob to " + "run the pipeline again." ) def _filter_by_content_hash(self, table: pa.Table) -> pa.Table: @@ -1299,6 +1315,50 @@ def _make_empty_table(self) -> "pa.Table": self.data_context.type_converter, ) + def _read_run_records(self) -> "pa.Table | None": + """Read the stored rows of this node's run: its content hash, current context. + + A loaded node names its run by the content hash its job file records. A file + saved before Orcapod recorded it leaves the node only the hash the loaded job + computes; when the store holds no rows under that hash, an empty answer would + make stored results look dropped, so this raises instead (ITL-1043). + + Returns: + The run's stored rows (``_read_stored_records``), or ``None`` when the table + does not exist. + + Raises: + RuntimeError: If a loaded node's job file does not record its run's + content hash and the store holds no rows under the hash the loaded + job computes. + """ + stored = self._read_current_context_records() + records = None if stored is None else self._filter_by_content_hash(stored) + if ( + self._loaded + and self._saved_run_content_hash is None + and (records is None or records.num_rows == 0) + ): + found = ( + "is absent from the attached store" + if stored is None + else f"holds {stored.num_rows} stored rows, none of them under the " + f"content hash the loaded job computes ({self.content_hash().to_string()})" + ) + raise _LoadedCacheUnreadableError( + f"OperatorJobNode {self.label!r}: the job file does not record the " + "content hash of this node's run (a file saved before Orcapod recorded " + "it), so the loaded job cannot name the run's rows, and its cache table " + f"({'/'.join(self._stored_table_path)}) {found}. Upgrading such files is " + "left to the migration tool after v0.3 (ITL-1048); until then build the " + "job from code and attach the node with CacheMode.REPLAY to read them.", + head_note=( + "loaded node: the job file does not record its run's content hash " + "(an older file; ITL-1048)" + ), + ) + return records + def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": """Read from DB in CacheMode.REPLAY only, without modifying node state. @@ -1330,31 +1390,7 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": return None if self._cache_mode != CacheMode.REPLAY: return None - stored = self._read_current_context_records() - records = None if stored is None else self._filter_by_content_hash(stored) - if ( - self._loaded - and self._saved_run_content_hash is None - and (records is None or records.num_rows == 0) - ): - found = ( - "is absent from the attached store" - if stored is None - else f"holds {stored.num_rows} stored rows, none of them under the " - f"content hash the loaded job computes ({self.content_hash().to_string()})" - ) - raise _LoadedCacheUnreadableError( - f"OperatorJobNode {self.label!r}: the job file does not record the " - "content hash of this node's run (a file saved before Orcapod recorded " - "it), so the loaded job cannot name the run's rows, and its cache table " - f"({'/'.join(self._stored_table_path)}) {found}. Upgrading such files is " - "left to the migration tool after v0.3 (ITL-1048); until then build the " - "job from code and attach the node with CacheMode.REPLAY to read them.", - head_note=( - "loaded node: the job file does not record its run's content hash " - "(an older file; ITL-1048)" - ), - ) + records = self._read_run_records() if self._loaded: self._check_expected_rows(0 if records is None else records.num_rows) if records is None: @@ -1398,19 +1434,25 @@ def execute( self, *input_streams: StreamProtocol, observer: ExecutionObserverProtocol | None = None, + load_only: bool = False, ) -> list[tuple[TagProtocol, DataProtocol]]: """Execute input streams: compute, persist, and cache. Args: *input_streams: Input streams to execute. observer: Optional execution observer for hooks. + load_only: A load-only run (ITL-1095): the operator still runs on the rows + that reach it (it is not a stored computation), but nothing is + written: a ``LOG`` node does not store its output. ``REPLAY`` replays + as in any run. Returns: Materialized list of (tag, data) pairs. """ from orcapod.pipeline.observer import NoOpObserver - if self._loaded: + # A loaded node runs only in a load-only run, with its rebuilt operator. + if self._loaded and (not load_only or self._operator is None): raise self._cannot_compute_error() node_label = self.label @@ -1419,49 +1461,60 @@ def execute( obs = observer if observer is not None else NoOpObserver() ctx_obs = obs.contextualize(*self.node_identity_path) - ctx_obs.on_node_start(node_label, node_hash) + try: + ctx_obs.on_node_start(node_label, node_hash) - # Check REPLAY cache first - cached_output = self.get_cached_output() - if cached_output is not None: - output = list(cached_output.iter_data()) - ctx_obs.on_node_end(node_label, node_hash) - return output + self._run_load_only = load_only + # Check REPLAY cache first + cached_output = self.get_cached_output() + if cached_output is not None: + output = list(cached_output.iter_data()) + ctx_obs.on_node_end(node_label, node_hash) + return output - # Compute - result_stream = self._operator.process(*input_streams) + # Compute + result_stream = self._operator.process(*input_streams) - # Materialize - output = list(result_stream.iter_data()) + # Materialize + output = list(result_stream.iter_data()) - # Cache - if output: - self._cached_output_stream = StaticOutputOperatorPod._materialize_to_stream( - output - ) - else: - self._cached_output_stream = result_stream + # Cache + if output: + self._cached_output_stream = ( + StaticOutputOperatorPod._materialize_to_stream(output) + ) + else: + self._cached_output_stream = result_stream - self._update_modified_time() + self._update_modified_time() - # Persist to DB only in LOG mode - if ( - self._pipeline_database is not None - and self._cache_mode == CacheMode.LOG - and self._cached_output_stream is not None - ): - self._store_output_stream(self._cached_output_stream) + # Persist to DB only in LOG mode, and never in a load-only run. + if ( + not load_only + and self._pipeline_database is not None + and self._cache_mode == CacheMode.LOG + and self._cached_output_stream is not None + ): + self._store_output_stream(self._cached_output_stream) + + ctx_obs.on_node_end(node_label, node_hash) + return output + except BaseException: + # A run that raised (its store write or an observer included) defines no rows. + self.clear_cache() + raise - ctx_obs.on_node_end(node_label, node_hash) - return output + def _compute_and_store(self, *, store: bool = True) -> None: + """Compute operator output, optionally store in DB. - def _compute_and_store(self) -> None: - """Compute operator output, optionally store in DB.""" + Args: + store: Whether a ``LOG`` node stores the output (not in a load-only run). + """ self._cached_output_stream = self._operator.process( *self._input_streams, ) - if self._cache_mode == CacheMode.OFF: + if self._cache_mode == CacheMode.OFF or not store: self._update_modified_time() return @@ -1502,12 +1555,15 @@ def _replay_from_cache(self) -> None: ) self._update_modified_time() - def run(self) -> StreamView: - """Execute the operator according to the current cache mode, and return a - ``StreamView`` of the node's rows. + def run(self, *, load_only: bool = False) -> StreamView: + """Run the node as a job run would, and return a ``StreamView`` of its rows. - Computes as a job run computes this node, unlike ``as_table()``, which only - reads. + State-driven (developer, November 13, 2026; ITL-1095): a node that holds the + output of a completed run and is not stale is not run again; the view shows it. + Otherwise the node runs, after every job node it depends on that holds no + current run (each once, in dependency order, as ``job.run()`` would run them; + ``last_run`` reports the counts), and the run's writes are committed. A node + built over streams outside a job runs its operator on them directly. Without a database: Always compute via the operator's ``process()`` method. @@ -1517,30 +1573,79 @@ def run(self) -> StreamView: - **LOG**: always compute, write results to DB. - **REPLAY**: skip computation, load from DB. + Args: + load_only: As ``job.run(load_only=True)``: the operator runs on the rows + that reach it, a ``LOG`` node writes nothing, and the nodes run first + load their stored results only. + Returns: A ``StreamView`` of every row ``as_table()`` then returns, naming its rows as ``head()`` then names them. + + Raises: + RuntimeError: If the node was loaded by ``PipelineJob.load`` and + ``load_only`` is false. """ + from orcapod.pipeline.node_run import ( + depends_on_job_nodes, + run_with_dependencies, + ) + from orcapod.pipeline.run_summary import warn_row_failures + + if self._loaded and not load_only: + raise self._cannot_compute_error() + if self.holds_current_run(load_only=load_only): + return self._run_view(computed=False) + if depends_on_job_nodes(self): + summary = run_with_dependencies(self, load_only=load_only) + self._last_run = summary + if not summary.ok: + warn_row_failures(summary, sys._getframe(1)) # pyright: ignore[reportPrivateUsage] # the caller's line is where the warning is shown + return self._run_view(computed=False) if self._loaded: raise self._cannot_compute_error() - if self.is_stale: + if self.is_stale or not self._holds_run(load_only=load_only): + # Stale, or a load-only run's output asked for by a full run: run again. self.clear_cache() computed = False if self._cached_output_stream is None: - if self._pipeline_database is not None: - if self._cache_mode == CacheMode.REPLAY: - self._replay_from_cache() + try: + if self._pipeline_database is not None: + if self._cache_mode == CacheMode.REPLAY: + self._replay_from_cache() + else: + self._compute_and_store(store=not load_only) + computed = True + if self._cache_mode == CacheMode.LOG and not load_only: + # Committed before run() returns, as at the end of a job + # run. + from orcapod.pipeline.node_run import commit_run + + commit_run([self._pipeline_database], [self]) + self._run_load_only = load_only else: - self._compute_and_store() + self._cached_output_stream = self._operator.process( + *self._input_streams, + ) + self._update_modified_time() computed = True - else: - self._cached_output_stream = self._operator.process( - *self._input_streams, - ) - self._update_modified_time() - computed = True - table = self.as_table() + except BaseException: + # A run that raised defines no rows. + self.clear_cache() + raise + return self._run_view(computed=computed) + + def _run_view(self, *, computed: bool) -> StreamView: + """The ``run()`` view of the node's output. + + Args: + computed: Whether this ``run()`` computed the output lazily just now. + """ + # Read the kept output itself: a lazy input that finishes computing during this + # read stamps itself after the node, which would make as_table() judge it stale. + assert self._cached_output_stream is not None + table = self._cached_output_stream.as_table() if computed: # Without a database and in OFF mode the output stream is lazy: as_table() # computes it, and an uncomputed input finishes only then. Stamp the node @@ -1551,28 +1656,47 @@ def run(self) -> StreamView: # computation, plain rows when replayed from the store or kept by execute(). kept = self._cached_output_stream origin = "computed" if kept is None else head_of(kept.head(1), self).origin - return self._view_of_rows(table, origin) + view = self._view_of_rows(table, origin) + if self._run_load_only: + return dataclasses.replace( + view, status=ViewStatus.LOADED_ONLY, note=LOADED_ONLY_OPERATOR_NOTE + ) + return view - def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: - """Return an iterator over (tag, data) pairs. + # Whether the output the node holds came from a load-only run (ITL-1095): its + # input may lack rows, so a full run does not take it as complete. + _run_load_only: bool = False + + def _holds_run(self, *, load_only: bool) -> bool: + """Whether the node holds a run's output; a load-only run's only for ``load_only``.""" + return self._cached_output_stream is not None and ( + load_only or not self._run_load_only + ) - Read-only: never triggers computation. Yields the rows of the latest - ``run()``, ``execute()`` or ``async_execute()``. With nothing cached in - memory (before a run, or after ``clear_cache()``), a node in - ``CacheMode.REPLAY`` with a database yields the rows stored there; in - any other mode it yields nothing. Call ``node.is_stale`` before - iterating if you need to detect outdated cached data. + def _current_output(self) -> StreamProtocol | None: + """The output of the node's current run, or ``None`` when it holds none. - Loaded nodes raise when no readable stored cache is available. + A node holds a run's output from the end of ``run()``, ``execute()`` or + ``async_execute()`` until it is cleared or becomes stale (an input or the + operator changed since). Before a run in this process it holds none, whatever + its cache mode and store hold (ITL-1095). """ - if self._cached_output_stream is not None: - return self._cached_output_stream.iter_data() - db_stream = self._load_cached_stream_from_db() - if db_stream is not None: - return db_stream.iter_data() - if self._loaded: - raise self._no_stored_results_error() - return iter([]) + if self._cached_output_stream is None or self.is_stale: + return None + return self._cached_output_stream + + def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: + """Return an iterator over the (tag, data) rows of the node's current run. + + Read-only: never computes and never reads the store. Yields the rows of the + latest ``run()``, ``execute()`` or ``async_execute()``; nothing before a run in + this process, after ``clear_cache()`` or once the node is stale, in every cache + mode (ITL-1095). ``as_source()`` reads what the node stored (``LOG``). + """ + current = self._current_output() + if current is None: + return iter([]) + return current.iter_data() def as_table( self, @@ -1580,13 +1704,12 @@ def as_table( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> "pa.Table": - """Return the output as a PyArrow table. + """Return the output of the node's current run as a PyArrow table. - Read-only: never triggers computation. Returns the rows of the latest - ``run()``, ``execute()`` or ``async_execute()``. With nothing cached in - memory (before a run, or after ``clear_cache()``), a node in - ``CacheMode.REPLAY`` with a database returns the rows stored there; in - any other mode it returns a zero-row table with the node's columns. + Read-only: never computes and never reads the store. Returns the rows of the + latest ``run()``, ``execute()`` or ``async_execute()``; before a run in this + process, after ``clear_cache()`` or once the node is stale, a zero-row table + with the node's columns, in every cache mode (ITL-1095). That zero-row table has the columns, Arrow types and nullability of the node's rows, system tags and source info included: it is the operator's @@ -1595,29 +1718,29 @@ def as_table( types without rows (a ``PollingSource`` before its first poll, an unbound input slot or ``SourceProxy``, a ``DerivedSource``, or a loaded node with no live upstream), the table is built from the node's Python output - schema, as before: an ``int8`` column reads as ``int64`` and the - system-tag columns are absent (ITL-1089, ITL-985). - - Loaded nodes raise when no readable stored cache is available. + schema: an ``int8`` column reads as ``int64`` and the system-tag columns + are absent (ITL-1089, ITL-985). """ - if self._cached_output_stream is not None: - return self._cached_output_stream.as_table( - columns=columns, all_info=all_info - ) - db_stream = self._load_cached_stream_from_db() - if db_stream is not None: - return db_stream.as_table(columns=columns, all_info=all_info) - if self._loaded: - raise self._no_stored_results_error() - # No cached or stored records yet: construct a zero-row table with the - # correct schema and route it through ArrowTableStream so that the - # ColumnConfig / all_info shaping is applied consistently. - # Requires a live operator for schema derivation (pre-existing limitation). + current = self._current_output() + if current is not None: + return current.as_table(columns=columns, all_info=all_info) if self._operator is None: - return pa.table({}) - empty_records = self._make_empty_table() - tag_keys = self.keys()[0] - empty_stream = ArrowTableStream(empty_records, tag_columns=tag_keys) + # A node rebuilt without its operator: the columns of its saved schema. + tag_schema, data_schema = self.output_schema() + empty = pa.Table.from_pylist( + [], + schema=self.data_context.type_converter.python_schema_to_arrow_schema( + tag_schema + data_schema + ), + ) + return ArrowTableStream( + empty, tag_columns=tuple(tag_schema), data_context=self.data_context + ).as_table(columns=columns, all_info=all_info) + empty_stream = ArrowTableStream( + self._make_empty_table(), + tag_columns=self.keys()[0], + data_context=self.data_context, + ) return empty_stream.as_table(columns=columns, all_info=all_info) def head( @@ -1627,66 +1750,31 @@ def head( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> StreamView: - """Return the first rows the node holds or has stored; never runs it over rows. + """Return the first rows of the node's current run; never runs the operator. - After a run (or ``execute``) the rows are the node's output in memory. Before - that, a node in ``CacheMode.REPLAY`` shows the first of its stored rows, read - from its whole stored table; in ``LOG`` and ``OFF`` it shows none, as - ``as_table`` does. A loaded node never raises here. See ``StreamBase.head``. + After a run (or ``execute``) the rows are the node's output in memory. Before a + run in this process the node shows no rows and says so, pointing to ``run()`` + and to ``as_source()``, in every cache mode; once stale it says its inputs + changed (ITL-1095). The store is never read. See ``StreamBase.head``. """ check_head_size(n) config = ColumnConfig.handle_config(columns, all_info=all_info) - if self._cached_output_stream is not None: - if not self._loaded and self.is_stale: - # run() would recompute; the kept rows are not the current result. - return self._head_without_rows( - config, ViewStatus.NOT_COMPUTED, note=STALE_NOTE - ) - return head_of(self._cached_output_stream.head(n, columns=config), self) - if self._cache_mode == CacheMode.REPLAY and self._pipeline_database is not None: - return self._head_from_store(n, config) - if self._loaded: + if self._cached_output_stream is None: return self._head_without_rows( - config, - ViewStatus.UNAVAILABLE, - note=( - "loaded node whose results are not stored (cache mode OFF)" - if self._cache_mode == CacheMode.OFF - else "loaded node whose stored results are not replayed " - f"(cache mode {self._cache_mode.value.upper()})" - ), - ) - if self._cache_mode == CacheMode.LOG and self._pipeline_database is not None: - note = ( - "not run in this session; rows of earlier runs are stored but not " - "replayed (cache mode LOG)" + config, ViewStatus.NOT_COMPUTED, note=JOB_NODE_NOT_RUN_NOTE ) - else: - note = "not run; cache mode OFF stores nothing" - return self._head_without_rows(config, ViewStatus.NOT_COMPUTED, note=note) - - def _head_from_store(self, n: int, config: ColumnConfig) -> StreamView: - """Return the first ``n`` of this node's stored rows. - - Reads the node's whole stored slice (no bounded read exists yet) through the - same path as ``as_table`` and keeps nothing on the node. A loaded node whose - run's rows cannot be read (its cache table is absent, or none of its rows is the - run's) shows no rows and says why, where ``as_table`` raises. - """ - try: - stream = self._load_cached_stream_from_db() - except _LoadedCacheUnreadableError as exc: + if self.is_stale: + # run() would recompute; the kept rows are not the current result. return self._head_without_rows( - config, ViewStatus.UNAVAILABLE, note=exc.head_note + config, ViewStatus.NOT_COMPUTED, note=STALE_NOTE ) - if stream is None or stream.as_table().num_rows == 0: - return self._head_without_rows( - config, - ViewStatus.NOTHING_STORED, - # A loaded job cannot run: say what the store holds instead. - note="the store holds no rows for its run" if self._loaded else None, + head = head_of(self._cached_output_stream.head(n, columns=config), self) + if self._run_load_only: + # Output of a load-only run: its input may lack rows (ITL-1095). + return dataclasses.replace( + head, status=ViewStatus.LOADED_ONLY, note=LOADED_ONLY_OPERATOR_NOTE ) - return stream.head(n, columns=config).for_stream(self, origin="stored") + return head # ------------------------------------------------------------------ # DB retrieval @@ -1707,7 +1795,9 @@ def get_all_records( if self._pipeline_database is None: return None - results = self._read_stored_records() + # A loaded node that cannot name its run's rows raises rather than show the + # stored rows as none (ITL-1043): this is the read behind node.as_source(). + results = self._read_run_records() if self._loaded: # The advisory check of every read of a loaded node (ITL-1043). self._check_expected_rows(0 if results is None else results.num_rows) @@ -1795,6 +1885,7 @@ async def async_execute( output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, observer: "ExecutionObserverProtocol | None" = None, + load_only: bool = False, ) -> None: """Async execution with cache mode handling when DB is attached. @@ -1816,10 +1907,12 @@ async def async_execute( inputs: Sequence of readable channels from upstream nodes. output: Writable channel for output (tag, data) pairs. observer: Optional execution observer for hooks. + load_only: As for ``execute``: a ``LOG`` node stores nothing (ITL-1095). """ from orcapod.pipeline.observer import NoOpObserver - if self._loaded: + # A loaded node runs only in a load-only run, with its rebuilt operator. + if self._loaded and (not load_only or self._operator is None): await output.close() raise self._cannot_compute_error() @@ -1830,6 +1923,7 @@ async def async_execute( try: ctx_obs.on_node_start(node_label, node_hash) + self._run_load_only = load_only if ( self._pipeline_database is not None @@ -1846,6 +1940,9 @@ async def async_execute( for tag, data in self._cached_output_stream.iter_data(): await output.send((tag, data)) _ = await asyncio.gather(*discards) + # The inputs are closed, so every upstream node has stamped itself: + # stamp after them, or the replayed rows would read as stale. + self._update_modified_time() finally: for discard in discards: _ = discard.cancel() @@ -1858,7 +1955,8 @@ async def async_execute( # Delegate to the operator, forward its rows downstream and keep them. intermediate: Channel[tuple[TagProtocol, DataProtocol]] = Channel() should_store = ( - self._pipeline_database is not None + not load_only + and self._pipeline_database is not None and self._cache_mode == CacheMode.LOG ) if should_store: @@ -1925,5 +2023,14 @@ async def forward() -> None: self._update_modified_time() ctx_obs.on_node_end(node_label, node_hash) + except BaseException: + # A run that raised or was cancelled defines no rows. + self.clear_cache() + raise finally: - await output.close() + try: + await output.close() + except BaseException: + # A close that raises or is cancelled fails the run: no rows. + self.clear_cache() + raise diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index fdee2631d..6cc61b804 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -10,6 +10,7 @@ import functools import logging from abc import ABC, abstractmethod +from datetime import datetime from collections.abc import AsyncIterator, Iterator from typing import TYPE_CHECKING, Any @@ -590,6 +591,20 @@ def __init__( # bound_source property — explicit binding with cache invalidation # ------------------------------------------------------------------ + def run(self, *, load_only: bool = False) -> StreamView: + """Read the bound source's rows and return a ``StreamView`` of all of them. + + A source is read, never computed, so ``load_only`` changes nothing; a job + node's ``run()`` takes it (ITL-1095). + + Args: + load_only: Accepted for every job node; a source reads its rows either way. + + Raises: + UnboundSourceError: When no concrete source is bound. + """ + return super().run() + @property def bound_source(self) -> StreamProtocol | None: """The concrete stream currently bound to this slot, or ``None``.""" @@ -597,10 +612,35 @@ def bound_source(self) -> StreamProtocol | None: @bound_source.setter def bound_source(self, value: StreamProtocol | None) -> None: - """Bind *value* as the concrete source and invalidate both hash caches.""" + """Bind *value* as the concrete source and invalidate both hash caches. + + Binding stamps the slot as modified, so every job node fed by it is stale and + reads empty until it runs again (ITL-1095). + """ self._bound_source = value self._invalidate_content_hash_cache() self._invalidate_pipeline_hash_cache() + self._update_modified_time() + + @property + def last_modified(self) -> datetime | None: + """When the slot's content last changed: its binding or its bound source. + + The later of the time the slot was bound and the bound source's own + ``last_modified`` (a ``CachedSource`` or ``PollingSource`` moves it when it + reads new rows). A bound source that does not know its modification time makes + the slot's unknown too (``None``), so every node below it is stale. A source + whose rows change without its modified time moving (a table read lazily, a file + rewritten in place) is not seen: build a new source object and bind it, or clear + the dependent nodes. + """ + own = self._modified_time + if self._bound_source is None: + return own + bound = self._bound_source.last_modified + if bound is None or own is None: + return bound + return max(own, bound) # ------------------------------------------------------------------ # Identity — delegate to bound source when set diff --git a/src/orcapod/core/nodes/staleness.py b/src/orcapod/core/nodes/staleness.py new file mode 100644 index 000000000..8ae8162a8 --- /dev/null +++ b/src/orcapod/core/nodes/staleness.py @@ -0,0 +1,129 @@ +"""Staleness of job nodes across the job graph (ITL-1095). + +A stream is stale when an upstream or its producer was modified after it +(``StreamBase.is_stale``), which looks one level up only. A job node holds the rows of +its current run, so it is stale also when any job node it depends on is stale: a source +rebound two nodes up makes every node below it stale, not only its direct consumer. +""" + +from __future__ import annotations + +from collections.abc import Iterable +from typing import Any + + +class RunHoldingNode: + """Mixin of the job nodes that hold the rows of their current run. + + ``FunctionJobNode``, ``OperatorJobNode`` and ``SideEffectJobNode`` mix it in ahead of + their stream base, so their ``is_stale`` covers their whole ancestry. + """ + + # The summary of the node's last ``run()`` (with the nodes it ran first); a class + # default so nodes rebuilt from a saved descriptor start without one. + _last_run: Any = None + + @property + def last_run(self) -> Any: + """The ``RunSummary`` of this node's last ``run()``, or ``None``. + + ``node.run()`` runs the node and the job nodes it depends on (ITL-1095); this is + what that run did, per function node, as ``job.last_run`` is for ``job.run()``. + A ``run()`` that found the node current ran nothing and leaves it unchanged. + """ + return self._last_run + + @property + def is_stale(self) -> bool: + """Whether the rows this node holds are out of date, or it holds none. + + True before the node's first run in this process and after ``clear_cache()`` + (no modified time), when an input or its producer was modified after the + node's run (``StreamBase.is_stale``), or when any job node it depends on is + stale (ITL-1095). A stale node reads empty until it runs again. Each node of + the ancestry is checked once, however many paths lead to it. + """ + return _stale(self, {}) + + def _own_staleness(self) -> bool: + """``StreamBase.is_stale`` of this node alone: its inputs and producer.""" + own: bool = super().is_stale # pyright: ignore[reportAttributeAccessIssue] # the stream base after this mixin in the MRO defines it + return own + + def holds_current_run(self, *, load_only: bool = False) -> bool: + """Whether the node holds the rows of a run that ``run()`` would not redo. + + A node that is not stale and holds the rows of a completed run in this process + is current: ``run()`` returns them without running it (developer, November 13, + 2026, ITL-1095). A node filled by a load-only run is current only for another + load-only run: a full run computes the inputs it skipped. So is every node + whose rows came from such a node: a node is current only if every job node + above it is current for the same kind of run. Each node of the ancestry is + checked once. + + Args: + load_only: Whether the question is for a load-only run. + + Returns: + ``True`` if the node holds a current run for such a run. + """ + return _current(self, load_only, {}, {}) + + def _holds_run(self, *, load_only: bool) -> bool: + """Whether the node holds the rows of a completed run good for such a run.""" + raise NotImplementedError + + +def _job_upstreams(node: Any) -> list[RunHoldingNode]: + """The direct upstreams of ``node`` that are job nodes holding run rows.""" + upstreams: tuple[Any, ...] = getattr(node, "upstreams", ()) + return [upstream for upstream in upstreams if isinstance(upstream, RunHoldingNode)] + + +def _stale(node: RunHoldingNode, memo: dict[int, bool]) -> bool: + """Whether ``node`` is stale; ``memo`` keeps each node's answer for one traversal.""" + key = id(node) + if key not in memo: + memo[key] = node._own_staleness() or any( # pyright: ignore[reportPrivateUsage] # the mixin's own helper + _stale(upstream, memo) for upstream in _job_upstreams(node) + ) + return memo[key] + + +def _current( + node: RunHoldingNode, + load_only: bool, + memo: dict[int, bool], + stale_memo: dict[int, bool], +) -> bool: + """Whether ``node`` holds a current run; memoized for one traversal.""" + key = id(node) + if key not in memo: + memo[key] = ( + node._holds_run(load_only=load_only) # pyright: ignore[reportPrivateUsage] # the mixin's own hook + and not _stale(node, stale_memo) + and all( + _current(upstream, load_only, memo, stale_memo) + for upstream in _job_upstreams(node) + ) + ) + return memo[key] + + +def upstream_job_node_stale(upstreams: Iterable[Any]) -> bool: + """Return whether any job node in ``upstreams`` that holds run rows is stale. + + Each such node answers with its own ``is_stale``, which applies this rule to its + own upstreams in turn, so the whole ancestry is covered. + + Args: + upstreams: The direct upstream streams of a job node. + + Returns: + ``True`` if any upstream ``RunHoldingNode`` is stale. + """ + memo: dict[int, bool] = {} + return any( + isinstance(upstream, RunHoldingNode) and _stale(upstream, memo) + for upstream in upstreams + ) diff --git a/src/orcapod/core/streams/view.py b/src/orcapod/core/streams/view.py index 9450f50eb..0e22e43ee 100644 --- a/src/orcapod/core/streams/view.py +++ b/src/orcapod/core/streams/view.py @@ -47,14 +47,19 @@ class ViewStatus(str, Enum): ROWS: The rows are the first rows of the stream's complete content. Zero rows means the content is empty. PARTIAL: A computation stopped part way; the rows are those computed so far. + LOADED_ONLY: A job node filled by a load-only run (``run(load_only=True)``): + the rows are the stored results of its inputs; inputs without one were + skipped, so the rows may be fewer than a full run gives (ITL-1095). NOT_COMPUTED: Nothing is computed, or what was computed is out of date. - NOTHING_STORED: A job node whose store holds no rows for it yet. + NOTHING_STORED: A stored view (``node.as_source()``) whose store holds no rows + for its node yet. UNAVAILABLE: No rows can exist here: a pipeline blueprint, an input slot with no source bound, or a source that is not available. """ ROWS = "rows" PARTIAL = "partial" + LOADED_ONLY = "loaded_only" NOT_COMPUTED = "not_computed" NOTHING_STORED = "nothing_stored" UNAVAILABLE = "unavailable" @@ -65,11 +70,27 @@ class ViewStatus(str, Enum): "not computed: inputs changed since the last computation; run() recomputes it" ) NOTHING_STORED_NOTE = "nothing stored; run the job" +# A job node shows the rows of its current run only; before one it shows none (ITL-1095). +JOB_NODE_NOT_RUN_NOTE = ( + "not run; run() fills it; node.as_source() shows every stored result" +) BLUEPRINT_NOTE = "pipeline blueprint; run it in a PipelineJob" SIDE_EFFECTS_NOTE = ( "not run; run() delivers its side effects; displaying never delivers side effects" ) +# An operator node whose output came from a load-only run: its input may lack rows that a +# full run computes (ITL-1095). +LOADED_ONLY_OPERATOR_NOTE = ( + "loaded only: the output of a load-only run, whose inputs may lack rows" +) + +# A side-effect node whose rows came from a load-only run: it passed them on and delivered +# nothing (ITL-1095). +LOADED_ONLY_SIDE_EFFECT_NOTE = ( + "loaded only: a load-only run passed these rows on and delivered nothing" +) + _DEFAULT_NOTES = { ViewStatus.NOT_COMPUTED: NOT_COMPUTED_NOTE, ViewStatus.NOTHING_STORED: NOTHING_STORED_NOTE, @@ -369,7 +390,7 @@ def _is_window(self) -> bool: can be named by position: a ``ROWS`` view with no ``description``, unsorted or sorted among all of the stream's rows.""" return ( - self.status is ViewStatus.ROWS + self.status in (ViewStatus.ROWS, ViewStatus.LOADED_ONLY) and self.description is None and (not self.sorted_by_tags or self._total_is_exact()) ) @@ -389,7 +410,11 @@ def _holds_every_row(self) -> bool: def _cuttable(self) -> StreamView: """Return this view in a form whose rows can be cut by position: itself when its rows are a run of the stream's rows (or of a pool), else a pool of its rows.""" - if self.status not in (ViewStatus.ROWS, ViewStatus.PARTIAL): + if self.status not in ( + ViewStatus.ROWS, + ViewStatus.PARTIAL, + ViewStatus.LOADED_ONLY, + ): return self if self.pool is not None or self._is_window(): return self @@ -420,7 +445,7 @@ def _describe(self) -> str: shown = self.table.num_rows if self.pool is not None: return _window_text(self.pool, shown, self.start, self.pool_rows) - if self.status is ViewStatus.ROWS: + if self.status in (ViewStatus.ROWS, ViewStatus.LOADED_ONLY): text = self._describe_rows(shown) elif self.status is ViewStatus.PARTIAL: total = self.total_rows if self.total_rows is not None else shown diff --git a/src/orcapod/pipeline/async_orchestrator.py b/src/orcapod/pipeline/async_orchestrator.py index e9c28147e..437b3ccd9 100644 --- a/src/orcapod/pipeline/async_orchestrator.py +++ b/src/orcapod/pipeline/async_orchestrator.py @@ -31,6 +31,9 @@ ) if TYPE_CHECKING: + from collections.abc import Mapping + + from orcapod.channels import WritableChannel from orcapod.pipeline.dag import GraphProtocol from orcapod.protocols.core_protocols import DataProtocol, TagProtocol from orcapod.protocols.observability_protocols import ExecutionObserverProtocol @@ -75,6 +78,8 @@ def run( materialize_results: bool = True, run_id: str | None = None, pipeline_uri: str = "", + load_only: bool = False, + held_rows: Mapping[Any, list[tuple[TagProtocol, DataProtocol]]] | None = None, ) -> OrchestratorResult: """Synchronous entry point — runs the async pipeline to completion. @@ -118,6 +123,8 @@ def run( observer=observer, run_id=run_id, pipeline_uri=pipeline_uri, + load_only=load_only, + held_rows=held_rows, ) ) @@ -138,6 +145,8 @@ def make_run() -> Coroutine[Any, Any, OrchestratorResult]: observer=watched, run_id=run_id, pipeline_uri=pipeline_uri, + load_only=load_only, + held_rows=held_rows, ) return _run_on_own_thread(make_run, rows_in_flight) @@ -150,6 +159,8 @@ async def run_async( materialize_results: bool = True, run_id: str | None = None, pipeline_uri: str = "", + load_only: bool = False, + held_rows: Mapping[Any, list[tuple[TagProtocol, DataProtocol]]] | None = None, ) -> OrchestratorResult: """Async entry point for callers already inside an event loop. @@ -161,6 +172,9 @@ async def run_async( generated automatically. pipeline_uri: Opaque URI string identifying this pipeline run. Forwarded verbatim to ``observer.on_run_start``. + load_only: As for ``SyncPipelineOrchestrator.run``. + held_rows: As for ``SyncPipelineOrchestrator.run``: nodes not run, whose + rows feed their successors. Returns: OrchestratorResult with node outputs. @@ -171,6 +185,8 @@ async def run_async( observer=observer, run_id=run_id, pipeline_uri=pipeline_uri, + load_only=load_only, + held_rows=held_rows, ) async def _run_async( @@ -181,6 +197,8 @@ async def _run_async( observer: ExecutionObserverProtocol | None = None, run_id: str | None = None, pipeline_uri: str = "", + load_only: bool = False, + held_rows: Mapping[Any, list[tuple[TagProtocol, DataProtocol]]] | None = None, ) -> OrchestratorResult: """Core async logic: wire channels, launch tasks, collect results.""" from orcapod.pipeline.observer import NoOpObserver @@ -242,7 +260,16 @@ async def _run_async( collector = collectors[node] writer = _CollectingWriter(writer, collector) - if is_source_node(node): + if held_rows is not None and node in held_rows: + # Not run: the rows of its current run feed its successors. Its + # own inputs are read and discarded, so no producer blocks on + # a channel nobody reads. + for upstream in in_edges.get(node, []): + tg.create_task(_drain(edge_readers[(upstream, node)])) + _start_node( + tg, failed_nodes, node, _send_held(held_rows[node], writer) + ) + elif is_source_node(node): _start_node( tg, failed_nodes, @@ -266,6 +293,7 @@ async def _run_async( observer=effective_observer, error_policy=self._error_policy, run_id=run_id, + load_only=load_only, ), ) elif is_operator_node(node): @@ -286,7 +314,10 @@ async def _run_async( failed_nodes, node, node.async_execute( - input_readers, writer, observer=effective_observer + input_readers, + writer, + observer=effective_observer, + load_only=load_only, ), ) elif is_side_effect_node(node): @@ -296,6 +327,8 @@ async def _run_async( "SideEffectNode expects exactly 1 upstream, got 0" ) input_reader = edge_readers[(predecessors[0], node)] + # In a load-only run the node passes its rows on and delivers + # nothing. _start_node( tg, failed_nodes, @@ -305,6 +338,7 @@ async def _run_async( writer, observer=effective_observer, run_id=run_id, + load_only=load_only, ), ) else: @@ -604,6 +638,24 @@ def create_data_logger(self, tag: Any, data: Any) -> Any: return NoOpLogger() +async def _drain(reader: Any) -> None: + """Read ``reader`` to its end, keeping nothing.""" + async for _ in reader: + pass + + +async def _send_held( + rows: list[tuple[TagProtocol, DataProtocol]], + writer: WritableChannel[tuple[TagProtocol, DataProtocol]], +) -> None: + """Send the rows of a node the run holds rather than runs, then close.""" + try: + for row in rows: + await writer.send(row) + finally: + await writer.close() + + def _start_node( tg: asyncio.TaskGroup, failed_nodes: dict[int, str], diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index a077015fa..80c32e95f 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -5,7 +5,6 @@ import contextlib import logging import sys -import warnings from collections.abc import Iterator from dataclasses import dataclass from typing import TYPE_CHECKING, Any @@ -23,7 +22,6 @@ unclaimed_tables_written_before_v03, ) from orcapod.errors import ( - RowFailuresWarning, StoreError, StoreReadError, StoreWriteError, @@ -31,14 +29,12 @@ ) from orcapod.pipeline.base import AbstractPipelineBase from orcapod.pipeline.dag import OrcaDAG +from orcapod.pipeline.node_run import RunPlan, clear_run_rows, plan_run from orcapod.pipeline.run_summary import ( - NodeRunSummary, RunCountingObserver, RunSummary, -) -from orcapod.protocols.node_protocols import ( - is_function_node, - is_side_effect_function_node, + run_node_summaries, + warn_row_failures, ) from orcapod.protocols import core_protocols as cp from orcapod.databases.utils import as_table_committing @@ -47,7 +43,6 @@ if TYPE_CHECKING: from pathlib import Path - from types import FrameType from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator from orcapod.pipeline.execution_context import ExecutionContext @@ -75,6 +70,7 @@ class _PreparedRun: pipeline_uri: ``@``, passed to the observer. pipeline_db: The job's view of the store. result_db: The job's function-result view of the store. + plan: What the run runs and holds (``plan_run``, ITL-1095). """ exec_dag: OrcaDAG[Any] @@ -82,6 +78,7 @@ class _PreparedRun: pipeline_uri: str pipeline_db: "ArrowDatabaseProtocol" result_db: "ArrowDatabaseProtocol" + plan: "RunPlan" def _with_counter( @@ -103,29 +100,6 @@ def _with_counter( return CompositeObserver(observer, counter) # pyright: ignore[reportReturnType] # forwards every hook; its **kwargs signatures do not match the protocol structurally -def _warn_row_failures(summary: RunSummary, caller: FrameType) -> None: - """Issue the run's ``RowFailuresWarning``, shown after every run with failed rows. - - Python's default filter shows a warning once per text and line, so a second identical - failing run from the same line (re-running a notebook cell) would show nothing. A - fresh registry for each run keeps every run's warning, while the user's filters - (``ignore``, ``error``) still apply. - - Args: - summary: The run's summary, with failed rows. - caller: The frame that called ``run``; the warning is shown at its line. - """ - warnings.warn_explicit( - summary.failures_message(), - RowFailuresWarning, - caller.f_code.co_filename, - caller.f_lineno, - module=caller.f_globals.get("__name__"), - registry=None, - module_globals=caller.f_globals, - ) - - class PipelineJob(AbstractPipelineBase[JobNode]): """Pipeline + source bindings + execution context. @@ -872,6 +846,8 @@ def run( observer: "ExecutionObserverProtocol | None" = None, materialize_results: bool = True, commit_every_rows: int = 100, + *, + load_only: bool = False, ) -> RunSummary: """Execute the resolvable subgraph of this job in place. @@ -883,6 +859,14 @@ def run( After a successful run, ``job.nodes`` returns the execution-ready nodes with populated database caches. + State-driven, as every job node's ``run()`` (developer, November 14, 2026; + ITL-1095): a node that holds a current run (completed in this process, not + stale, and for a full run not filled by a load-only run) is held, not run; its + rows feed the nodes after it, and nothing is computed, read, written or + delivered for it. A source is read only for the nodes that run. So a second run + of an unchanged job does nothing. To run a node again, call ``clear_cache()`` on + it (every node below it runs too), or bind a new source object. + ``job.run(orchestrator=AsyncPipelineOrchestrator())`` works the same in a script and inside a running event loop, such as a Jupyter kernel: there the run goes on a separate thread while the call waits, and blocks that loop @@ -920,6 +904,15 @@ def run( default 1000): the store commits every pending table by itself once one table's pending rows reach that size, which would break the node's results-before-records commit order. + load_only: Fill the nodes from what is already stored and compute + nothing (ITL-1095). Sources are read and operators run on the rows + that reach them; each function node takes the results stored for + this run's inputs (its own records under its content hash) and skips + the inputs without one, which the summary counts (``not_stored``); + a function node forwards only the rows it loaded. No user function + is called, no side effect is delivered (a side-effect node passes its + rows on) and nothing is written to the store, a ``LOG`` operator's + cache included. Afterwards each node shows the rows it loaded. Rows whose function raised (with the default ``error_policy="continue"``) are skipped, and the run goes on. At the end, if any row failed, a single @@ -952,20 +945,24 @@ def run( """ from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator - prepared = self._prepare_run(commit_every_rows) + prepared = self._prepare_run(commit_every_rows, load_only=load_only) + # The state rule (ITL-1095): a node that holds a current run is not run again. + plan = prepared.plan counter = RunCountingObserver() effective_orchestrator = orchestrator or SyncPipelineOrchestrator() - with self._executing(prepared, commit_every_rows): + with self._executing(prepared, plan, commit_every_rows, load_only=load_only): _ = effective_orchestrator.run( - prepared.exec_dag, + plan.dag, observer=_with_counter(observer, counter), run_id=prepared.run_id, pipeline_uri=prepared.pipeline_uri, materialize_results=materialize_results, + load_only=load_only, + held_rows=plan.held_rows, ) - summary = self._finish_run(prepared, counter) + summary = self._finish_run(prepared, counter, plan, load_only=load_only) if not summary.ok: - _warn_row_failures(summary, sys._getframe(1)) # pyright: ignore[reportPrivateUsage] # the caller's line is where the warning is shown + warn_row_failures(summary, sys._getframe(1)) # pyright: ignore[reportPrivateUsage] # the caller's line is where the warning is shown return summary async def run_async( @@ -974,6 +971,8 @@ async def run_async( observer: "ExecutionObserverProtocol | None" = None, materialize_results: bool = True, commit_every_rows: int = 100, + *, + load_only: bool = False, ) -> RunSummary: """Execute the job with the async orchestrator, from async code. @@ -989,6 +988,7 @@ async def run_async( observer: Optional execution observer. materialize_results: As for ``run``. commit_every_rows: As for ``run``. + load_only: As for ``run``. Returns: The run's ``RunSummary``, as ``run`` returns. @@ -1011,26 +1011,33 @@ async def run_async( f"{type(orchestrator).__name__}. Use job.run(orchestrator=...) to run " "it." ) - prepared = self._prepare_run(commit_every_rows) + prepared = self._prepare_run(commit_every_rows, load_only=load_only) + # The state rule (ITL-1095): a node that holds a current run is not run again. + plan = prepared.plan counter = RunCountingObserver() - with self._executing(prepared, commit_every_rows): + with self._executing(prepared, plan, commit_every_rows, load_only=load_only): _ = await orchestrator.run_async( - prepared.exec_dag, + plan.dag, observer=_with_counter(observer, counter), run_id=prepared.run_id, pipeline_uri=prepared.pipeline_uri, materialize_results=materialize_results, + load_only=load_only, + held_rows=plan.held_rows, ) - summary = self._finish_run(prepared, counter) + summary = self._finish_run(prepared, counter, plan, load_only=load_only) if not summary.ok: - _warn_row_failures(summary, sys._getframe(1)) # pyright: ignore[reportPrivateUsage] # the caller's line is where the warning is shown + warn_row_failures(summary, sys._getframe(1)) # pyright: ignore[reportPrivateUsage] # the caller's line is where the warning is shown return summary - def _prepare_run(self, commit_every_rows: int) -> "_PreparedRun": + def _prepare_run( + self, commit_every_rows: int, *, load_only: bool = False + ) -> "_PreparedRun": """Check the job can run and build what a run needs; shared by both run forms. Args: commit_every_rows: As for ``run``. + load_only: As for ``run``: a loaded job can run only this way. Returns: The execution DAG, run id, pipeline URI and the store views of the run. @@ -1057,14 +1064,17 @@ def _prepare_run(self, commit_every_rows: int) -> "_PreparedRun": "PipelineJob has no compiled pipeline. " "Record invocations inside a 'with job:' block first." ) - # A loaded job reads the results its saved run stored and never runs: - # function pods are not saved with it, and even an operator-only loaded - # job, whose operators are rebuilt, stays read-only. - if self._loaded: + # A loaded job never computes: function pods are not saved with it, and even + # an operator-only loaded job, whose operators are rebuilt, stays read-only. + # A load-only run fills it from the store (ITL-1095). + if self._loaded and not load_only: raise RuntimeError( - "PipelineJob.run() cannot execute a loaded job: PipelineJob.load() " - "is for reading stored results only. To run a new computation, " - "create a fresh PipelineJob from the original pipeline definition." + "PipelineJob.run() cannot execute a loaded job: its function pods are " + "not saved with it, so it cannot compute. Fill its nodes from the " + "store with run(load_only=True) (bind in-memory sources first), or " + "read a node's stored results with node.as_source(). To run a new " + "computation, create a fresh PipelineJob from the original pipeline " + "definition." ) # Until this run finishes, what the store holds for a node is not certain. self._last_run_finished = False @@ -1105,7 +1115,11 @@ def _prepare_run(self, commit_every_rows: int) -> "_PreparedRun": isinstance(node, (FunctionJobNode, OperatorJobNode)) and node.load_status == LoadStatus.UNAVAILABLE ) - or (isinstance(node, FunctionJobNode) and node.producer is None) + or ( + not load_only + and isinstance(node, FunctionJobNode) + and node.producer is None + ) ] if unavailable_labels: raise RuntimeError( @@ -1128,6 +1142,10 @@ def _prepare_run(self, commit_every_rows: int) -> "_PreparedRun": excluded_hashes: set[str] = set() for node_hash in hash_dag.topological_sort(): if node_hash not in self._persistent_node_map: + # A node a loaded job did not restore (ITL-1094): below an unbound + # source it is left out too, so the nodes below it are as well. + if any(p in excluded_hashes for p in hash_dag.predecessors(node_hash)): + excluded_hashes.add(node_hash) continue node = self._persistent_node_map[node_hash] if isinstance(node, SourceJobNode) and node.bound_source is None: @@ -1135,7 +1153,30 @@ def _prepare_run(self, commit_every_rows: int) -> "_PreparedRun": elif any(p in excluded_hashes for p in hash_dag.predecessors(node_hash)): excluded_hashes.add(node_hash) - self._report_unclaimed_tables_written_before_v03(excluded_hashes) + # A loaded job whose graph lost a node (PipelineJob.load does not restore + # side-effect nodes yet, ITL-1094) cannot feed the nodes below it: every saved + # input of a node must be a restored node. + if self._loaded: + for node_hash, node in self._persistent_node_map.items(): + if node_hash in excluded_hashes: + continue # below an unbound source: not part of this run + if not isinstance(node, (FunctionJobNode, OperatorJobNode)): + continue + if node_hash not in self._hash_graph: + continue + missing = [ + upstream + for upstream in self._hash_graph.predecessors(node_hash) + if upstream not in self._persistent_node_map + ] + if missing: + raise RuntimeError( + f"Node {node.label!r} of the loaded job has lost " + f"{len(missing)} of its input nodes: PipelineJob.load does not " + "restore side-effect nodes yet (ITL-1094), so the nodes below " + "one cannot run, not even load-only. Read their stored results " + "with node.as_source()." + ) # --- Build execution OrcaDAG from existing node objects (no cloning) --- exec_dag: OrcaDAG = OrcaDAG() @@ -1174,7 +1215,16 @@ def _prepare_run(self, commit_every_rows: int) -> "_PreparedRun": # this run writes. Those rows would otherwise count towards a table's # batch_size, which could then be reached mid-run and commit this run's pipeline # records ahead of their results (ITL-919). - if table_committing is not None: + # The state rule (ITL-1095): a node that holds a current run is not run again. + plan = plan_run(exec_dag, load_only=load_only) + # Pre-v0.3 tables are reported for a run that runs a node or leaves one out + # (ITL-954): an excluded node reports nothing itself, so the job's total counts + # its tables. Only a run that runs nothing and leaves nothing out lists no store. + if plan.to_run or excluded_hashes: + self._report_unclaimed_tables_written_before_v03(excluded_hashes) + # A load-only run writes nothing, and a run that runs no node touches no store, + # so neither commits anything, not even pending writes of earlier operations. + if table_committing is not None and not load_only and plan.to_run: try: pipeline_db.flush() except StoreError as exc: @@ -1191,43 +1241,66 @@ def _prepare_run(self, commit_every_rows: int) -> "_PreparedRun": pipeline_uri=pipeline_uri, pipeline_db=pipeline_db, result_db=result_db, + plan=plan, ) @contextlib.contextmanager def _executing( - self, prepared: "_PreparedRun", commit_every_rows: int + self, + prepared: "_PreparedRun", + plan: RunPlan, + commit_every_rows: int, + *, + load_only: bool = False, ) -> Iterator[None]: """Hold every function node to the run's commit cadence while the orchestrator runs. - A failure inside commits the writes that succeeded before it, then propagates. + A failure inside commits the writes that succeeded before it (not in a load-only + run, which writes nothing), clears every node the run ran (whether their writes + were committed is unknown), then propagates. Args: prepared: The run, from ``_prepare_run``. + plan: The nodes the run runs (``plan_run``). commit_every_rows: As for ``run``. + load_only: As for ``run``. """ try: with contextlib.ExitStack() as batched: # Function nodes commit every commit_every_rows rows (ITL-919). - for node in prepared.exec_dag.nodes(): + for node in plan.to_run: if isinstance(node, FunctionJobNode): batched.enter_context( node._batched_commits(commit_every_rows) # pyright: ignore[reportPrivateUsage] # the job owns its nodes' commit cadence ) yield - except Exception as run_error: + except BaseException as run_error: # Commit the writes that did succeed before the failure, so their rows - # do not look dropped, then report the failure itself. - flush_best_effort(run_error, prepared.pipeline_db, prepared.result_db) + # do not look dropped, then report the failure itself. A cancelled or + # interrupted run commits nothing more, but clears its nodes too. + if isinstance(run_error, Exception) and not load_only: + flush_best_effort(run_error, prepared.pipeline_db, prepared.result_db) + clear_run_rows(plan.to_run) raise def _finish_run( - self, prepared: "_PreparedRun", counter: RunCountingObserver + self, + prepared: "_PreparedRun", + counter: RunCountingObserver, + plan: RunPlan, + *, + load_only: bool = False, ) -> RunSummary: """Commit the run's last writes, mark the job as run and keep its summary. + A failed commit clears the nodes the run ran (they hold no completed run); + the nodes it held keep the rows of their earlier run. + Args: prepared: The run, from ``_prepare_run``. counter: The observer that counted the run's rows. + plan: The nodes the run ran and held (``plan_run``). + load_only: Whether the run loaded stored results only. Returns: The run's summary, also kept as ``last_run``. @@ -1246,27 +1319,37 @@ def _finish_run( + "have been stored." ) try: - pipeline_db.flush() - result_db.flush() + if not load_only and plan.to_run: + # A load-only run writes nothing and commits nothing, and a run that + # ran no node touches no store (ITL-1095). + pipeline_db.flush() + result_db.flush() except StoreReadError as exc: # A stored table could not be read: report the read failure itself. exc.add_note(pending_note) + # A run whose commit failed did not complete: no node holds its rows. + clear_run_rows(plan.to_run) raise except Exception as exc: flush_error = StoreWriteError(pipeline_db.base_path, flush=True) flush_error.add_note(pending_note) + clear_run_rows(plan.to_run) raise flush_error from exc + except BaseException: + # An interruption (KeyboardInterrupt, cancellation) leaves the commit's + # outcome unknown: no node holds the run's rows. Re-raised unchanged. + clear_run_rows(plan.to_run) + raise # Mutate self in place. self._has_run = True self._last_run_finished = True self._run_id = prepared.run_id - node_summaries: list[NodeRunSummary] = [] - for node in prepared.exec_dag.topological_sort(): - label: str = node.label - if is_function_node(node) or is_side_effect_function_node(node): - node_summaries.append(counter.node_summary(label)) - self._last_run = RunSummary("/".join(self._name), node_summaries) + self._last_run = RunSummary( + "/".join(self._name), + run_node_summaries(prepared.exec_dag, counter, held=plan.held_counts), + load_only=load_only, + ) return self._last_run def __repr__(self) -> str: @@ -1402,10 +1485,12 @@ def load( """Deserialize a ``PipelineJob`` from a JSON file to read its stored results. A loaded job is for reading the results its saved run stored; it cannot be - run again, because function pods are not saved with the job. Read a - function node with ``job.nodes[label].as_table()``, ``iter_data()`` or - ``get_all_records()``. Operators can store outputs when caching is enabled; - reading a loaded operator without a readable cache raises ``RuntimeError``. + run again, because function pods are not saved with the job. Its nodes have + not run in this process, so they are empty (ITL-1095): read a node's stored + results through its stored view, ``job.nodes[label].as_source()``, or with + ``get_all_records()``. Operators store outputs only when caching is enabled; + the stored view of a loaded operator whose run's rows cannot be named raises + ``RuntimeError``. The embedded ``Pipeline`` blueprint is always restored. Sources whose data lives in files (CSV, Delta) are rebuilt from the saved config. In-memory @@ -1421,7 +1506,8 @@ def load( over any database configuration in the file. Returns: - A read-only ``PipelineJob`` whose nodes serve their stored results. + A read-only ``PipelineJob`` whose nodes' stored views serve their stored + results. Raises: ValueError: If the file's format version is unsupported, or if it was diff --git a/src/orcapod/pipeline/node_run.py b/src/orcapod/pipeline/node_run.py new file mode 100644 index 000000000..159578c94 --- /dev/null +++ b/src/orcapod/pipeline/node_run.py @@ -0,0 +1,273 @@ +"""``node.run()`` of a job node: run it and the job nodes it depends on (ITL-1095). + +Developer, November 13, 2026: "given that the initially loaded job is empty, I think it's +fair that node.run will cascade node.run on all nodes it depends on recursively. +Critically, node.run(load_only=True) would trigger the upstream node.run also with +load_only=True." And: "if it's not stale, it shouldn't even trigger a run and simply +return what's in the stream cached already." + +The cascade is ``job.run()`` restricted to the node's dependencies, on the synchronous +orchestrator: a dependency that holds a current run is not run again (its rows feed the +nodes after it), every other one runs once, in dependency order, and the run ends with +the commit ``job.run()`` makes. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any + +from orcapod.errors import StoreError, StoreWriteError +from orcapod.pipeline.run_summary import ( + RunCountingObserver, + RunSummary, + run_node_summaries, +) +from orcapod.utils.store_errors import flush_best_effort + +if TYPE_CHECKING: + from orcapod.protocols.database_protocols import ArrowDatabaseProtocol + + +def _is_job_node(stream: Any) -> bool: + """Whether ``stream`` is a node of a job (as opposed to a stream outside any job).""" + from orcapod.core.nodes.source_node import SourceJobNode + from orcapod.core.nodes.staleness import RunHoldingNode + from orcapod.side_effects import SideEffectJobNode + + return isinstance(stream, (RunHoldingNode, SideEffectJobNode, SourceJobNode)) + + +def depends_on_job_nodes(node: Any) -> bool: + """Whether any direct input of ``node`` is a job node. + + A node built directly over streams outside a job (a source, a pod's stream) has + nothing to run first: its ``run()`` reads its input as it always has. + """ + return any(_is_job_node(upstream) for upstream in node.upstreams) + + +def _stores(nodes: list[Any]) -> list[ArrowDatabaseProtocol]: + """The distinct stores the given nodes write to: pipeline and result databases.""" + found: list[ArrowDatabaseProtocol] = [] + for node in nodes: + pipeline_database = getattr(node, "_pipeline_database", None) + cached_pod = getattr(node, "_cached_function_pod", None) + result_database = getattr(cached_pod, "result_database", None) + for database in (pipeline_database, result_database): + if database is not None and not any(database is d for d in found): + found.append(database) + return found + + +def _job_name(node: Any) -> str: + """The name of the job ``node`` belongs to: its pipeline database's location.""" + database = getattr(node, "_pipeline_database", None) + base_path: tuple[str, ...] = getattr(database, "base_path", ()) + return "/".join(base_path) + + +def clear_run_rows(nodes: Iterable[Any]) -> None: + """Drop the rows of a run whose end-of-run commit failed from every node it filled. + + A run that raised defines no rows (ITL-1093): each function and operator job node of + the run is cleared (not run, stale) before the commit's error propagates. What was + committed stays stored. A load-only run writes nothing, so its commit has nothing + to fail on. + + Args: + nodes: The nodes of the run. + """ + from orcapod.core.nodes.staleness import RunHoldingNode + + for node in nodes: + if isinstance(node, RunHoldingNode): + node.clear_cache() # pyright: ignore[reportAttributeAccessIssue] # every RunHoldingNode is a job node with clear_cache + + +@dataclass(frozen=True) +class RunPlan: + """What a run does with each node of its graph (ITL-1095). + + Attributes: + dag: The nodes the run touches: those it runs, and the predecessors that feed + them (a source slot is read; any other one is held). + held_rows: The rows of each held node of ``dag``: the rows of its current run, + or of a stream outside the job; they feed the nodes after it as if it had + just run. + to_run: The nodes the run runs, in dependency order. + held_counts: Every node of the whole graph that holds a current run, with its + row count, for the summary. + """ + + dag: Any + held_rows: dict[Any, list[Any]] + to_run: list[Any] + held_counts: dict[Any, int] + + +def plan_run(graph: Any, *, load_only: bool, outside: Sequence[Any] = ()) -> RunPlan: + """Choose the nodes a run runs: the state rule of every job node's run (ITL-1095). + + A node that holds a current run (``holds_current_run``) is not run; a source slot + is read only when a node that runs reads it; a stream outside the job + (``outside``) is read for the nodes it feeds; every other node runs. The same plan + serves ``job.run()`` (its whole graph) and ``node.run()`` (the node and what it + depends on). + + Args: + graph: The run's node graph (``topological_sort``, ``predecessors``). + load_only: Whether the run is load-only (a node filled by a load-only run is + current only for one). + outside: Streams of ``graph`` that are not job nodes. + + Returns: + The plan. + """ + from orcapod.core.nodes.source_node import SourceJobNode + from orcapod.core.nodes.staleness import RunHoldingNode + from orcapod.pipeline.dag import OrcaDAG + + order = list(graph.topological_sort()) + + def is_outside(node: Any) -> bool: + """Whether ``node`` is one of the ``outside`` streams: by identity, not by + equality (two equal streams are still two inputs).""" + return any(node is other for other in outside) + + def is_current(node: Any) -> bool: + """Whether ``node`` holds a run this run need not redo (it is held).""" + return isinstance(node, RunHoldingNode) and node.holds_current_run( + load_only=load_only + ) + + def is_source(node: Any) -> bool: + """Whether ``node`` is a source slot: read when a node that runs reads it.""" + return isinstance(node, SourceJobNode) + + current = [node for node in order if is_current(node)] + to_run = [ + node + for node in order + if not is_source(node) + and not is_outside(node) + and not any(node is c for c in current) + ] + included: list[Any] = [] + for node in order: + feeds_a_run = any( + any(successor is r for r in to_run) for successor in graph.successors(node) + ) + if any(node is r for r in to_run) or feeds_a_run: + included.append(node) + dag: OrcaDAG[Any] = OrcaDAG() + for node in included: + dag.add_node(node) + for node in to_run: + for upstream in graph.predecessors(node): + dag.add_edge(upstream, node) + held_rows = { + node: list(node.iter_data()) + for node in included + if not any(node is r for r in to_run) and not is_source(node) + } + held_counts = {node: sum(1 for _ in node.iter_data()) for node in current} + return RunPlan(dag, held_rows, to_run, held_counts) + + +def commit_run(stores: Sequence[ArrowDatabaseProtocol], ran: Sequence[Any]) -> None: + """Commit a run's writes at its end; a failed commit clears the nodes that ran. + + A run whose commit failed did not complete, so no node it ran holds its rows + (ITL-1095); nodes it held keep the rows of their earlier run. + + Raises: + StoreError: The commit's own store error. + StoreWriteError: If committing failed for another reason. + """ + for store in stores: + try: + store.flush() + except StoreError: + clear_run_rows(ran) + raise + except Exception as exc: + clear_run_rows(ran) + raise StoreWriteError(store.base_path, flush=True) from exc + except BaseException: + # An interruption during the commit leaves its outcome unknown too. + clear_run_rows(ran) + raise + + +def run_with_dependencies(target: Any, *, load_only: bool) -> RunSummary: + """Run ``target`` and every job node it depends on that holds no current run. + + The dependencies are found through ``upstreams``; the plan (``plan_run``) is the + one ``job.run()`` makes over the job: a dependency that holds a current run is not + run and its rows feed the nodes after it, a stream outside the job is read, every + other node runs once, in dependency order, on the ``SyncPipelineOrchestrator`` + (error policy ``"continue"``). The run ends with a commit of every store it wrote + to, also, best-effort, when it fails; a load-only run commits nothing (it writes + nothing, and pending writes of others stay pending). A run that raises leaves no + node it ran holding rows. + + Args: + target: The job node whose ``run()`` this is. + load_only: Whether to load stored results only. + + Returns: + The run's summary, one entry per function node of the graph. + + Raises: + StoreWriteError: If committing the run's writes failed. + """ + from orcapod.core.nodes.source_node import SourceJobNode + from orcapod.pipeline.dag import OrcaDAG + from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator + + graph: OrcaDAG[Any] = OrcaDAG() + seen: list[Any] = [] + outside: list[Any] = [] + + def visit(node: Any) -> None: + """Add ``node`` and what it depends on, up to the sources of the job.""" + if any(node is other for other in seen): + return + seen.append(node) + graph.add_node(node) + if isinstance(node, SourceJobNode): + return + if not _is_job_node(node): + outside.append(node) + return + for upstream in node.upstreams: + visit(upstream) + graph.add_edge(upstream, node) + + visit(target) + plan = plan_run(graph, load_only=load_only, outside=outside) + stores = _stores(plan.to_run) + counter = RunCountingObserver() + try: + _ = SyncPipelineOrchestrator().run( + plan.dag, observer=counter, load_only=load_only, held_rows=plan.held_rows + ) + except BaseException as run_error: + if isinstance(run_error, Exception) and not load_only: + # Commit the writes that succeeded before the failure, as job.run() does; + # a load-only run writes nothing, so it commits nothing (ITL-1095). + flush_best_effort(run_error, *stores) + # A run that raised leaves no node it ran holding rows: whether their writes + # were committed is unknown. + clear_run_rows(plan.to_run) + raise + if not load_only: + commit_run(stores, plan.to_run) + return RunSummary( + _job_name(target), + run_node_summaries(graph, counter, held=plan.held_counts), + load_only=load_only, + target=target.label, + ) diff --git a/src/orcapod/pipeline/run_summary.py b/src/orcapod/pipeline/run_summary.py index cf094c1c4..e3bc3dac2 100644 --- a/src/orcapod/pipeline/run_summary.py +++ b/src/orcapod/pipeline/run_summary.py @@ -19,6 +19,7 @@ if TYPE_CHECKING: from collections.abc import Mapping, Sequence + from types import FrameType from orcapod.protocols.core_protocols import DataProtocol, TagProtocol from orcapod.types import SchemaLike @@ -37,6 +38,13 @@ class NodeRunSummary: failed: Rows whose function raised; the run skipped them. first_error: The exception of the first failed row, with its traceback, or ``None`` when no row failed. + not_stored: Inputs a load-only run skipped because no result is stored for + them (``job.run(load_only=True)``; always 0 in a full run). + inputs_incomplete: The labels of the function nodes upstream of this one that + skipped inputs in a load-only run, in run order: this node's inputs lack + the rows those results would have given (a group may be partial). + held: The rows of a node the run did not run because it held a current run + from its previous run (ITL-1095): it computed, read and wrote nothing. """ label: str @@ -44,11 +52,14 @@ class NodeRunSummary: reused: int failed: int first_error: BaseException | None + not_stored: int = 0 + inputs_incomplete: tuple[str, ...] = () + held: int = 0 @property def rows(self) -> int: - """All rows the node received: computed, reused and failed.""" - return self.computed + self.reused + self.failed + """All rows of the node: computed, reused, failed, not stored and held.""" + return self.computed + self.reused + self.failed + self.not_stored + self.held class RunSummary: @@ -61,17 +72,32 @@ class RunSummary: Attributes: job_name: The job's name. nodes: Per function node, in run order, keyed by label. + load_only: Whether the run was a load-only run (``job.run(load_only=True)``): + it computed nothing; ``reused`` counts the rows it loaded. + target: For the run of one node and the nodes it depends on + (``node.run()``), that node's label; ``None`` for a job run. """ - def __init__(self, job_name: str, nodes: Sequence[NodeRunSummary]) -> None: + def __init__( + self, + job_name: str, + nodes: Sequence[NodeRunSummary], + *, + load_only: bool = False, + target: str | None = None, + ) -> None: """Build the summary. Args: job_name: The job's name. nodes: One entry per function node of the run, in run order. + load_only: Whether the run loaded stored results only. + target: The label of the node whose ``run()`` this was, if any. """ self.job_name: str = job_name self.nodes: Mapping[str, NodeRunSummary] = {node.label: node for node in nodes} + self.load_only: bool = load_only + self.target: str | None = target @property def failed_rows(self) -> int: @@ -92,7 +118,7 @@ def failures_message(self) -> str: return " ".join( f"{node.failed} of {_rows(node.rows)} failed in node {node.label!r} " f"(first error: {_error_line(node.first_error)}). " - f"Full traceback: {_traceback_call(node.label)}." + f"Full traceback: {_traceback_call(node.label, self.target)}." for node in self.nodes.values() if node.failed ) @@ -120,28 +146,50 @@ def print_traceback(self, label: str | None = None) -> None: print(f"No rows failed{where} in this run.") def _headline(self) -> str: - """The first line: the job and whether any row failed.""" + """The first line: the job (or node) and whether any row failed.""" + subject = ( + f"job {self.job_name!r}" + if self.target is None + else f"node {self.target!r} and the nodes it depends on" + ) + if self.nodes and all( + node.held and not (node.computed or node.reused or node.failed) + for node in self.nodes.values() + ): + run = "Load-only run" if self.load_only else "Run" + return f"{run} of {subject}: nothing to do; every node holds a current run." + if self.load_only: + skipped = sum(node.not_stored for node in self.nodes.values()) + outcome = ( + f"{_rows(skipped)} without a stored result skipped." + if skipped + else "every input had a stored result." + ) + return f"Load-only run of {subject}: {outcome}" if self.failed_rows: outcome = f"{_rows(self.failed_rows)} failed." else: outcome = "all rows succeeded." - return f"Run of job {self.job_name!r}: {outcome}" + return f"Run of {subject}: {outcome}" def __repr__(self) -> str: """The summary as text: one line per node, and the first error of a failing one.""" lines = [self._headline()] for node in self.nodes.values(): lines.append( - f" {node.label}: {_rows(node.rows)}: {node.computed} computed, " - f"{node.reused} from store, {node.failed} failed" + f" {node.label}: {_rows(node.rows)}: {_counts(node, self.load_only)}" ) if node.failed: lines.append(f" first error: {_error_line(node.first_error)}") - lines.append(f" Full traceback: {_traceback_call(node.label)}") + lines.append( + f" Full traceback: {_traceback_call(node.label, self.target)}" + ) return "\n".join(lines) def _repr_html_(self) -> str: """The notebook display: the same content as ``repr``, as a table.""" + if self.load_only: + return self._load_only_html() header = "".join( f"{name}" for name in ( @@ -162,7 +210,7 @@ def _repr_html_(self) -> str: if node.failed: error = ( f"{html.escape(_error_line(node.first_error))}
" - f"{html.escape(_traceback_call(node.label))}" + f"{html.escape(_traceback_call(node.label, self.target))}" ) body.append( "" @@ -177,6 +225,45 @@ def _repr_html_(self) -> str: f"{''.join(body)}" ) + def _load_only_html(self) -> str: + """The notebook display of a load-only run: rows loaded and skipped per node.""" + header = "".join( + f"{name}" + for name in ("node", "rows", "loaded", "not stored", "inputs incomplete") + ) + body = [ + "" + f"{html.escape(node.label)}" + f"{node.rows}{node.reused}{node.not_stored}" + f"{html.escape(', '.join(node.inputs_incomplete))}" + "" + for node in self.nodes.values() + ] + return ( + f"

{html.escape(self._headline())}

" + f"{header}" + f"{''.join(body)}
" + ) + + +def _counts(node: NodeRunSummary, load_only: bool) -> str: + """A node's counts as text: what a full run did, or what a load-only run loaded.""" + if node.held and not (node.computed or node.reused or node.failed): + text = "held from the previous run" + if node.not_stored: + text += f" ({node.not_stored} inputs without a stored result)" + return text + if not load_only: + return ( + f"{node.computed} computed, {node.reused} from store, {node.failed} failed" + ) + text = f"{node.reused} loaded" + if node.not_stored: + text += f", {node.not_stored} not stored (skipped)" + if node.inputs_incomplete: + text += f" (inputs incomplete: {', '.join(node.inputs_incomplete)})" + return text + def _rows(n: int) -> str: """``1 row`` or ``n rows``.""" @@ -195,12 +282,41 @@ def _error_line(error: BaseException | None) -> str: return f"{type(error).__name__}: {first}" if first else type(error).__name__ -def _traceback_call(label: str) -> str: +def _traceback_call(label: str, target: str | None = None) -> str: """The call that prints a node's traceback, with the label as a valid string literal. - A JSON string is a valid Python string literal, and keeps double quotes. + A JSON string is a valid Python string literal, and keeps double quotes. A node + run's summary is kept on its node (``target``), a job run's on the job. """ - return f"job.last_run.print_traceback({json.dumps(label)})" + owner = "job" if target is None else f"job.nodes[{json.dumps(target)}]" + return f"{owner}.last_run.print_traceback({json.dumps(label)})" + + +def warn_row_failures(summary: RunSummary, caller: FrameType) -> None: + """Issue the run's ``RowFailuresWarning``, shown after every run with failed rows. + + Python's default filter shows a warning once per text and line, so a second identical + failing run from the same line (re-running a notebook cell) would show nothing. A + fresh registry for each run keeps every run's warning, while the user's filters + (``ignore``, ``error``) still apply. + + Args: + summary: The run's summary, with failed rows. + caller: The frame that called ``run``; the warning is shown at its line. + """ + import warnings + + from orcapod.errors import RowFailuresWarning + + warnings.warn_explicit( + summary.failures_message(), + RowFailuresWarning, + caller.f_code.co_filename, + caller.f_lineno, + module=caller.f_globals.get("__name__"), + registry=None, + module_globals=caller.f_globals, + ) class RunCountingObserver: @@ -289,3 +405,78 @@ def on_data_crash( def create_data_logger(self, tag: TagProtocol, data: DataProtocol) -> Any: """No logging of its own.""" return _NOOP_LOGGER + + +def run_node_summaries( + dag: Any, counter: RunCountingObserver, held: Mapping[Any, int] | None = None +) -> list[NodeRunSummary]: + """One summary per function node of a finished run, in run order. + + The counts come from ``counter``; a load-only run's skipped inputs come from each + node (``last_run_not_stored``), and every node below a function node that skipped + inputs names it in ``inputs_incomplete`` (ITL-1095). + + Args: + dag: The run's node graph (``topological_sort``, ``predecessors``). + counter: The observer that counted the run's rows. + held: Nodes of ``dag`` the run did not run because they hold a current run, + with their row counts: reported as held (ITL-1095). Inputs they skipped in + an earlier load-only run still make the nodes below them incomplete. + + Returns: + The summaries of the run's function nodes, in run order. + """ + from dataclasses import replace + + from orcapod.protocols.node_protocols import ( + is_function_node, + is_side_effect_function_node, + ) + + order = list(dag.topological_sort()) + position = {id(node): index for index, node in enumerate(order)} + skipping_above: dict[int, set[int]] = {} + summaries: list[NodeRunSummary] = [] + for node in order: + above: set[int] = set() + for upstream in dag.predecessors(node): + above |= skipping_above.get(id(upstream), set()) + if getattr(upstream, "last_run_not_stored", 0): + above.add(id(upstream)) + skipping_above[id(node)] = above + # A stream outside the job (in a node.run() graph) is no node of the run. + if getattr(node, "node_type", None) is None: + continue + if not (is_function_node(node) or is_side_effect_function_node(node)): + continue + held_rows = next( + (count for other, count in (held or {}).items() if other is node), None + ) + labels = tuple( + order[index].label for index in sorted(position[key] for key in above) + ) + label: str = getattr(node, "label") # every graph node is labelled + if held_rows is not None: + # A held node keeps what its run skipped: a load-only run's missing + # inputs are still missing. + summaries.append( + NodeRunSummary( + label, + 0, + 0, + 0, + None, + not_stored=getattr(node, "last_run_not_stored", 0), + inputs_incomplete=labels, + held=held_rows, + ) + ) + continue + summaries.append( + replace( + counter.node_summary(label), + not_stored=getattr(node, "last_run_not_stored", 0), + inputs_incomplete=labels, + ) + ) + return summaries diff --git a/src/orcapod/pipeline/sync_orchestrator.py b/src/orcapod/pipeline/sync_orchestrator.py index e6efa36a3..a02723c57 100644 --- a/src/orcapod/pipeline/sync_orchestrator.py +++ b/src/orcapod/pipeline/sync_orchestrator.py @@ -22,6 +22,8 @@ ) if TYPE_CHECKING: + from collections.abc import Mapping + from orcapod.pipeline.dag import GraphProtocol from orcapod.protocols.observability_protocols import ExecutionObserverProtocol from orcapod.protocols.core_protocols import DataProtocol, TagProtocol @@ -54,6 +56,8 @@ def run( materialize_results: bool = True, run_id: str | None = None, pipeline_uri: str = "", + load_only: bool = False, + held_rows: Mapping[Any, list[tuple[TagProtocol, DataProtocol]]] | None = None, ) -> OrchestratorResult: """Execute the node graph synchronously. @@ -68,6 +72,14 @@ def run( pipeline_uri: Opaque URI string identifying this pipeline run (e.g. ``"my_pipeline@a1b2c3d4e5f6a1b2"``). Forwarded verbatim to ``observer.on_run_start``. + load_only: Fill the nodes from what is stored and compute nothing + (ITL-1095): function nodes load their inputs' stored results and skip + the others, operators run without storing, and a side-effect node + passes its rows on and delivers nothing. + held_rows: Nodes of ``graph`` that are not run: each one's rows (the rows + of its current run, or of a stream outside the job) feed its + successors as if it had just run. Used by ``node.run()``, which runs + only the nodes it depends on that hold no current run (ITL-1095). Returns: OrchestratorResult with node outputs. @@ -85,9 +97,12 @@ def run( buffers: dict[Any, list[tuple[TagProtocol, DataProtocol]]] = {} processed: set[Any] = set() + held = held_rows if held_rows is not None else {} for node in topo_order: current = node - if is_source_node(node): + if node in held: + buffers[node] = list(held[node]) + elif is_source_node(node): buffers[node] = node.execute(observer=effective_observer) elif is_function_node(node) or is_side_effect_function_node(node): upstream_buf = self._gather_upstream(node, graph, buffers) @@ -100,6 +115,7 @@ def run( observer=effective_observer, error_policy=self._error_policy, run_id=run_id, + load_only=load_only, ) elif is_operator_node(node): upstream_buffers = self._gather_upstream_multi(node, graph, buffers) @@ -108,7 +124,7 @@ def run( for buf, upstream_node in upstream_buffers ] buffers[node] = node.execute( - *input_streams, observer=effective_observer + *input_streams, observer=effective_observer, load_only=load_only ) elif is_side_effect_node(node): upstream_buf = self._gather_upstream(node, graph, buffers) @@ -116,10 +132,12 @@ def run( input_stream = self._materialize_as_stream( upstream_buf, upstream_node ) + # In a load-only run the node passes its rows on and delivers nothing. buffers[node] = node.execute( input_stream, observer=effective_observer, run_id=run_id, + load_only=load_only, ) else: raise TypeError( diff --git a/src/orcapod/protocols/node_protocols.py b/src/orcapod/protocols/node_protocols.py index 4807a382b..0f7954a8c 100644 --- a/src/orcapod/protocols/node_protocols.py +++ b/src/orcapod/protocols/node_protocols.py @@ -88,6 +88,7 @@ def execute( observer: ExecutionObserverProtocol | None = None, error_policy: Literal["continue", "fail_fast"] = "continue", run_id: str | None = None, + load_only: bool = False, ) -> list[tuple[TagProtocol, DataProtocol]]: ... async def async_execute( @@ -98,6 +99,7 @@ async def async_execute( observer: ExecutionObserverProtocol | None = None, error_policy: Literal["continue", "fail_fast"] = "continue", run_id: str | None = None, + load_only: bool = False, ) -> None: ... def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: @@ -126,6 +128,7 @@ def execute( self, *input_streams: StreamProtocol, observer: ExecutionObserverProtocol | None = None, + load_only: bool = False, ) -> list[tuple[TagProtocol, DataProtocol]]: ... async def async_execute( @@ -134,6 +137,7 @@ async def async_execute( output: WritableChannel[tuple[TagProtocol, DataProtocol]], *, observer: ExecutionObserverProtocol | None = None, + load_only: bool = False, ) -> None: ... def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: @@ -157,6 +161,7 @@ def execute( *, observer: "ExecutionObserverProtocol | None" = None, run_id: str | None = None, + load_only: bool = False, ) -> "list[tuple[TagProtocol, DataProtocol]]": ... async def async_execute( @@ -166,6 +171,7 @@ async def async_execute( *, observer: "ExecutionObserverProtocol | None" = None, run_id: str | None = None, + load_only: bool = False, ) -> None: ... def attach_databases( diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 38526a201..cc9e05b16 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -18,7 +18,8 @@ from typing_extensions import override from orcapod.core.base import TraceableBase -from orcapod.core.nodes.function_node import _build_record_id_preimage +from orcapod.core.nodes.staleness import RunHoldingNode +from orcapod.core.nodes.function_node import _build_record_id_preimage, run_job_node from orcapod.core.streams.base import ( StreamBase, check_head_size, @@ -26,6 +27,7 @@ typed_zero_row_input, ) from orcapod.core.streams.view import ( + LOADED_ONLY_SIDE_EFFECT_NOTE, SIDE_EFFECTS_NOTE, ViewStatus, StreamView, @@ -995,7 +997,7 @@ def node_uri(self) -> tuple[str, ...]: # --------------------------------------------------------------------------- -class SideEffectJobNode(SideEffectNode): +class SideEffectJobNode(RunHoldingNode, SideEffectNode): """DB-backed execution node for side-effect pods. Created at pipeline compile time by ``PipelineJob``. Receives a @@ -1030,6 +1032,28 @@ def __init__( # The rows the last completed run passed on, in the order it passed them; # ``None`` when no run in this process completed (ITL-1091). self._held_rows: list[tuple[TagProtocol, DataProtocol]] | None = None + # Whether those rows came from a load-only run, which delivers nothing. + self._run_load_only = False + # A node that has not run in this process holds nothing and is stale (ITL-1095). + self._set_modified_time(None) + + def _holds_run(self, *, load_only: bool) -> bool: + """Whether the node holds a run's rows; a load-only run's only for ``load_only``. + + A load-only run delivers nothing, so a full run does not take its rows as the + result of a delivery. + """ + return self._held_rows is not None and (load_only or not self._run_load_only) + + def clear_cache(self) -> None: + """Drop the rows the last run passed on: the node is stale and reads empty. + + Its next ``run()`` (or job run) delivers again what its invocation log does not + record (every row without ``track_completion``) (ITL-1095). + """ + self._held_rows = None + self._run_load_only = False + self._set_modified_time(None) def _refuse_log_of_another_data_context(self) -> None: """Refuse an invocation log holding deliveries of another data context. @@ -1116,6 +1140,7 @@ def execute( *, observer: ExecutionObserverProtocol | None = None, run_id: str | None = None, + load_only: bool = False, ) -> list[tuple[TagProtocol, DataProtocol]]: """Execute side-effect delivery for all rows in ``input_stream``. @@ -1123,6 +1148,8 @@ def execute( input_stream: Stream of ``(tag, data)`` pairs to process. observer: Optional execution observer (currently unused). run_id: Pipeline run identifier forwarded from the orchestrator. + load_only: A load-only run (ITL-1095): every row passes on and nothing is + delivered or logged. The node keeps the rows it passed on, which every later read returns without delivering (ITL-1091). A run that raises leaves the node with no rows. @@ -1135,6 +1162,24 @@ def execute( logged under another data context; nothing is delivered then. """ self._held_rows = None + if load_only: + rows = list(input_stream.iter_data()) + self._hold(rows, load_only=True) + return rows + try: + results = self._deliver_rows(input_stream, run_id=run_id) + except BaseException: + # A run that raised defines no rows: the node is stale, and so is every + # node below it (ITL-1095). + self.clear_cache() + raise + self._hold(results) + return results + + def _deliver_rows( + self, input_stream: StreamProtocol, *, run_id: str | None + ) -> list[tuple[TagProtocol, DataProtocol]]: + """Deliver every row of ``input_stream``; return the rows passed on.""" self._refuse_log_of_another_data_context() results: list[tuple[TagProtocol, DataProtocol]] = [] for tag, data in input_stream.iter_data(): @@ -1154,16 +1199,22 @@ def execute( ) if result is not None: results.append(result) - self._hold(results) return results - def _hold(self, rows: list[tuple[TagProtocol, DataProtocol]]) -> None: + def _hold( + self, rows: list[tuple[TagProtocol, DataProtocol]], *, load_only: bool = False + ) -> None: """Keep the rows a completed run passed on and stamp the node as modified. Stamped after the upstream nodes, which stamp at the end of their own run, so the rows are current until an upstream changes again (``is_stale``). + + Args: + rows: The rows the run passed on. + load_only: Whether the run was load-only (it delivered nothing). """ self._held_rows = rows + self._run_load_only = load_only self._update_modified_time() def _current_rows(self) -> list[tuple[TagProtocol, DataProtocol]] | None: @@ -1221,27 +1272,44 @@ def head( return self._head_without_rows( config, ViewStatus.ROWS, note="its last run passed on no rows" ) - return StreamView.of_rows( + view = StreamView.of_rows( self, self.as_table(columns=config).slice(0, n), tag_columns=self.keys(columns=config)[0], total_rows=len(rows), origin="held", ) + return self._with_run_status(view) + + def _with_run_status(self, view: StreamView) -> StreamView: + """``view`` with the status of the node's last run: ``loaded_only`` after a + load-only run, which passed the rows on and delivered nothing (ITL-1095).""" + if not self._run_load_only: + return view + return dataclasses.replace( + view, status=ViewStatus.LOADED_ONLY, note=LOADED_ONLY_SIDE_EFFECT_NOTE + ) @override - def run(self) -> StreamView: + def run(self, *, load_only: bool = False) -> StreamView: """Run the node as ``job.run()`` runs it and return a ``StreamView`` of its rows. - Reads the input and delivers through the invocation log, as a job run does: - with ``track_completion`` only the rows the log does not record as delivered are - delivered; without it every row is delivered again, as a second ``job.run()`` - delivers it. With no database attached nothing is logged and every row is - delivered. The node then holds the rows it passed on, which ``as_table()`` and - every other read return without delivering. The invocation log is committed - before ``run()`` returns, as at the end of a job run; after a failed delivery - the log rows of the deliveries that succeeded are committed too, and the - delivery's error is raised. + State-driven, as every job node's ``run()`` (developer, November 13, 2026; + ITL-1095): a node that holds the rows of a completed, non-stale run delivers + nothing and returns them; call ``clear_cache()`` first to deliver again. + Otherwise the job nodes it depends on that hold no current run run first, and + the node reads its input and delivers through the invocation log, as a job run + does: with ``track_completion`` only the rows the log does not record as + delivered; without it every row. With no database attached nothing is logged + and every row is delivered. The node then holds the rows it passed on, which + ``as_table()`` and every other read return without delivering. The run's writes + (the invocation log included) are committed before ``run()`` returns; after a + failed delivery the log rows of the deliveries that succeeded are committed too, + and the delivery's error is raised. + + Args: + load_only: Deliver nothing: the rows pass on, and the nodes run first load + their stored results only (as ``job.run(load_only=True)``). Returns: A ``StreamView`` of the rows this run passed on. @@ -1250,9 +1318,22 @@ def run(self) -> StreamView: SideEffectLogDataContextError: As for ``execute``. StoreWriteError: If committing the invocation log fails. """ - database = self._pipeline_database + import sys + + run_job_node( + self, + load_only=load_only, + run_alone=lambda: self._run_alone(load_only=load_only), + caller=sys._getframe(1), # pyright: ignore[reportPrivateUsage] # the caller's line is where a failure warning is shown + ) + return self._with_run_status(self._view_of_rows(self.as_table(), "held")) + + def _run_alone(self, *, load_only: bool) -> None: + """Run this node over an input stream that is not a job node; commit its log.""" + # A load-only run writes nothing, so it commits nothing (ITL-1095). + database = None if load_only else self._pipeline_database try: - _ = self.execute(self._input_stream) + _ = self.execute(self._input_stream, load_only=load_only) except Exception as run_error: if database is not None: flush_best_effort(run_error, database) @@ -1264,12 +1345,11 @@ def run(self) -> StreamView: except StoreReadError: # A stored table could not be read: report the read failure itself. # The run raised, so it leaves the node no rows. - self._held_rows = None + self.clear_cache() raise except Exception as exc: - self._held_rows = None + self.clear_cache() raise StoreWriteError(database.base_path, flush=True) from exc - return self._view_of_rows(self.as_table(), "held") # ------------------------------------------------------------------ # Async execution @@ -1282,6 +1362,7 @@ async def async_execute( *, observer: ExecutionObserverProtocol | None = None, run_id: str | None = None, + load_only: bool = False, ) -> None: """Async side-effect delivery with semaphore-bounded concurrency. @@ -1302,6 +1383,8 @@ async def async_execute( output: Writable channel for pass-through ``(tag, data)`` pairs. observer: Optional execution observer (currently unused). run_id: Pipeline run identifier from the orchestrator. + load_only: A load-only run (ITL-1095): every row passes on and nothing is + delivered or logged. Raises: ValueError: If ``inputs`` does not contain exactly one channel. @@ -1311,12 +1394,26 @@ async def async_execute( f"SideEffectJobNode.async_execute expects exactly 1 input channel; " f"got {len(inputs)}." ) + self._held_rows = None + # A load-only run reads no pod configuration: it delivers nothing. + if load_only: + try: + rows: list[tuple[TagProtocol, DataProtocol]] = [] + async for item in inputs[0]: + rows.append(item) + await output.send(item) + # Held and stamped before the output closes (ITL-1042). + self._hold(rows, load_only=True) + except BaseException: + self.clear_cache() + raise + finally: + await self._close_output(output) + return max_concurrency = self._pod.pod_config.max_concurrency sem = ( asyncio.Semaphore(max_concurrency) if max_concurrency is not None else None ) - - self._held_rows = None # This run's rows, in the order they were sent downstream. passed_on: list[tuple[TagProtocol, DataProtocol]] = [] # Makes recording a row and sending it one step: a send blocked by a full channel @@ -1358,5 +1455,19 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: # Held and stamped before the output closes, so every downstream node, which # stamps after it has seen that close, is newer than this one (ITL-1042). self._hold(passed_on) + except BaseException: + # A run that raised or was cancelled defines no rows (ITL-1095). + self.clear_cache() + raise finally: + await self._close_output(output) + + async def _close_output( + self, output: WritableChannel[tuple[TagProtocol, DataProtocol]] + ) -> None: + """Close ``output``; a close that fails fails the run, which then holds no rows.""" + try: await output.close() + except BaseException: + self.clear_cache() + raise diff --git a/superpowers/plans/2026-11-13-itl-1095-node-current-run-view.md b/superpowers/plans/2026-11-13-itl-1095-node-current-run-view.md new file mode 100644 index 000000000..24aae79a1 --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1095-node-current-run-view.md @@ -0,0 +1,196 @@ +# ITL-1095: a job node shows the current run; load-only run + +Status: all four stages implemented on draft PR #380 (stage 1 da66b836, stage 2 c90eb966, stage 3 b693b50e, stage 4 3610fd20). Next: wait for #378 to merge, merge main, CI, CodeRabbit. +Base: PR #378 branch head `2e9972e0` (ITL-1093). Branch: `eywalker/itl-1095-design-what-a-job-node-shows-the-current`. + +Rule (developer, November 13, 2026): a job node itself (`as_table`, `iter_data`, dataframe +methods, `flow`, `head`, the preview, `run`) is the CURRENT RUN view; `node.as_source()` is the +view of everything stored for the node. Before a run a node is empty; a run fills it, also when +everything is loaded. `run(load_only=True)` fills nodes from what is stored and computes nothing. +`node.run()` cascades to the job nodes it depends on (developer, later the same day). + +## 1. Measurements on the base (`2e9972e0`) + +Scratch scripts: `measure.py`, `replay.py`, `noderun.py` in the session scratchpad. +Pipeline: `DictSource(source_id="s", tags g,k)` -> `inc` (`rowwise`) -> `group_by("g")` +(`grouped`, operator) -> `total` (`summed`). Runs over 6 rows then 3 rows, one Delta store. + +| Read | sync | async | +|---|---|---| +| after run over 3 rows: rowwise / grouped / summed `as_table` | 3 / 2 / 2 | 3 / 2 / 2 | +| `as_source()` of rowwise / summed | 6 / 4 | 6 / 4 | +| fresh job (3 source rows), NOT run: rowwise / grouped(OFF) / summed `as_table` | **6** / 0 / **4** | same | +| fresh job, not run: `head()` rowwise | `rows`, 6 of 6 (stored) | same | +| fresh job: `summed.run()` alone (upstream OFF operator not run) | 0 rows, nothing computed | - | +| fresh job: source node `as_table` | 3 (bound source) | same | + +Operator node `grouped` before a run (fresh job over a store with two earlier runs): + +| cache mode | `as_table` | `head` | `as_source` | +|---|---|---|---| +| OFF | 0 | not_computed "not run; cache mode OFF stores nothing" | 0 | +| LOG | 0 | not_computed "rows of earlier runs are stored but not replayed" | 4 (both runs) | +| REPLAY (after two LOG runs) | **4** (both runs) | rows 4 | 4 | + +REPLAY after a run over 3 rows (sync and async alike): `grouped` holds **4 groups** +`(0,[1,21]) (0,[1,21,41]) (1,[11]) (1,[11,31,51])` -- every logged run, not the current run -- +and `summed` downstream gets 4 rows. The operator cache (`odb_v1`) stores no run id or time, so +"the latest logged run" cannot be told apart without a new stored column. + +`node.run()` with an upstream job node that has not run (chain `a` -> `b`, store holds a 6-row run, +source now 3 rows): `b.run()` reads `a`'s STORED rows (6, not the current 3) and returns 6 rows, +0 function calls; over an empty store it returns 0 rows silently. `node.run()` never runs upstream. + +Loaded job (`PipelineJob.load`, in-memory source unbound): function nodes are `READ_ONLY` stubs +showing every stored row (rowwise 6, summed 4); the OFF operator raises `RuntimeError` from +`as_table` (head: unavailable); source node raises `UnboundSourceError`; `loaded.run()` raises. +Loaded `rowwise.as_source()` reads 6 rows. + +Load statuses: a built job's function nodes are always `FULL`. A loaded node without a pod is +`READ_ONLY` (store attached) or `UNAVAILABLE`. `CACHE_ONLY` and `READ_ONLY` with a +`DataFunctionProxy` arise only from `FunctionJobNode.from_descriptor(..., function_pod=...)`, +which no public path calls (`PipelineJob.load` passes no pod). + +Orchestrators: both pass rows between nodes in their own buffers/channels; no node read is +involved in `job.run()`. Only `node.run()` reads its upstream node. + +Experiment (reverted): making every un-run read empty fails 400 of 4,464 job-related tests in +21 files (list in section 5). + +## 2. Decisions (coordinator, November 13, 2026; proposal in 2026-11-13-itl-1095-proposal.md) + +- D1 REPLAY: R1 for the tag (after a run: union of every logged run; only the before-run read + becomes empty). ASKED of the developer. +- D2 (a): load-only uses only the node's own pipeline records; result-store-only hits are not stored. +- D3 (a)+(i): operators run on the rows that reach them (LOG writes nothing); summary and node + status mark every node below a skipped input "inputs incomplete". +- D4: orchestrator passes side-effect node rows through uncalled in load-only; node reads empty. +- D5: NodeRunSummary.not_stored, .inputs_incomplete; RunSummary.load_only; loaded rows in reused; + repr "rowwise: 6 rows: 3 loaded, 3 not stored (skipped)", "summed: 2 rows: 2 loaded (inputs incomplete: rowwise)". +- D6: amend ITL-1081 rule: run() takes no positional arguments; a job node's run() (and + job.run/run_async) takes keyword-only load_only; non-job streams' run() stays argument-less. +- D7 + developer: state-driven run. A node (itself or ancestor) that holds a completed run and is + not stale does not run again: run() returns its held view. Staleness = the existing + modified-time mechanism; a freshly built or loaded job has every node stale. Load-only-filled + is its own status (ViewStatus e.g. LOADED_ONLY, caption "N loaded, M inputs without a stored + result"), never complete: a full run re-runs it in full. A load-only run that finds everything + stays loaded-only by default (ASKED). Cascade = ancestor sub-DAG on the sync orchestrator, one + mechanism shared with job.run. Summary on node.last_run. Commits per row + flush at end. + job.run() behaviour NOT changed (give options). Side-effect re-delivery with + track_completion=False: clear_cache() then run(). Table-backed/polling sources changing without + modified time: known limit, documented. +- D8 loaded jobs as recommended. D9 as recommended. D10 remove NOTHING_STORED. D11 as recommended. + +Order: 1) empty before a run for every node kind + test updates; 2) load_only on job.run/run_async ++ summary; 3) node.run cascade + state rule + status; 4) loaded jobs. Each stage its own commit, +report counts after each. Async tests also on 3.11. Golden unchanged. + +## 3. Implementation sketch (after approval) + +- `FunctionJobNode`: no-run read empty (all load statuses); `head` NOT_COMPUTED with note; + `execute`/`async_execute` take `load_only` (lookup `get_cached_results` only, skip misses, count + them, no `_process_data_internal`, no writes); `_last_run_skipped` count; run mode flag. +- `OperatorJobNode`: REPLAY before run empty; load-only: OFF/LOG run the operator, LOG writes + nothing; REPLAY replays. +- Orchestrators: `load_only` threaded through; side-effect node passes rows through uncalled. +- `PipelineJob.run/run_async(*, load_only=False)`; loaded job allowed with `load_only=True`. +- One internal "run these job nodes" function shared by `job.run` and the `node.run` cascade. +- `RunSummary`/`NodeRunSummary` new field(s). +- Tests first (public interface, both orchestrators, 3.11 for async). +- Docs, changelog fragment, design_issues entry, CLAUDE.md + .zed/rules. + +## 5. Tests and docs that change + +Failing under the experiment (count per file): test_node_head_sorting 162, +test_loaded_operator_cache 74, test_function_node_arrow_types 58, test_loaded_job_readback 29, +test_delta_arrow_type_roundtrip 25, test_node_head 9, test_loaded_logical_schemas 7, +test_provisional_hash_warning 6, test_function_pod_node_stream 5, test_stored_data_context 4, +test_function_node_empty_data 4, test_job_node_rows_after_run 3, test_tensor_job_load_back 2, +test_materialize_source_info 2, test_function_node_caching 2, test_cached_function_pod_job 2, +test_doc_samples (function-pods.md) 1, test_operator_node 1, test_operator_node_non_active 1, +test_function_node_iteration 1, test_readback_datagram_id 1, test_polars_dataframe_values 1. +Plus `tests/test_core/streams/test_run.py::test_every_stream_class_has_run_without_arguments` +(a `load_only` keyword on a node's `run`). Full names: scratchpad `hack_failed_tests.txt`. + +Docs: docs/concepts/streams.md (run table row "A job node", head table row "A job node", the +"stream that has all of its rows" paragraph, line ~262), docs/concepts/function-pods.md (339-341 +no-run read; "Saving and loading a job" 662-746 and its sample at 689-694, the only doc sample +that reads a node without a run), docs/concepts/sources.md 169-172; docstrings of +`PipelineJob.load`/`run`, `FunctionJobNode.head/as_table/iter_data/run/clear_cache/_cannot_compute_error`, +`OperatorJobNode.iter_data/as_table/head`, `ViewStatus.NOTHING_STORED`, `StreamBase.head`. + +### Appendix: tests failing under the empty-before-run experiment (parametrizations collapsed) + +``` +tests/test_core/function_pod/test_cached_function_pod_job.py::test_saved_job_equals_the_uncached_pods_and_loads +tests/test_core/function_pod/test_function_node_arrow_types.py::test_store_read_back_keeps_tag_type +tests/test_core/function_pod/test_function_node_caching.py::TestPhase1Phase2PipelineEntryId::test_all_existing_entry_ids_skipped_in_phase2 +tests/test_core/function_pod/test_function_node_caching.py::TestPhase1Phase2PipelineEntryId::test_phase1_yields_existing_results +tests/test_core/function_pod/test_function_pod_node_stream.py::TestFunctionNodeInactive::test_as_table_returns_cached_results_when_data_function_inactive +tests/test_core/function_pod/test_function_pod_node_stream.py::TestFunctionNodeInactive::test_inactive_filled_db_serves_cached_results +tests/test_core/function_pod/test_function_pod_node_stream.py::TestFunctionNodeStaleness::test_as_table_auto_detects_stale_and_repopulates +tests/test_core/function_pod/test_function_pod_node_stream.py::TestFunctionNodeStaleness::test_clear_cache_produces_same_results_on_re_iteration +tests/test_core/function_pod/test_function_pod_node_stream.py::TestIterDatasDbPhase::test_db_served_results_have_correct_row_count +tests/test_core/function_pod/test_polars_dataframe_values.py::TestPipelineStores::test_saved_job_loaded_back_reads_the_frames +tests/test_core/function_pod/test_readback_datagram_id.py::test_a_fresh_job_on_the_store_reads_the_stored_ids +tests/test_core/nodes/test_function_node_empty_data.py::TestAsyncProcessDataInternalEmptyData::test_async_empty_data_cache_hit_returns_cached_result +tests/test_core/nodes/test_function_node_empty_data.py::TestAsyncProcessDataInternalEmptyData::test_async_empty_data_cache_miss_raises +tests/test_core/nodes/test_function_node_empty_data.py::TestProcessDataInternalEmptyData::test_cache_hit_returns_cached_result +tests/test_core/nodes/test_function_node_empty_data.py::TestProcessDataInternalEmptyData::test_cache_miss_raises_ephemeral_result_missing_error +tests/test_core/nodes/test_function_node_iteration.py::TestIterDatasReadOnly::test_iter_after_db_populated_hot_loads_without_compute +tests/test_core/operators/test_materialize_source_info.py::TestStoreWrittenBeforeFix::test_rerun_after_pre_fix_store +tests/test_core/operators/test_materialize_source_info.py::TestStoreWrittenBeforeFix::test_v014_store_is_not_read +tests/test_core/operators/test_operator_node_non_active.py::TestCacheModeNonActive::test_iter_data_replay_mode_returns_db_contents +tests/test_core/operators/test_operator_node.py::TestOperatorNodeRunAndStorage::test_replay_from_cache +tests/test_databases/test_delta_arrow_type_roundtrip.py::test_a_fresh_job_reads_the_tag_type +tests/test_databases/test_delta_arrow_type_roundtrip.py::test_a_loaded_job_reads_the_tag_type +tests/test_databases/test_delta_arrow_type_roundtrip.py::test_a_rerun_on_the_same_store_writes_no_pipeline_rows +tests/test_databases/test_delta_arrow_type_roundtrip.py::test_an_operator_replay_reads_the_tag_type +tests/test_docs/test_doc_samples.py::test_doc_page_samples_run +tests/test_logical_types/test_provisional_hash_warning.py::test_reading_stored_results_of_the_type_warns_once +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_binding_a_store_makes_a_storeless_loaded_node_readable +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_datetime_output_feeding_an_operator_loads_with_its_type +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_file_without_the_expected_rows_section_reads_without_a_check +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_latest_readable_result_wins_across_both_stores +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_live_node_hot_load_reads_the_latest_recomputed_result +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_loaded_function_node_as_table_all_info_has_stream_columns +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_loaded_function_node_as_table_returns_stored_rows +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_loaded_function_node_iter_data_yields_stored_rows +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_loaded_node_with_ephemeral_results_does_not_crash +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_loaded_operator_node_raises_instead_of_returning_empty +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_logical_type_output_schema_fails_naming_the_column_and_saved_type +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_newer_unavailable_result_does_not_expose_old_persistent_result +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_other_context_records_do_not_satisfy_saved_expectation +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_path_output_feeding_join_still_allows_job_load +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_recomputed_entry_is_read_once_with_its_latest_result +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_saving_a_loaded_job_keeps_the_expected_rows +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_store_content_warning_names_a_delta_stores_location +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_store_with_only_another_runs_records_reads_none_with_a_warning +tests/test_pipeline/loaded_job/test_loaded_job_readback.py::test_unreconstructable_file_source_is_a_warning_with_the_reason +tests/test_pipeline/loaded_job/test_loaded_logical_schemas.py::test_registered_logical_output_keeps_join_available +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_a_format_0_1_0_file_finds_its_join_table_and_never_shows_its_rows_as_none +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_a_loaded_run_with_an_empty_result_reads_as_empty +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_a_pipeline_loaded_and_bound_to_its_sources_reuses_the_operator_cache +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_a_store_holding_only_other_runs_rows_gives_an_empty_result +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_an_absent_cache_table_gives_an_empty_result +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_fewer_rows_than_expected_warn_once_and_read_what_the_store_holds +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_get_cached_output_of_a_loaded_node_never_caches_unreadable_rows_as_none +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_loaded_replay_operator_reads_the_rows_its_run_stored +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_matching_counts_and_zero_expected_zero_found_do_not_warn +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_more_rows_than_expected_warn_and_read_them_all +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_without_saved_nullability_the_operator_still_reads_its_runs_table +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_without_the_run_content_hash_an_absent_table_raises_too +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_without_the_run_content_hash_head_says_why_it_shows_no_rows +tests/test_pipeline/loaded_job/test_loaded_operator_cache.py::test_without_the_run_content_hash_the_loaded_operator_raises +tests/test_pipeline/test_job_node_rows_after_run.py::TestRaisedRun::test_raised_run_reads_as_no_run +tests/test_pipeline/test_node_head_sorting.py::test_a_function_nodes_head_is_the_prefix_of_as_table_in_any_completion_order +tests/test_pipeline/test_node_head_sorting.py::test_an_operator_nodes_head_is_the_prefix_of_as_table +tests/test_pipeline/test_node_head.py::test_a_loaded_job_shows_stored_rows_and_never_raises +tests/test_pipeline/test_node_head.py::test_a_new_job_over_a_filled_store_reads_stored_rows_without_computing +tests/test_pipeline/test_node_head.py::test_head_keeps_the_tag_arrow_type_of_the_input +tests/test_pipeline/test_node_head.py::test_nodes_before_a_run_show_no_rows_and_compute_nothing +tests/test_pipeline/test_node_head.py::test_operator_node_cache_modes +tests/test_pipeline/test_stored_data_context.py::TestRowsOfAnotherDataContext::test_function_node_reads_current_rows_only +tests/test_pipeline/test_stored_data_context.py::TestSideEffects::test_function_node_upstream_of_a_refused_sink_is_not_refused +tests/test_torch/test_tensor_job_load_back.py::test_a_loaded_job_reads_its_stored_tensors +``` diff --git a/superpowers/plans/2026-11-13-itl-1095-proposal.md b/superpowers/plans/2026-11-13-itl-1095-proposal.md new file mode 100644 index 000000000..4478eaed4 --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1095-proposal.md @@ -0,0 +1,196 @@ +# ITL-1095 proposal (sent to the coordinator for decisions) + +Measurements: `2026-11-13-itl-1095-node-current-run-view.md` section 1 (same folder). +Recommendations are marked **(rec)**. Decisions needed: D1 to D11. + +## A. The rule as built, per node kind + +| Node | Before a run in this process (every read) | After a run | +|---|---|---| +| `FunctionJobNode` (FULL; the only state a built job has) | zero rows, typed columns; `head`/preview status `not_computed`, note below | the run's rows (#378); a load-only run: the loaded rows | +| same, after a run that raised / `clear_cache()` / stale (upstream re-ran or was rebound) | empty, as before a run (today: every stored row) | - | +| loaded function node (`READ_ONLY` stub) | empty (today: every stored row) | after `loaded.run(load_only=True)`: loaded rows | +| `CACHE_ONLY`, `READ_ONLY` with a proxy pod (only via `from_descriptor(..., function_pod=...)`, no public path) | empty | D9 | +| `OperatorJobNode` OFF / LOG | empty (unchanged: already so) | run's output (unchanged) | +| `OperatorJobNode` REPLAY | empty (today: union of every logged run) | D1 | +| loaded operator node | empty, no raise (today OFF raises `RuntimeError`) | after a load-only run: its output on the loaded rows | +| `SideEffectJobNode` | #377 (not touched): reads deliver nothing and return zero rows when nothing is held. Already fits the rule. | #377 | +| `SourceJobNode` | the bound source's rows, before and after (a source is not computed); unbound raises as today. **Yes, unchanged.** | same | + +`node.as_source()` unchanged: every stored record of the node (current data context). + +Head/preview note before a run (D10) **(rec)**: status `not_computed`, header like +`FunctionJobNode[rowwise] — not run; run() fills it, run(load_only=True) loads stored results only; node.as_source() shows every stored result`. +The stored count is NOT available without reading rows (the database protocol has no count; today's +head reads the whole stored slice to show stored rows). Exception: a loaded job whose file records +`expected_rows` for the node: "the saved run stored N rows". **(rec)** no count otherwise, and +`ViewStatus.NOTHING_STORED` then has no producer: remove it (greenfield) **(rec)**, or keep. + +## B. Load-only run: `job.run(*, load_only=False)`, `job.run_async(*, load_only=False)`, `node.run(*, load_only=False)` + +Keyword-only, default False. + +- Sources are read. +- Function node: computes each input's entry id, looks up the node's OWN stored entries (pipeline + records under its content hash, the lookup a run does today), forwards only the rows it loaded, + skips and counts inputs without one. No user function call, no side effect, no write to result + or pipeline tables. An entry whose result was ephemeral and is gone counts as not stored. + - **D2**: an input with no pipeline record for this node but a matching result in the + function's result store (same function in another node, or a source renamed with the same + rows; a normal run "reuses" it and writes a pipeline record). (a) skip it, count it as not + stored **(rec)**: what the node shows is then always a subset of `as_source()`. (b) show it + without writing a record. +- Operator node, **D3**: (a) OFF and LOG run their operator on the rows that reach them (cheap, + not stored; LOG writes nothing in a load-only run), REPLAY replays as in a normal run + **(rec)**; (b) only cached operators produce rows: OFF (the only mode a job sets) forwards + nothing, so everything below any operator is empty: load-only becomes useless for + `group_by`, `join`, `select` pipelines. + - GroupBy hazard under (a): if the function node above lost inputs (not stored), the groups are + PARTIAL; a partial group's input differs from the full group's, so the function below misses + -- unless an earlier, smaller run stored exactly that partial group. Measured case: store has + the 3-row run, source now 6 rows, `rowwise` stored for 3 of 6 -> load-only groups 3 rows -> + `summed` HITS the 3-row run's totals (0,22),(1,11) and shows them as this run's. Options: + (i) the summary marks every node downstream of a node with skipped inputs as "inputs + incomplete (rowwise skipped 3)" and the docs say so **(rec for the tag)**; (ii) a node below + a node with skipped inputs forwards nothing (strict; also blanks row-wise chains, which are + exact); (iii) nothing. +- Side-effect node (D4): **(rec)** the orchestrator passes its input rows through without calling + it (no delivery, no log write; no change to `side_effects.py`, which #377 owns); the node then + holds nothing and reads empty. After #377 merges it could hold the passed rows instead. + A ctx_arg function node is a function node: loads, calls nothing. +- Both orchestrators: same per-node behaviour; `load_only` passed through `run`/`run_async` to + `execute`/`async_execute`. +- (d) `error_policy`: nothing to apply it to (no function runs); store read errors and + `TypeIdentityError` raise as in any run, operator errors raise as today. Observers: + `on_run_start/end`, `on_node_start/end` per node; a loaded row gets `on_data_start` + + `on_data_end(cached=True)`; a skipped input fires no row hook (no observer protocol change; + the node counts it and the job reads the count). A user-supplied status/logging observer writes + its own observability rows as in any run (not result/pipeline tables) -- say if load-only should + suppress them. +- **D5 RunSummary shape (rec)**: `NodeRunSummary` gains `not_stored: int = 0` (inputs skipped + because no result is stored, load-only only) and `inputs_incomplete: tuple[str, ...] = ()` + (upstream nodes that skipped inputs, D3(i)); loaded rows count in the existing `reused` ("read + from the store"), `computed` and `failed` are 0; `rows` = computed + reused + failed + + not_stored; `RunSummary.load_only: bool`. Repr line: + `rowwise: 6 rows: 3 loaded, 3 not stored (skipped)` and + `summed: 2 rows: 2 loaded (inputs incomplete: rowwise)`. Alternative: a separate `loaded` field. + Summary nodes stay function nodes only, as today. +- (f) Hashes: none move (no identity, path or record-id change). A load-only run writes nothing: + tests snapshot every table path and row count of the store before and after, both + orchestrators, including the operator cache in LOG and a side-effect log. + +## C. `node.run()` cascade (developer's rule) + +- Today (measured): `node.run()` never runs upstream. With an un-run upstream function node it + reads the upstream's STORED rows (6, though the source has 3 now) and processes them; over an + empty store, or behind an un-run OFF operator, it processes nothing and returns 0 rows silently. +- **(rec)** `node.run(*, load_only=False)` runs the sub-graph of the job nodes it depends on plus + itself, in dependency order, each once (diamonds), with ONE mechanism shared with `job.run`: an + internal function that takes a node DAG and runs it on the `SyncPipelineOrchestrator` + (`job.run` passes its whole DAG; `node.run` the ancestor DAG found by walking `upstreams`, + which in a job are the job nodes). Same rows, stored records, error_policy ("continue") and + per-node counts as `job.run` for that sub-graph. Sources are read, not run. +- **D7 "not necessarily trigger upstreams"** **(rec)**: an ancestor that holds a CURRENT run is + not run again; its held rows feed its successors. Current = it holds a completed run in this + process and `is_stale` is False (the existing judgement: own `last_modified` set and no upstream + or producer modified later; a re-run or rebound upstream makes it stale) and, for a full + cascade, that run was not load-only. So after a load-only fill, a full `node.run()` re-runs + every load-only-filled ancestor in full first (computes their missing entries) **(rec, + matches your expectation)**; option (b): use the loaded rows as they are. A load-only cascade + accepts a fully-run ancestor as current (its rows are all stored). +- Side-effect ancestors: a full cascade delivers exactly as `job.run` (with `track_completion` + only the undelivered); a load-only cascade delivers nothing (passes rows through). An ancestor + side-effect node that holds a current run (#377) is not run again, so nothing is re-delivered. +- Counts: `node.run()` still returns the node's `StreamView`. **(rec)** the cascade's + `RunSummary` is kept on the node as `node.last_run` (mirrors `job.last_run`), and a cascade + with failed rows issues the same `RowFailuresWarning` as `job.run`. Option: attach it to the + returned view (`view.run_summary`). +- Commits: as a node run on its own does today (every row) plus a flush at the end. Option: the + job's `commit_every_rows=100` cadence. +- **D6 conflict with a standing rule**: CLAUDE.md (ITL-1081, developer) "Every stream, source and + node has `run()` with no arguments ... Never give `run()` arguments", enforced by + `tests/test_core/streams/test_run.py::test_every_stream_class_has_run_without_arguments`. + **(rec)** amend to: "`run()` takes no positional arguments; a job node's `run()` takes one + keyword-only argument, `load_only`" and narrow the test to that. Option: `load_only` on + `job.run` only, and `node.run_load_only()`... (not recommended). + +## D. Loaded jobs (`PipelineJob.load`) + +- Every loaded node is empty before a run (function stubs and operators alike; operators no + longer raise from a read). `node.as_source()` reads everything stored (measured: loaded + `rowwise.as_source()` = 6 rows), with or without sources. +- `loaded.run(load_only=True)` **(rec)**: allowed. Sources come from the file (CSV, Delta) or + `loaded.bind(sources=...)` (in-memory sources are not saved). Nodes below an unbound source are + excluded and stay empty, as in any partial run. A function stub can do a load-only run: entry + ids need no function pod (system tags + input data hash), and the stub reads through its stored + paths and the saved run's content hash. Operators are rebuilt from the blueprint and run. + Without any live source there are no inputs to look up: a load-only run is impossible below + that source, and `as_source()` is the only way in. +- `loaded.run()` without `load_only`, and a loaded `node.run()`: raise as today, message pointing + to `run(load_only=True)` and `as_source()`. Loaded `node.run(load_only=True)`: the cascade. +- D8: confirm (rec) above. Risk: the stub's lookups use the saved run's content hash; a bound + source with another `source_id` finds nothing (all inputs counted not stored). + +## E. Other load statuses (D9) + +`CACHE_ONLY` (no upstream) and `READ_ONLY` with a proxy pod are reachable only through +`FunctionJobNode.from_descriptor(..., function_pod=...)` (tests only). Today their `run()` returns +every stored row. **(rec)** both empty before a run; `READ_ONLY` proxy: `run(load_only=True)` +loads, `run()` raises (cannot compute); `CACHE_ONLY`: no input, so `run(load_only=True)` raises +pointing to `as_source()`. (Candidates for removal later; out of scope.) + +## F. REPLAY (D1, not decided by me) + +Measured: after a run over 3 rows a REPLAY node holds the union of every LOG run (4 groups, two +from a 6-row run) and feeds it downstream. The rule forces only the BEFORE-a-run part (empty). +Options: (R1) REPLAY after a run keeps meaning "the stored output instead of computing": the +union of every logged run under its content hash; only the before-run read changes **(minimal +for the tag)**. (R2) REPLAY replays only the latest logged run: `odb_v1` stores no run id or time, +so this needs a new stored column -> layout change (`odb_v2`), not for the tag. (R3) REPLAY +replays only the logged rows of THIS run's inputs: operator outputs have no input entry ids, so +not possible without a stored-schema change either. + +## G. Tests and docs that change (e) + +Experiment (every no-run read empty, reverted): 400 of 4,464 job-related tests fail in 21 files: +test_node_head_sorting 162, test_loaded_operator_cache 74, test_function_node_arrow_types 58, +test_loaded_job_readback 29, test_delta_arrow_type_roundtrip 25, test_node_head 9, +test_loaded_logical_schemas 7, test_provisional_hash_warning 6, test_function_pod_node_stream 5, +test_stored_data_context 4, test_function_node_empty_data 4, test_job_node_rows_after_run 3 +(#378's `test_raised_run_reads_as_no_run`), test_tensor_job_load_back 2, +test_materialize_source_info 2, test_function_node_caching 2, test_cached_function_pod_job 2, +test_operator_node 1, test_operator_node_non_active 1, test_function_node_iteration 1, +test_readback_datagram_id 1, test_polars_dataframe_values 1, test_doc_samples +(function-pods.md) 1; plus test_run.py's no-argument test (D6) and the loaded-job "cannot run" +tests (test_loaded_job_still_cannot_run, ..._operator_only_job_cannot_run, +..._without_a_store_cannot_run, test_loaded_function_node_cannot_execute message). Each is +changed to read after `run()`/`run(load_only=True)`, or to `as_source()` where it is about the +stored records. Workshop scenario tests: unaffected (they run first, or read the store directly). +Full list: plan file appendix. + +Docs: docs/concepts/streams.md (run table and head table rows for "A job node", the "stream that +has all of its rows" paragraph, ~262), docs/concepts/function-pods.md (339-341 no-run read; +"Saving and loading a job" 662-746 and its sample 689-694, the only doc sample that reads a node +without a run: becomes bind + `loaded.run(load_only=True)`, or `as_source()`; a new "Loading +without computing" subsection), docs/concepts/sources.md 169-172; docstrings of +`PipelineJob.load/run/run_async`, node `head/as_table/iter_data/run/clear_cache`, operator +`iter_data/as_table/head`, `ViewStatus`, `StreamBase.head`. Changelog fragment `ITL-1095.md` +(Breaking: a node read before a run is empty, loaded nodes too, `node.run()` cascades; Added: +`load_only`, summary fields). CLAUDE.md/.zed/rules: replaces #378's sentence, amends the run() +rule (D6). + +## H. PR #346 (g) + +Only `tests/test_core/alignment/test_nodes.py::read_back(how="load")` reads a loaded job +without a run: `test_rows_read_back_from_the_store_are_aligned` (216 parametrizations: loaded +`scaled`/`joined` `as_table`/`head`) and `test_stored_records_read_back_are_aligned` (54; reads +`iter_data`). It will need `loaded.bind(sources=...)` (its sources are in-memory) and +`loaded.run(load_only=True)` before `return loaded`; `joined` is attached in REPLAY (D1 decides +what it then holds). `test_job.py` and the rerun modes read after a run: unaffected. + +## Sizing + +Large for Sunday: node rule + load-only through both orchestrators + summary + cascade + loaded +jobs + ~400 test updates + docs. If something must be cut for the tag, I would drop first D's +loaded-job load-only run (loaded reads empty, `as_source()` the way in, `loaded.run()` raises), +then C's "current ancestor" skip (a cascade always runs every ancestor). diff --git a/superpowers/plans/2026-11-14-itl-1095-merge-main-itl-912.md b/superpowers/plans/2026-11-14-itl-1095-merge-main-itl-912.md new file mode 100644 index 000000000..cd93664f1 --- /dev/null +++ b/superpowers/plans/2026-11-14-itl-1095-merge-main-itl-912.md @@ -0,0 +1,50 @@ +# ITL-1095: merge main (with #375, ITL-912) into PR #380 + +Coordinator task task_841fb1367e00, 2026-11-14. Branch `eywalker/itl-1095-design-what-a-job-node-shows-the-current`, +head before merge `7de3512b`, main `52d67858`. + +## Goal + +Both rules hold after the merge: + +- #380 (ITL-1095): a job node shows its current run; before a run, after `clear_cache()`, after a + failed run and while stale it reads zero rows and touches no store; `LOADED_ONLY`; state-driven runs. +- #375 (ITL-912): that zero-row table has the Arrow schema of the node's populated table (types, + nullability, system tags, source info, column order), derived from input schemas only. + +## Conflicts and resolution + +| File | Resolution | +|---|---| +| `CLAUDE.md`, `.zed/rules` | main's head() paragraph (ITL-991, ITL-912), last sentence replaced by #380's `load_only` rule | +| `core/nodes/function_node.py` | both: #380's `_read_finds_stale` and main's `_zero_row_output_table` / `_typed_zero_row_table` / `_zero_row_table`; the empty current-run table already goes through `_table_from_pairs([])`, which main types | +| `core/nodes/operator_node.py` | `as_table` docstring: #380's current-run text + main's typed zero-row paragraph (REPLAY store read and "loaded nodes raise" dropped: #380 removed them). Body is #380's: `_make_empty_table()`, which main made typed. `head` summary: #380's | +| `tests/.../test_loaded_operator_cache.py` | helper keeps #380's stored-view comparison (`as_source()`) and main's `same_types` parameter | + +## Test changes (ask coordinator before push) + +- `test_an_absent_cache_table_gives_the_running_nodes_types` (#375, strict xfail ITL-1089): drop the + `pytest.warns(StoreContentWarning)` wrapper; under #380 `as_table()` of a node that has not run + never reads the store, so it never warns. The schema comparison (the xfail's point) is unchanged. + +## New test + +`tests/test_pipeline/test_current_run_view_zero_row_types.py`: function node and operator nodes +(group_by, join) over an int8 not-null tag / int32 data source; `as_table(all_info=True).schema` is the +same before a run, after a run, after `clear_cache()`, after a failed run, stale, after +`run(load_only=True)`; sync and async. + +## Verification + +tests/test_pipeline, tests/test_core/{streams,nodes,operators,function_pod,side_effect_pod}, +golden suite on 3.12; async files on 3.11; ruff, basedpyright, changelog/design-issue checks. + +## Progress notes + +- Merge commit made locally (`Merge origin/main (ITL-912, #375) into ITL-1095`). +- Coordinator approved the xfail test change (1) and Option A for (2). +- (2) Pre-existing on main (checked against an export of origin/main): a Join node's zero-row + table has the populated table's fields in another order (data before system tags). The new + test compares that one case by field name, with a strict xfail of the exact order + (`test_a_join_with_no_input_rows_keeps_its_column_order`, "ITL-1131" until the + coordinator's issue id arrives). Reproduction: scratchpad `order.py` (filter -> join, sync and async). diff --git a/test-objective/integration/test_caching_flows.py b/test-objective/integration/test_caching_flows.py index 6cdca8a8f..ed91d25e4 100644 --- a/test-objective/integration/test_caching_flows.py +++ b/test-objective/integration/test_caching_flows.py @@ -85,6 +85,8 @@ def test_second_run_uses_cache(self): pipeline_database=pipeline_db, result_database=result_db, ) + # A node shows rows only after a run (ITL-1095); this run reads them all. + node2.run() data = list(node2.iter_data()) assert len(data) == 3 diff --git a/tests/test_arrays/test_array_v02_store.py b/tests/test_arrays/test_array_v02_store.py index da43e93be..8706caa41 100644 --- a/tests/test_arrays/test_array_v02_store.py +++ b/tests/test_arrays/test_array_v02_store.py @@ -103,6 +103,10 @@ def test_v02_rows_are_ignored_recomputed_and_kept( # Read through a newly built job: a Delta database object that read a table before # the run does not see the rows the run added to it (not specific to hash labels). stored = build_job(store) + # A node shows rows only after a run (ITL-1095): a load-only run reads them from the + # store and computes nothing. + stored.run(orchestrator=orchestrator, load_only=True) + assert CALLS == ["plus_two"] * len(ARRAYS) rows = [ (t.as_dict()["k"], d.as_dict()["y"]) for t, d in stored.nodes["double"].iter_data() @@ -154,5 +158,8 @@ def test_second_run_reuses_the_rows_of_this_version(store: Path) -> None: assert CALLS == [] stored = build_job(store) + # A node shows rows only after a run (ITL-1095): a load-only run reads them. + stored.run(load_only=True) + assert CALLS == [] for label in ("double", "total", "direct_sum", "plus_one", "after_drop"): assert stored.nodes[label].as_table().num_rows == len(ARRAYS), label diff --git a/tests/test_core/function_pod/test_cached_function_pod_job.py b/tests/test_core/function_pod/test_cached_function_pod_job.py index 6ada9c6d4..e14541b98 100644 --- a/tests/test_core/function_pod/test_cached_function_pod_job.py +++ b/tests/test_core/function_pod/test_cached_function_pod_job.py @@ -233,9 +233,10 @@ def test_saved_job_equals_the_uncached_pods_and_loads( doc["bindings"].pop("database") assert cached_saved == plain_saved - # A loaded job reads its stored results; the pod cache is not part of it. + # A loaded job's stored results are read through the stored view (its nodes are + # empty until it runs, ITL-1095); the pod cache is not part of it. loaded = PipelineJob.load(tmp_path / "cached.json", database=store) - assert loaded.nodes["add"].as_table().to_pylist() == [ + assert loaded.nodes["add"].as_source().as_table().to_pylist() == [ {"k": 1, "total": 5}, {"k": 2, "total": 10}, ] diff --git a/tests/test_core/function_pod/test_ephemeral_result.py b/tests/test_core/function_pod/test_ephemeral_result.py index b7fb8dbeb..53bec7e79 100644 --- a/tests/test_core/function_pod/test_ephemeral_result.py +++ b/tests/test_core/function_pod/test_ephemeral_result.py @@ -581,8 +581,8 @@ def test_iter_data_serves_result_after_cross_session_recompute(self): iter_data() serves the fresh result without triggering additional computation. Verifies that: - - after clearing the in-memory cache, iter_data() hot-loads the recomputed - result from the pipeline + ephemeral DBs (exercises _load_cached_entries) + - after clearing the node, its next run loads the recomputed result from the + pipeline + ephemeral DBs (exercises _load_cached_entries) - repeated iter_data() calls return the same result without recomputing """ call_count = {"n": 0} @@ -621,11 +621,12 @@ def counting_double(x: int) -> int: node2.execute(stream) assert call_count["n"] == 2 # recomputed once due to cross-session miss - # execute() calls _update_modified_time() at the end, so is_stale is False here. - # Manually clearing forces the iter_data() hot-load branch (_load_cached_entries → - # _fetch_joined_records) without relying on staleness, so this tests the DB join - # path explicitly rather than the in-memory dict fast-path. - node2._cached_output_datas.clear() + # Clearing the node drops its in-memory results; running it again loads the + # recomputed result through the DB join path (_load_cached_entries → + # _fetch_joined_records) rather than the in-memory dict fast-path. A node shows + # rows only after a run (ITL-1095). + node2.clear_cache() + node2.execute(stream) # iter_data() must serve the recomputed result — no additional computation results = list(node2.iter_data()) diff --git a/tests/test_core/function_pod/test_function_node_arrow_types.py b/tests/test_core/function_pod/test_function_node_arrow_types.py index 0a3764a2a..e2253d91f 100644 --- a/tests/test_core/function_pod/test_function_node_arrow_types.py +++ b/tests/test_core/function_pod/test_function_node_arrow_types.py @@ -230,7 +230,7 @@ def test_as_table_mixing_stored_and_new_rows_keeps_tag_type() -> None: def test_store_read_back_keeps_tag_type( name: str, store: str, tmp_path: pathlib.Path ) -> None: - """Reads from the store (``get_all_records``, a fresh job's reads) keep the tag type.""" + """Reads from the store (``get_all_records``, a fresh job's run) keep the tag type.""" database = _new_store(store, tmp_path) table = _source_table(name) _build(database, table).run() @@ -238,11 +238,14 @@ def test_store_read_back_keeps_tag_type( # A new store object sees the committed tables, as a new process would. database = _new_store(store, tmp_path) - fresh = _build(database, table).nodes["double"] + fresh_job = _build(database, table) + fresh = fresh_job.nodes["double"] records = fresh.get_all_records() assert records is not None assert records.schema.field("k").type == _expected(name) + # The node shows rows once its run has read them from the store (ITL-1095). + fresh_job.run() tag, _ = next(iter(fresh.iter_data())) assert tag.as_table().schema.field("k").type == _expected(name) assert fresh.as_table().schema.field("k").type == _expected(name) diff --git a/tests/test_core/function_pod/test_function_node_caching.py b/tests/test_core/function_pod/test_function_node_caching.py index c58ba463b..2e194db9b 100644 --- a/tests/test_core/function_pod/test_function_node_caching.py +++ b/tests/test_core/function_pod/test_function_node_caching.py @@ -231,6 +231,7 @@ def test_phase1_yields_existing_results(self): # Create a second node with the same stream and shared DB stream2 = _make_stream([{"id": 0, "x": 10}, {"id": 1, "x": 20}]) node2, _ = _make_node(stream2, db=db) + node2.run() # a node shows its rows only after a run (ITL-1095) results = list(node2.iter_data()) # All results should come from Phase 1 (DB), not recomputed @@ -319,6 +320,7 @@ def test_all_existing_entry_ids_skipped_in_phase2(self): node2._versioned_pipeline_path ).num_rows + node2.run() results = list(node2.iter_data()) assert len(results) == 2 diff --git a/tests/test_core/function_pod/test_function_pod_node_stream.py b/tests/test_core/function_pod/test_function_pod_node_stream.py index 2ab30e0e7..8ba09ec66 100644 --- a/tests/test_core/function_pod/test_function_pod_node_stream.py +++ b/tests/test_core/function_pod/test_function_pod_node_stream.py @@ -169,7 +169,8 @@ def test_as_table_returns_cached_results_when_data_function_inactive( ): """ Cache filled by node1 (active) is shared with node2 (inactive). - node2.as_table() must return full results from Phase 1 (DB). + node2's run serves the full results from Phase 1 (DB); a node shows its rows + only after a run (ITL-1095). """ n = 3 db = InMemoryArrowDatabase() @@ -181,6 +182,7 @@ def test_as_table_returns_cached_results_when_data_function_inactive( double_pf.set_active(False) node2 = _make_node(double_pf, n=n, db=db) + node2.run() table2 = node2.as_table() assert isinstance(table2, pa.Table) @@ -201,6 +203,7 @@ def test_inactive_filled_db_serves_cached_results(self, double_pf): double_pf.set_active(False) node2 = _make_node(double_pf, n=n, db=db) + node2.run() # a node shows its rows only after a run (ITL-1095) data = list(node2.iter_data()) assert len(data) == n @@ -266,7 +269,9 @@ def test_db_served_results_have_correct_row_count(self, double_pf): n = 5 db = InMemoryArrowDatabase() _fill_node(_make_node(double_pf, n=n, db=db)) - data = list(_make_node(double_pf, n=n, db=db).iter_data()) + node2 = _make_node(double_pf, n=n, db=db) + node2.run() # a node shows its rows only after a run (ITL-1095) + data = list(node2.iter_data()) assert len(data) == n def test_fresh_db_always_computes(self, double_pf): @@ -373,9 +378,11 @@ def counting_double(x: int) -> int: class TestFunctionNodeStaleness: # --- is_stale --- - def test_is_stale_false_immediately_after_creation(self, double_pf): - """A freshly created FunctionNode whose upstream has not changed is not stale.""" + def test_is_stale_immediately_after_creation(self, double_pf): + """A freshly created job node holds no run, so it is stale (ITL-1095).""" node = _make_node(double_pf, n=3) + assert node.is_stale + node.run() assert not node.is_stale def test_is_stale_true_after_upstream_modified(self, double_pf): @@ -388,14 +395,16 @@ def test_is_stale_true_after_upstream_modified(self, double_pf): input_stream=input_stream, pipeline_database=db, ) - list(node.iter_data()) + node.run() # a node that has not run is stale already (ITL-1095) + assert not node.is_stale + assert len(list(node.iter_data())) == 3 time.sleep(0.01) input_stream._update_modified_time() assert node.is_stale - def test_is_stale_false_after_clear_cache(self, double_pf): + def test_is_stale_after_clear_cache_until_it_runs(self, double_pf): import time db = InMemoryArrowDatabase() @@ -405,13 +414,16 @@ def test_is_stale_false_after_clear_cache(self, double_pf): input_stream=input_stream, pipeline_database=db, ) - list(node.iter_data()) + node.run() time.sleep(0.01) input_stream._update_modified_time() assert node.is_stale + # A cleared node holds no run: it stays stale until it runs (ITL-1095). node.clear_cache() + assert node.is_stale + node.run() assert not node.is_stale # --- clear_cache --- @@ -431,6 +443,9 @@ def test_clear_cache_produces_same_results_on_re_iteration(self, double_pf): table_before = node.as_table() node.clear_cache() + # A cleared node reads empty until it runs again (ITL-1095). + assert node.as_table().num_rows == 0 + node.run() table_after = node.as_table() assert sorted(table_before.column("result").to_pylist()) == sorted( @@ -449,12 +464,17 @@ def test_iter_data_auto_detects_stale_and_repopulates(self, double_pf): input_stream=input_stream, pipeline_database=db, ) + node.run() first = list(node.iter_data()) + assert len(first) == 3 time.sleep(0.01) input_stream._update_modified_time() assert node.is_stale + # A stale node reads empty (ITL-1095); its run reads the stored results again. + assert list(node.iter_data()) == [] + node.run() second = list(node.iter_data()) assert len(second) == len(first) assert [p["result"] for _, p in second] == [p["result"] for _, p in first] @@ -476,6 +496,9 @@ def test_as_table_auto_detects_stale_and_repopulates(self, double_pf): time.sleep(0.01) input_stream._update_modified_time() + # A stale node reads empty (ITL-1095); its run reads the stored results again. + assert node.as_table().num_rows == 0 + node.run() table_after = node.as_table() assert len(table_after) == 3 assert sorted(table_after.column("result").to_pylist()) == sorted( @@ -484,14 +507,20 @@ def test_as_table_auto_detects_stale_and_repopulates(self, double_pf): def test_no_auto_clear_when_not_stale(self, double_pf): node = _make_node(double_pf, n=3) + node.run() # a node that has not run holds nothing (ITL-1095) + assert not node.is_stale list(node.iter_data()) cached_count = len(node._cached_output_datas) + assert cached_count == 3 list(node.iter_data()) assert len(node._cached_output_datas) == cached_count def test_as_table_no_auto_clear_when_not_stale(self, double_pf): node = _make_node(double_pf, n=3) + node.run() # a node that has not run holds nothing (ITL-1095) + assert not node.is_stale table_before = node.as_table() + assert table_before.num_rows == 3 table_after = node.as_table() assert table_before.equals(table_after) diff --git a/tests/test_core/function_pod/test_polars_dataframe_values.py b/tests/test_core/function_pod/test_polars_dataframe_values.py index 170d0f53b..c18a38515 100644 --- a/tests/test_core/function_pod/test_polars_dataframe_values.py +++ b/tests/test_core/function_pod/test_polars_dataframe_values.py @@ -462,19 +462,23 @@ def test_saved_job_loaded_back_reads_the_frames(self, tmp_path: Path) -> None: node = loaded.nodes["make"] _, data_schema = node.output_schema() assert data_schema["df"] is pl.DataFrame - got = {tag["k"]: data["df"] for tag, data in node.iter_data()} + # A loaded job's nodes are empty until it runs (ITL-1095); its stored results + # are read through the stored view. + stored_view = node.as_source() + got = {tag["k"]: data["df"] for tag, data in stored_view.iter_data()} _assert_exact(got["a"], _frame(1)) _assert_exact(got["b"], _frame(3)) stored = dict( zip( - node.as_table().column("k").to_pylist(), - _stored_frames(node.as_table().column("df")), + stored_view.as_table().column("k").to_pylist(), + _stored_frames(stored_view.as_table().column("df")), strict=True, ) ) _assert_exact(stored["a"], _frame(1)) heights = { - tag["k"]: data["height"] for tag, data in loaded.nodes["height"].iter_data() + tag["k"]: data["height"] + for tag, data in loaded.nodes["height"].as_source().iter_data() } assert heights == {"a": 1, "b": 3} diff --git a/tests/test_core/function_pod/test_readback_datagram_id.py b/tests/test_core/function_pod/test_readback_datagram_id.py index 661df3598..9b922b8ae 100644 --- a/tests/test_core/function_pod/test_readback_datagram_id.py +++ b/tests/test_core/function_pod/test_readback_datagram_id.py @@ -89,7 +89,10 @@ def test_a_fresh_job_on_the_store_reads_the_stored_ids() -> None: _ = first.run(orchestrator=SyncPipelineOrchestrator()) stored = stored_ids(first) - fresh = build(store) # never run: every row comes from the store + fresh = build(store) + # A node shows rows only after a run (ITL-1095): this run computes nothing, every + # row comes from the store. + _ = fresh.run(orchestrator=SyncPipelineOrchestrator()) assert read_back_ids(fresh) == stored diff --git a/tests/test_core/nodes/test_function_node_empty_data.py b/tests/test_core/nodes/test_function_node_empty_data.py index 3158dc148..b6ca7555b 100644 --- a/tests/test_core/nodes/test_function_node_empty_data.py +++ b/tests/test_core/nodes/test_function_node_empty_data.py @@ -13,6 +13,7 @@ from orcapod.core.nodes.function_node import FunctionJobNode from orcapod.core.sources import ArrowTableSource from orcapod.databases import InMemoryArrowDatabase +from orcapod.protocols.core_protocols import DataProtocol, TagProtocol from orcapod.system_constants import constants from orcapod.types import NodeConfig from orcapod.errors import EphemeralResultMissingError @@ -174,6 +175,20 @@ def test_normal_result_wins_over_empty_data_token(self, persistent_node): assert isinstance(data, Data) +def _entries(node: FunctionJobNode) -> list[tuple[TagProtocol, DataProtocol]]: + """The node's entries for its input rows, as its cache façade returns them. + + These tests drive ``execute_data`` (one row, not a run), so the node shows no rows + of a run (ITL-1095); ``get_cached_results`` gives the entries it holds or has + stored, an ``EmptyData`` placeholder for a result whose ephemeral store is gone. + """ + entry_ids = [ + node.compute_base_entry_id(tag, data) + for tag, data in node._input_stream.iter_data() + ] + return list(node.get_cached_results(entry_ids).values()) + + class TestProcessDataInternalEmptyData: def test_cache_hit_returns_cached_result(self, ephemeral_node): """Downstream cache hit on EmptyData input returns the cached result.""" @@ -203,7 +218,7 @@ def triple_value(result: int) -> int: ) # Step 3: Run downstream with real data to populate its result cache - for tag, data in node.iter_data(): + for tag, data in _entries(node): downstream.execute_data(tag, data) # Step 4: Simulate upstream ephemeral expiry @@ -211,7 +226,7 @@ def triple_value(result: int) -> int: node._cached_output_datas.clear() # Step 5: Upstream now emits EmptyData - upstream_items = list(node.iter_data()) + upstream_items = _entries(node) assert len(upstream_items) == 1 tag_in, data_in = upstream_items[0] assert isinstance(data_in, EmptyData) @@ -252,7 +267,7 @@ def triple_value(result: int) -> int: result_database=InMemoryArrowDatabase(), ) - upstream_items = list(node.iter_data()) + upstream_items = _entries(node) tag_in, data_in = upstream_items[0] assert isinstance(data_in, EmptyData) @@ -296,7 +311,7 @@ def triple_value(result: int) -> int: ) # Populate downstream result cache with real data (sync) - for tag, data in node.iter_data(): + for tag, data in _entries(node): downstream.execute_data(tag, data) # Simulate ephemeral expiry @@ -304,7 +319,7 @@ def triple_value(result: int) -> int: node._cached_output_datas.clear() # Upstream now emits EmptyData - upstream_items = list(node.iter_data()) + upstream_items = _entries(node) assert len(upstream_items) == 1 tag_in, data_in = upstream_items[0] assert isinstance(data_in, EmptyData) @@ -341,7 +356,7 @@ def triple_value(result: int) -> int: result_database=InMemoryArrowDatabase(), ) - upstream_items = list(node.iter_data()) + upstream_items = _entries(node) tag_in, data_in = upstream_items[0] assert isinstance(data_in, EmptyData) diff --git a/tests/test_core/nodes/test_function_node_iteration.py b/tests/test_core/nodes/test_function_node_iteration.py index fee923c42..10a5266a9 100644 --- a/tests/test_core/nodes/test_function_node_iteration.py +++ b/tests/test_core/nodes/test_function_node_iteration.py @@ -59,14 +59,17 @@ def test_iter_does_not_call_process_data_internal(self): list(node.iter_data()) mock_proc.assert_not_called() - def test_iter_after_db_populated_hot_loads_without_compute(self): - """iter_data() on a node with DB records hot-loads without _process_data_internal.""" + def test_iter_after_db_populated_yields_nothing_until_a_run_loads(self): + """A node over a populated DB yields nothing before a run (ITL-1095); its run + loads every result from the DB without _process_data_internal.""" db = InMemoryArrowDatabase() node1 = _make_node(n=3, db=db) node1.run() # populate DB node2 = _make_node(n=3, db=db) + assert list(node2.iter_data()) == [] with patch.object(node2, "_process_data_internal") as mock_proc: + node2.run() results = list(node2.iter_data()) mock_proc.assert_not_called() assert len(results) == 3 @@ -120,10 +123,10 @@ def test_as_table_fresh_node_returns_empty_no_compute(self): assert "id" in table.column_names assert "result" in table.column_names - def test_run_cache_only_is_noop(self): - """run() on a CACHE_ONLY node computes nothing and returns the stored rows' view.""" + def test_run_cache_only_raises_and_points_to_the_stored_view(self): + """run() on a CACHE_ONLY node computes nothing: its input is unavailable, so it + has nothing to run or look up, and it says to read ``as_source()`` (ITL-1095).""" from orcapod.pipeline.serialization import LoadStatus - from orcapod.streams import StreamView node = _make_node() # A loaded node whose upstream is unavailable keeps that input stream; only its @@ -131,9 +134,10 @@ def test_run_cache_only_is_noop(self): node._load_status = LoadStatus.CACHE_ONLY with patch.object(node, "execute") as mock_exec: - view = node.run() + for load_only in (False, True): + with pytest.raises(RuntimeError, match=r"as_source\(\)"): + node.run(load_only=load_only) mock_exec.assert_not_called() - assert isinstance(view, StreamView) def test_run_unavailable_raises(self): """run() on an UNAVAILABLE node raises RuntimeError.""" diff --git a/tests/test_core/operators/test_materialize_source_info.py b/tests/test_core/operators/test_materialize_source_info.py index 59a7c9f8b..057291375 100644 --- a/tests/test_core/operators/test_materialize_source_info.py +++ b/tests/test_core/operators/test_materialize_source_info.py @@ -694,6 +694,7 @@ def test_rerun_after_pre_fix_store(self, monkeypatch: pytest.MonkeyPatch): ) replay = self._node(db, CacheMode.REPLAY) + _ = replay.run() # a node shows rows only after a run (ITL-1095) assert len(list(replay.iter_data())) == 5 def test_v014_store_is_not_read(self, tmp_path: Path): @@ -739,6 +740,7 @@ def test_v014_store_is_not_read(self, tmp_path: Path): tokens = _source_info_tokens(list(_pin_source().iter_data())) assert _stored_source(new) == tokens replay = self._node(db, CacheMode.REPLAY) + _ = replay.run() # a node shows rows only after a run (ITL-1095) assert _source_info_tokens(list(replay.iter_data())) == tokens diff --git a/tests/test_core/operators/test_operator_node.py b/tests/test_core/operators/test_operator_node.py index 1c54d45c6..53cab260f 100644 --- a/tests/test_core/operators/test_operator_node.py +++ b/tests/test_core/operators/test_operator_node.py @@ -375,6 +375,9 @@ def test_replay_from_cache(self, simple_stream, db): node_replay = _make_node( op, (simple_stream,), db=db, cache_mode=CacheMode.REPLAY ) + # A node shows rows only after a run (ITL-1095); REPLAY's run reads the cache. + assert node_replay.as_table().num_rows == 0 + node_replay.run() table = node_replay.as_table() assert table.num_rows == 3 assert "renamed_x" in table.column_names diff --git a/tests/test_core/operators/test_operator_node_non_active.py b/tests/test_core/operators/test_operator_node_non_active.py index 52922a3de..c3215141f 100644 --- a/tests/test_core/operators/test_operator_node_non_active.py +++ b/tests/test_core/operators/test_operator_node_non_active.py @@ -160,8 +160,11 @@ def test_iter_data_replay_mode_no_records_returns_empty( result = list(node.iter_data()) assert result == [] - def test_iter_data_replay_mode_returns_db_contents(self, simple_source, map_op): - """REPLAY + prior LOG run → iter_data returns DB records without run().""" + def test_iter_data_replay_mode_returns_db_contents_after_its_run( + self, simple_source, map_op + ): + """REPLAY + prior LOG run → iter_data returns nothing before the REPLAY node's + run (ITL-1095) and the DB records after it.""" db = InMemoryArrowDatabase() # Populate DB via LOG run node_log = _node(map_op, (simple_source,), db=db, cache_mode=CacheMode.LOG) @@ -171,6 +174,8 @@ def test_iter_data_replay_mode_returns_db_contents(self, simple_source, map_op): node_replay = _node( map_op, (simple_source,), db=db, cache_mode=CacheMode.REPLAY ) + assert list(node_replay.iter_data()) == [] + node_replay.run() result = list(node_replay.iter_data()) assert len(result) == 3 for _, data in result: diff --git a/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py b/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py index 2411ff8d9..b72f5ff7f 100644 --- a/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py +++ b/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py @@ -184,15 +184,26 @@ def test_node_run_delivers_what_the_log_does_not_record( @pytest.mark.parametrize("orchestrator", ["sync", "async"]) -def test_node_run_without_track_completion_delivers_every_row_as_a_job_run_does( +def test_node_run_without_track_completion_delivers_again_only_after_clear_cache( orchestrator: Orchestrator, ) -> None: - """Without ``track_completion``, ``node.run()`` delivers every row again, as ``job.run()`` does.""" + """Without ``track_completion``, a node that holds a current run delivers nothing on + ``node.run()`` (ITL-1095); after ``clear_cache()`` it delivers every row again, as + ``job.run()`` does.""" job, sent = _job(track_completion=False) _run(job, orchestrator) view = job.nodes["delivered"].run() + assert len(sent) == ROWS + assert len(view) == ROWS + job.nodes["delivered"].clear_cache() + view = job.nodes["delivered"].run() assert len(sent) == 2 * ROWS assert len(view) == ROWS + # job.run() follows the same rule: the node holds a current run, so it delivers + # nothing; cleared, it delivers every row again. + _run(job, orchestrator) + assert len(sent) == 2 * ROWS + job.nodes["delivered"].clear_cache() _run(job, orchestrator) assert len(sent) == 3 * ROWS # The reads after either kind of run deliver nothing. @@ -218,6 +229,8 @@ def test_an_async_run_holds_exactly_its_own_rows(track_completion: bool) -> None """After two async runs, the node holds the rows of the last run only.""" job, sent = _job(track_completion=track_completion) _run(job, "async") + # A node holding a current run is not run again (ITL-1095): clear it first. + job.nodes["delivered"].clear_cache() _run(job, "async") node = job.nodes["delivered"] assert len(node.as_table()) == ROWS @@ -319,6 +332,9 @@ def test_rows_held_from_a_run_are_dropped_when_an_upstream_changes( _run(job, orchestrator) node = job.nodes["delivered"] assert len(node.as_table()) == ROWS + # The upstream runs again: a node that holds a current run runs nothing (ITL-1095), + # so it is cleared first. + job.nodes["up"].clear_cache() _ = job.nodes["up"].run() for name, read in _reads(node).items(): _ = read() diff --git a/tests/test_core/streams/test_run.py b/tests/test_core/streams/test_run.py index 4b6c18b80..d713c9710 100644 --- a/tests/test_core/streams/test_run.py +++ b/tests/test_core/streams/test_run.py @@ -109,12 +109,26 @@ def _all_stream_classes() -> list[type]: @pytest.mark.parametrize("cls", _all_stream_classes(), ids=lambda c: c.__qualname__) -def test_every_stream_class_has_run_without_arguments(cls: type) -> None: - """``run`` exists on every stream class and takes no argument (developer decision).""" +def test_every_stream_class_has_run_without_positional_arguments(cls: type) -> None: + """``run`` exists on every stream class and takes no positional argument (developer + decision, ITL-1081); a job node's ``run`` takes one keyword-only argument, + ``load_only`` (ITL-1095).""" run = getattr(cls, "run", None) assert callable(run), f"{cls.__qualname__} has no run()" - parameters = list(inspect.signature(run).parameters) - assert parameters == ["self"], f"{cls.__qualname__}.run{parameters}" + parameters = inspect.signature(run).parameters + names = list(parameters) + if cls.__name__ in ( + "FunctionJobNode", + "OperatorJobNode", + "SideEffectJobNode", + "SourceJobNode", + ): + assert names == ["self", "load_only"], f"{cls.__qualname__}.run{names}" + load_only = parameters["load_only"] + assert load_only.kind is inspect.Parameter.KEYWORD_ONLY + assert load_only.default is False + else: + assert names == ["self"], f"{cls.__qualname__}.run{names}" def test_the_stream_protocol_declares_run() -> None: diff --git a/tests/test_databases/test_delta_arrow_type_roundtrip.py b/tests/test_databases/test_delta_arrow_type_roundtrip.py index f6ba81599..d87183636 100644 --- a/tests/test_databases/test_delta_arrow_type_roundtrip.py +++ b/tests/test_databases/test_delta_arrow_type_roundtrip.py @@ -194,7 +194,9 @@ def _chained_job(root: pathlib.Path, table: pa.Table) -> PipelineJob: def test_a_rerun_on_the_same_store_writes_no_pipeline_rows( name: str, mode: str, tmp_path: pathlib.Path ) -> None: - """The ITL-990 case: source -> pod -> pod, run twice; a fresh job reads 3 rows.""" + """The ITL-990 case: source -> pod -> pod, run twice; a fresh job's run shows 3 rows + and the store holds 3 records per node (a node shows rows only after a run, + ITL-1095).""" from orcapod.pipeline import AsyncPipelineOrchestrator table = _source_table(TAGS[name]) @@ -206,6 +208,7 @@ def test_a_rerun_on_the_same_store_writes_no_pipeline_rows( job.run() fresh = _chained_job(tmp_path, table) + fresh.run() for label in ("double", "increment"): node = fresh.nodes[label] assert node.as_table().num_rows == 3 @@ -217,11 +220,14 @@ def test_a_rerun_on_the_same_store_writes_no_pipeline_rows( @pytest.mark.parametrize("name", list(TAGS)) def test_a_fresh_job_reads_the_tag_type(name: str, tmp_path: pathlib.Path) -> None: - """A new job on the same store reads the tag type through every read of a node.""" + """A new job on the same store, run, reads the tag type through every read of a + node (a node shows rows only after a run, ITL-1095).""" table = _source_table(TAGS[name]) _chained_job(tmp_path, table).run() - fresh = _chained_job(tmp_path, table).nodes["double"] + fresh_job = _chained_job(tmp_path, table) + fresh_job.run() + fresh = fresh_job.nodes["double"] assert fresh.as_table().schema.field("k").type == TAGS[name].type tag, _ = next(iter(fresh.iter_data())) @@ -232,7 +238,8 @@ def test_a_fresh_job_reads_the_tag_type(name: str, tmp_path: pathlib.Path) -> No # Run in a child process by ``test_a_loaded_job_reads_the_tag_type``: load the job -# saved under ``argv[1]`` and print each function node's tag type as JSON. +# saved under ``argv[1]`` and print each function node's stored tag type as JSON (a +# loaded node is empty until it runs, ITL-1095: its stored rows are its stored view). _READ_LOADED_TAG_TYPES = """ import json, pathlib, sys from orcapod import PipelineJob @@ -240,7 +247,7 @@ def test_a_fresh_job_reads_the_tag_type(name: str, tmp_path: pathlib.Path) -> No root = pathlib.Path(sys.argv[1]) job = PipelineJob.load(root / "job.json", database=DeltaTableDatabase(root / "store")) print(json.dumps({ - label: str(job.nodes[label].as_table().schema.field("k").type) + label: str(job.nodes[label].as_source().as_table().schema.field("k").type) for label in ("double", "increment") })) """ @@ -299,7 +306,9 @@ def build(mode: CacheMode) -> PipelineJob: build(CacheMode.LOG).run() - replayed = build(CacheMode.REPLAY).nodes["sel"].as_table() + replay_job = build(CacheMode.REPLAY) + replay_job.run() # a node shows rows only after a run (ITL-1095) + replayed = replay_job.nodes["sel"].as_table() assert replayed.schema.field("k").type == TAGS[name].type diff --git a/tests/test_logical_types/test_provisional_hash_warning.py b/tests/test_logical_types/test_provisional_hash_warning.py index 5a450ee09..c3e102a73 100644 --- a/tests/test_logical_types/test_provisional_hash_warning.py +++ b/tests/test_logical_types/test_provisional_hash_warning.py @@ -232,7 +232,8 @@ def run_and_save() -> None: _quiet(run_and_save) with recorded() as caught: loaded = PipelineJob.load(tmp_path / "job.json") - table = loaded.nodes["make"].as_table() + # A loaded node is empty until it runs (ITL-1095): read the stored view. + table = loaded.nodes["make"].as_source().as_table() assert table.num_rows == 3 assert _warned_names(caught) == [name] diff --git a/tests/test_pipeline/loaded_job/test_loaded_job_readback.py b/tests/test_pipeline/loaded_job/test_loaded_job_readback.py index 962131eb7..53bd101b5 100644 --- a/tests/test_pipeline/loaded_job/test_loaded_job_readback.py +++ b/tests/test_pipeline/loaded_job/test_loaded_job_readback.py @@ -69,6 +69,39 @@ def _rewrite_job_file( return str(path) +def _loaded_and_filled(saved: pathlib.Path) -> PipelineJob: + """Load the saved job, bind its in-memory sources and fill it from the store. + + A loaded node is empty until a load-only run fills it (ITL-1095). The sources bound + are those of ``save_job_script.py``; the CSV and Delta sources are rebuilt. + """ + from orcapod.sources import ArrowTableSource, ListSource + + loaded = _load(saved) + arrow = ArrowTableSource( + pa.table( + {"id": [0, 1], "y": [3, 4]}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("y", pa.int64(), nullable=False), + ] + ), + ), + tag_columns=["id"], + source_id="asrc", + ) + loaded.bind( + sources={ + "dsrc": _dict_source(), + "asrc": arrow, + "lsrc": ListSource("z", [1, 2], source_id="lsrc"), + } + ) + _ = loaded.run(load_only=True) + return loaded + + def _dict_source() -> DictSource: """Return a two-row in-memory source with tag ``id`` and data ``x``.""" return DictSource( @@ -142,7 +175,8 @@ def record() -> None: path = _save_small_job(tmp_path, record) node = PipelineJob.load(path).nodes["stamp"] assert dict(node.output_schema()[1]) == {"t": datetime.datetime} - assert node.as_table().num_rows == 2 + # A loaded node is empty until it runs (ITL-1095); its stored view reads the rows. + assert node.as_source().as_table().num_rows == 2 def test_datetime_output_with_nothing_stored_returns_typed_zero_rows( @@ -189,8 +223,9 @@ def record() -> None: node = PipelineJob.load(_save_small_job(tmp_path, record)).nodes["paths"] assert node.output_schema()[1]["p"] is pathlib.Path - # The stored rows are still readable, with their stored logical type. - table = node.as_table() + # The stored rows are still readable, with their stored logical type (through the + # stored view: a loaded node is empty until it runs, ITL-1095). + table = node.as_source().as_table() assert table.num_rows == 2 assert isinstance(table.schema.field("p").type, pa.ExtensionType) @@ -219,16 +254,16 @@ def record() -> None: Join()(paths, other, label="joined") loaded = PipelineJob.load(_save_small_job(tmp_path, record)) - assert loaded.nodes["paths"].as_table().num_rows == 2 + assert loaded.nodes["paths"].as_source().as_table().num_rows == 2 @pytest.mark.parametrize("label", FUNCTION_NODES) def test_loaded_function_node_as_table_returns_stored_rows( saved: pathlib.Path, label: str ) -> None: - """A loaded node's table equals the live table: schema and values.""" + """A loaded node's stored view equals the live table: schema and values.""" expected = pq.read_table(saved / "expected" / f"{label}.parquet") - table = _load(saved).nodes[label].as_table() + table = _load(saved).nodes[label].as_source().as_table() assert table.schema == expected.schema assert _sorted(table, TAG_OF[label]).equals(_sorted(expected, TAG_OF[label])) @@ -237,13 +272,14 @@ def test_loaded_function_node_as_table_returns_stored_rows( def test_loaded_function_node_as_table_all_info_has_stream_columns( saved: pathlib.Path, label: str ) -> None: - """With ``all_info``, a loaded node's table has every live column and more.""" + """With ``all_info``, a loaded node's table, once a load-only run has filled it, + has every live column and more.""" # The loaded table follows the documented stream semantics: every column of the # live table, with the same values and types, plus the data context and the # source-info columns. (The live FunctionJobNode.as_table() drops those two # groups -- a separate, pre-existing bug the loaded node does not copy.) expected = pq.read_table(saved / "expected" / f"{label}.all_info.parquet") - table = _load(saved).nodes[label].as_table(all_info=True) + table = _loaded_and_filled(saved).nodes[label].as_table(all_info=True) data_column = next(iter(_load(saved).nodes[label].output_schema()[1])) assert "_context_key" in table.column_names assert f"_source_{data_column}" in table.column_names @@ -258,7 +294,7 @@ def test_loaded_function_node_iter_data_yields_stored_rows(saved: pathlib.Path) rows = sorted( ( (tag.as_dict(), data.as_dict()) - for tag, data in _load(saved).nodes["dbl"].iter_data() + for tag, data in _load(saved).nodes["dbl"].as_source().iter_data() ), key=lambda row: row[0]["id"], ) @@ -373,8 +409,10 @@ def test_other_context_records_do_not_satisfy_saved_expectation( node = PipelineJob.load(path, database=other).nodes["dbl"] # The store is the source of truth (ITL-1043): it holds none of the run's # records, so none is read; the job file's count makes it a warning. + # A loaded node is empty until it runs (ITL-1095): its stored view reads the store. + reader = node if read_method == "get_all_records" else node.as_source() with pytest.warns(StoreContentWarning, match="records 2 rows .* holds 0"): - result = getattr(node, read_method)() + result = getattr(reader, read_method)() rows = list(result) if read_method == "iter_data" else result assert rows is None or len(rows) == 0 @@ -382,15 +420,20 @@ def test_other_context_records_do_not_satisfy_saved_expectation( def test_recomputed_entry_is_read_once_with_its_latest_result( tmp_path: pathlib.Path, ) -> None: - """A loaded node reads one row per entry: the greatest recomputation index wins.""" + """A loaded node's load-only run reads one row per entry: the greatest recomputation + index wins.""" path = _save_small_job(tmp_path, _record_double) node = PipelineJob.load(path).nodes["dbl"] - entry_id = next(iter(node.iter_data()))[0].as_dict()["id"] + # A loaded node is empty until it runs (ITL-1095): read its stored view. + entry_id = next(iter(node.as_source().iter_data()))[0].as_dict()["id"] # Index 2 is stored before index 1, so "last record wins" would pick 111. _append_recomputed_record(node, entry_id, recomputation_index=2, value=222) _append_recomputed_record(node, entry_id, recomputation_index=1, value=111) - reloaded = PipelineJob.load(path).nodes["dbl"] + reloaded_job = PipelineJob.load(path) + reloaded_job.bind(sources={"dsrc": _dict_source()}) + _ = reloaded_job.run(load_only=True) + reloaded = reloaded_job.nodes["dbl"] table = reloaded.as_table() assert table.num_rows == 2 rows = {row["id"]: row["dbl"] for row in table.to_pylist()} @@ -418,6 +461,8 @@ def test_live_node_hot_load_reads_the_latest_recomputed_result( fresh = PipelineJob(name="small", database=store) with fresh: _record_double() + # A node shows rows only after a run (ITL-1095): the run loads every stored entry. + _ = fresh.run() values = { t.as_dict()["id"]: d.as_dict()["dbl"] for t, d in fresh.nodes["dbl"].iter_data() } @@ -428,7 +473,8 @@ def test_live_node_hot_load_reads_the_latest_recomputed_result( def test_loaded_node_with_ephemeral_results_does_not_crash( tmp_path: pathlib.Path, ) -> None: - """Ephemeral results come back as placeholders; the table says why it has none.""" + """Results kept in an ephemeral store are gone after the run: a load-only run of the + loaded job counts them as not stored and shows no rows; it does not crash.""" @function_pod(output_keys="dbl") def double(x: int) -> int: @@ -446,18 +492,12 @@ def make_ephemeral(job: PipelineJob) -> None: node.set_ephemeral_store(InMemoryArrowDatabase()) path = _save_small_job(tmp_path, record, configure=make_ephemeral) - node = PipelineJob.load(path).nodes["dbl"] - assert [type(data).__name__ for _, data in node.iter_data()] == [ - "EmptyData", - "EmptyData", - ] - with pytest.raises(RuntimeError) as excinfo: - _ = node.as_table() - assert str(excinfo.value) == ( - "FunctionJobNode 'dbl': the saved run kept 2 of its results in an ephemeral " - "store, which is not saved with the job, so they cannot be read into a table. " - "iter_data() yields them as EmptyData placeholders." - ) + loaded = PipelineJob.load(path) + loaded.bind(sources={"dsrc": _dict_source()}) + summary = loaded.run(load_only=True) + assert summary.nodes["dbl"].not_stored == 2 + assert summary.nodes["dbl"].reused == 0 + assert loaded.nodes["dbl"].as_table().num_rows == 0 def test_newer_unavailable_result_does_not_expose_old_persistent_result( @@ -482,10 +522,13 @@ def test_newer_unavailable_result_does_not_expose_old_persistent_result( records = records.set_column(index, field, pa.array(values, type=field.type)) store.add_records(table_path, records, record_id_column="__rid", flush=True) - node = PipelineJob.load(path).nodes["dbl"] - assert [type(data).__name__ for _, data in node.iter_data()] == ["EmptyData"] * 2 - with pytest.raises(RuntimeError, match="ephemeral store"): - node.as_table() + # The newest result of each entry is gone with its ephemeral store: a load-only run + # counts the entry as not stored and never shows the older persistent result. + loaded = PipelineJob.load(path) + loaded.bind(sources={"dsrc": _dict_source()}) + summary = loaded.run(load_only=True) + assert summary.nodes["dbl"].not_stored == 2 + assert loaded.nodes["dbl"].as_table().num_rows == 0 # --------------------------------------------------------------------------- @@ -503,11 +546,14 @@ def test_loaded_function_node_warns_once_on_a_store_without_its_runs_records( issued once per node. """ node = _load(saved, database=InMemoryArrowDatabase()).nodes["dbl"] + # A loaded node is empty until it runs (ITL-1095): the store is read through its + # stored view, which warns once. + stored = node.as_source() with pytest.warns(StoreContentWarning) as record: - table = node.as_table() - rows = list(node.iter_data()) + table = stored.as_table() + rows = list(stored.iter_data()) records = node.get_all_records() - head = node.head() + head = stored.head() assert table.num_rows == 0 assert table.column_names == ["id", "dbl"] assert rows == [] @@ -532,7 +578,7 @@ def test_store_content_warning_names_a_delta_stores_location( other = tmp_path / "other_store" node = _load(saved, database=DeltaTableDatabase(str(other))).nodes["dbl"] with pytest.warns(StoreContentWarning, match="the store at '") as record: - _ = node.as_table() + _ = node.as_source().as_table() assert str(other) in str(record[0].message) @@ -568,7 +614,7 @@ def double(x: int) -> int: assert node._pipeline_database is not None assert node._pipeline_database.table_exists(node._versioned_pipeline_path) with pytest.warns(StoreContentWarning, match="records 2 rows .* holds 0"): - assert node.as_table().num_rows == 0 + assert node.as_source().as_table().num_rows == 0 def test_saved_file_records_the_rows_each_node_had_in_the_store( @@ -598,7 +644,7 @@ def test_saving_a_loaded_job_keeps_the_expected_rows( assert run["expected_rows"] == original node = PipelineJob.load(str(resaved), database=InMemoryArrowDatabase()).nodes["dbl"] with pytest.warns(StoreContentWarning, match="records 2 rows"): - assert node.as_table().num_rows == 0 + assert node.as_source().as_table().num_rows == 0 def test_loaded_function_node_with_nothing_stored_returns_zero_rows( @@ -634,8 +680,9 @@ def drop_section(data: dict[str, Any]) -> None: with warnings.catch_warnings(): warnings.simplefilter("error", StoreContentWarning) job = PipelineJob.load(path, database=InMemoryArrowDatabase()) - assert job.nodes["dbl"].as_table().num_rows == 0 - assert PipelineJob.load(path).nodes["dbl"].as_table().num_rows == 2 + assert job.nodes["dbl"].as_source().as_table().num_rows == 0 + stored = PipelineJob.load(path).nodes["dbl"].as_source() + assert stored.as_table().num_rows == 2 def test_a_file_with_the_old_stored_records_flag_loads_without_a_check( @@ -689,7 +736,7 @@ def drop_store(data: dict[str, Any]) -> None: job = PipelineJob.load(_rewrite_job_file(saved, tmp_path, drop_store)) job.bind(database=DeltaTableDatabase(str(saved / "store"))) - assert job.nodes["dbl"].as_table().num_rows == 2 + assert job.nodes["dbl"].as_source().as_table().num_rows == 2 def test_loaded_job_still_cannot_run(saved: pathlib.Path) -> None: @@ -770,8 +817,8 @@ def test_loaded_function_node_cannot_execute(saved: pathlib.Path) -> None: _ = job.nodes["dbl"].execute(job.nodes["dsrc"]) assert str(excinfo.value) == ( "FunctionJobNode 'dbl' was loaded by PipelineJob.load and has no function " - "pod, so it cannot compute; read its stored results with as_table() or " - "iter_data()." + "pod, so it cannot compute; fill it from the store with run(load_only=True), " + "or read its stored results with as_source()." ) @@ -799,21 +846,19 @@ async def run() -> list[Any]: # --------------------------------------------------------------------------- -def test_loaded_operator_node_raises_instead_of_returning_empty( +def test_loaded_operator_node_reads_empty_and_says_it_has_not_run( saved: pathlib.Path, ) -> None: - """A loaded operator node, with nothing stored, raises and names its inputs.""" + """A loaded operator node, like every node, is empty until it runs (ITL-1095): no + read raises, and its head points to ``run()`` and ``as_source()``.""" + from orcapod.core.streams import ViewStatus + node = _load(saved).nodes["joined"] - expected = ( - "OperatorJobNode 'joined' has no stored results available for this read. " - "Operator outputs require a stored cache and replay configuration. Read the stored results of " - "its input nodes ('dbl', 'asrc') instead; an input source whose data was not " - "saved must be bound first with job.bind(sources=...)." - ) - for read in (node.as_table, lambda: list(node.iter_data())): - with pytest.raises(RuntimeError) as excinfo: - _ = read() - assert str(excinfo.value) == expected + assert node.as_table().num_rows == 0 + assert list(node.iter_data()) == [] + head = node.head() + assert head.status is ViewStatus.NOT_COMPUTED + assert "as_source()" in head.header() # --------------------------------------------------------------------------- @@ -867,7 +912,7 @@ def move_csv(data: dict[str, Any]) -> None: "source node 'csrc' itself, bind a source: job.bind(sources={'csrc': })." ) # The stored results downstream of the missing source are still readable. - assert job.nodes["inc"].as_table().num_rows == 2 + assert job.nodes["inc"].as_source().as_table().num_rows == 2 def test_unreconstructable_store_is_a_warning_with_the_remedy( @@ -976,5 +1021,8 @@ def timestamp(x: int) -> int: assert result is not None expected = result["value"] node.clear_cache() - assert node.as_table().column("value").to_pylist() == [expected] - assert [result["value"] for _, result in node.iter_data()] == [expected] + # A cleared node is empty until it runs (ITL-1095); the readback of the entry, the + # lookup a run makes, picks the newest recomputation from whichever store holds it. + assert node.as_table().num_rows == 0 + loaded = node.get_cached_results([node.compute_base_entry_id(tag, data)]) + assert [result["value"] for _, result in loaded.values()] == [expected] diff --git a/tests/test_pipeline/loaded_job/test_loaded_load_only.py b/tests/test_pipeline/loaded_job/test_loaded_load_only.py new file mode 100644 index 000000000..a313ebe7f --- /dev/null +++ b/tests/test_pipeline/loaded_job/test_loaded_load_only.py @@ -0,0 +1,166 @@ +"""A loaded job is empty until ``loaded.run(load_only=True)`` fills it (ITL-1095). + +Developer, November 13, 2026: "the initially loaded job is empty", and a load-only run +fills the nodes from what is stored. A loaded job has no function pods, so it can never +compute; with its sources available (rebuilt from files, or bound), a load-only run reads +them, runs the rebuilt operators, and gives each function node the results its store +holds for those inputs. ``loaded.run()`` without ``load_only`` still raises. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.operators import Join +from orcapod.core.streams import ViewStatus +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import ArrowTableSource, CSVSource, DictSource +from orcapod.types import CacheMode + +JOB = "loaded_load_only" + + +@function_pod(output_keys="dbl") +def double(x: int) -> int: + """Double *x*.""" + return 2 * x + + +@function_pod(output_keys="cc") +def increment(c: int) -> int: + """Add one to *c*.""" + return c + 1 + + +def _dict_source() -> DictSource: + return DictSource( + [{"id": 0, "x": 5}, {"id": 1, "x": 10}], tag_columns=["id"], source_id="dsrc" + ) + + +def _arrow_source() -> ArrowTableSource: + table = pa.table( + {"id": [0, 1], "y": [3, 4]}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("y", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="asrc") + + +def _save(tmp_path: Path, db: Any, cache_mode: CacheMode | None = None) -> Path: + """Run and save a job: ``dbl`` over an in-memory source joined with another one + (``joined``), and ``inc`` over a CSV source.""" + csv = tmp_path / "c.csv" + csv.write_text("id,c\n0,7\n1,8\n") + job = PipelineJob(name=JOB, database=db) + with job: + Join()(double.pod(_dict_source(), label="dbl"), _arrow_source(), label="joined") + increment.pod( + CSVSource(str(csv), tag_columns=["id"], source_id="csrc"), label="inc" + ) + if cache_mode is not None: + job.nodes["joined"].attach_databases( + pipeline_database=db.at(JOB), cache_mode=cache_mode + ) + job.run() + path = tmp_path / "job.json" + job.save(path) + return path + + +def _snapshot(db: Any) -> dict[tuple[str, ...], int]: + db.flush() + counts: dict[tuple[str, ...], int] = {} + for path in db.list_table_paths(): + records = db.get_all_records(path) + counts[path] = 0 if records is None else records.num_rows + return counts + + +def _bound(path: Path, db: Any) -> PipelineJob: + loaded = PipelineJob.load(path, database=db) + loaded.bind(sources={"dsrc": _dict_source(), "asrc": _arrow_source()}) + return loaded + + +@pytest.mark.parametrize("orchestrator", ["sync", "async"]) +def test_a_load_only_run_fills_a_loaded_job_and_writes_nothing( + tmp_path: Path, orchestrator: str +) -> None: + db = InMemoryArrowDatabase() + path = _save(tmp_path, db) + before = _snapshot(db) + loaded = _bound(path, db) + for label in ("dbl", "joined", "inc"): + assert loaded.nodes[label].as_table().num_rows == 0, label + if orchestrator == "sync": + summary = loaded.run(load_only=True) + else: + summary = asyncio.run(loaded.run_async(load_only=True)) + assert summary.load_only + assert summary.nodes["dbl"].reused == 2 + assert summary.nodes["inc"].reused == 2 + assert _snapshot(db) == before + dbl = {r["id"]: r["dbl"] for r in loaded.nodes["dbl"].as_table().to_pylist()} + assert dbl == {0: 10, 1: 20} + joined = loaded.nodes["joined"].as_table().to_pylist() + assert sorted((r["id"], r["dbl"], r["y"]) for r in joined) == [ + (0, 10, 3), + (1, 20, 4), + ] + assert loaded.nodes["dbl"].head().status is ViewStatus.LOADED_ONLY + + +def test_nodes_below_an_unbound_source_stay_empty(tmp_path: Path) -> None: + db = InMemoryArrowDatabase() + loaded = PipelineJob.load(_save(tmp_path, db), database=db) + summary = loaded.run(load_only=True) + # The CSV source is rebuilt from its file; the in-memory ones are not saved. + assert loaded.nodes["inc"].as_table().num_rows == 2 + assert loaded.nodes["dbl"].as_table().num_rows == 0 + assert loaded.nodes["joined"].as_table().num_rows == 0 + assert "dbl" not in summary.nodes + + +def test_a_loaded_job_still_cannot_compute(tmp_path: Path) -> None: + db = InMemoryArrowDatabase() + loaded = _bound(_save(tmp_path, db), db) + with pytest.raises(RuntimeError, match=r"run\(load_only=True\)"): + loaded.run() + with pytest.raises(RuntimeError, match=r"run\(load_only=True\)"): + loaded.nodes["dbl"].run() + + +def test_a_loaded_node_runs_load_only_with_what_it_depends_on(tmp_path: Path) -> None: + db = InMemoryArrowDatabase() + loaded = _bound(_save(tmp_path, db), db) + view = loaded.nodes["joined"].run(load_only=True) + assert view.as_table().num_rows == 2 + assert loaded.nodes["dbl"].as_table().num_rows == 2 + summary = loaded.nodes["joined"].last_run + assert summary is not None and summary.load_only + + +def test_a_loaded_replay_operator_replays_its_run_in_a_load_only_run( + tmp_path: Path, +) -> None: + db = InMemoryArrowDatabase() + path = _save(tmp_path, db, cache_mode=CacheMode.LOG) + loaded = _bound(path, db) + loaded.nodes["joined"].attach_databases( + pipeline_database=db.at(JOB), cache_mode=CacheMode.REPLAY + ) + before = _snapshot(db) + loaded.run(load_only=True) + assert _snapshot(db) == before + assert loaded.nodes["joined"].as_table().num_rows == 2 diff --git a/tests/test_pipeline/loaded_job/test_loaded_logical_schemas.py b/tests/test_pipeline/loaded_job/test_loaded_logical_schemas.py index b4b90b38d..09f204881 100644 --- a/tests/test_pipeline/loaded_job/test_loaded_logical_schemas.py +++ b/tests/test_pipeline/loaded_job/test_loaded_logical_schemas.py @@ -127,7 +127,8 @@ def test_registered_logical_output_keeps_join_available( loaded = op.PipelineJob.load(path) assert loaded.nodes["value"].output_schema()[1]["value"] is expected_type - assert loaded.nodes["value"].as_table().num_rows == 1 + # A loaded node is empty until it runs (ITL-1095): read its stored view. + assert loaded.nodes["value"].as_source().as_table().num_rows == 1 assert loaded.nodes["joined"].load_status == LoadStatus.FULL assert loaded.nodes["joined"].output_schema()[1]["value"] is expected_type diff --git a/tests/test_pipeline/loaded_job/test_loaded_operator_cache.py b/tests/test_pipeline/loaded_job/test_loaded_operator_cache.py index a362b6920..c0a885efe 100644 --- a/tests/test_pipeline/loaded_job/test_loaded_operator_cache.py +++ b/tests/test_pipeline/loaded_job/test_loaded_operator_cache.py @@ -4,10 +4,11 @@ second one and stores the join's output (``CacheMode.LOG``). Columns are declared nullable or not. A job file records each column's nullability and the run content hash of every operator node, so the loaded job reproduces each node's pipeline hash, content hash, table -path and output schema, and a loaded operator node attached in ``CacheMode.REPLAY`` reads -the rows its run stored. A file without those fields (``0.1.0``, or a ``0.2.0`` file written -before them) loads as before, lossy: the operator node still looks under its run's path, -and raises rather than show the stored rows as none. +path and output schema, and the stored view (``as_source()``) of a loaded operator node +attached in ``CacheMode.REPLAY`` reads the rows its run stored. A loaded node itself is +empty until it runs (ITL-1095). A file without those fields (``0.1.0``, or a ``0.2.0`` file +written before them) loads as before, lossy: the operator node still looks under its run's +path, and its stored view raises rather than show the stored rows as none. """ from __future__ import annotations @@ -178,17 +179,20 @@ def test_every_node_keeps_its_identity_and_schema_through_save_and_load( def test_loaded_replay_operator_reads_the_rows_its_run_stored( store_kind: str, nullability: str, read: str, tmp_path: pathlib.Path ) -> None: - """A loaded operator node in REPLAY reads exactly the rows read before saving.""" + """A loaded operator node's stored view reads exactly the rows read before saving; + the node itself is empty until it runs (ITL-1095).""" store = _store(store_kind, tmp_path) job = _run(store, nullability) expected = _sorted(job.nodes["joined"].as_table()) node = _load_replaying(_save(job, tmp_path), store).nodes["joined"] + assert node.as_table().num_rows == 0 + stored = node.as_source() if read == "as_table": - assert _sorted(node.as_table()).equals(expected) + assert _sorted(stored.as_table()).equals(expected) elif read == "iter_data": - assert len(list(node.iter_data())) == expected.num_rows + assert len(list(stored.iter_data())) == expected.num_rows else: - assert _sorted(node.head().table).equals(expected) + assert _sorted(stored.head().table).equals(expected) @pytest.mark.parametrize("nullability", list(NULLABILITY)) @@ -214,9 +218,9 @@ def rebuilt(cache_mode: CacheMode) -> PipelineJob: ) return again - assert _sorted(rebuilt(CacheMode.REPLAY).nodes["joined"].as_table()).equals( - expected - ) + replaying = rebuilt(CacheMode.REPLAY) + _ = replaying.run() # a node shows rows only after a run (ITL-1095) + assert _sorted(replaying.nodes["joined"].as_table()).equals(expected) _ = rebuilt(CacheMode.LOG).run() assert set(store.at(JOB_NAME).list_table_paths()) == tables_after_run @@ -238,7 +242,9 @@ def test_without_saved_nullability_the_operator_still_reads_its_runs_table( node = _load_replaying(saved, store).nodes["joined"] assert node.pipeline_hash() == job.nodes["joined"].pipeline_hash() assert node.node_identity_path == job.nodes["joined"].node_identity_path - assert _sorted(node.as_table()).equals(_sorted(job.nodes["joined"].as_table())) + assert _sorted(node.as_source().as_table()).equals( + _sorted(job.nodes["joined"].as_table()) + ) def _drop_both(data: dict[str, Any]) -> None: @@ -259,12 +265,13 @@ def _drop_both(data: dict[str, Any]) -> None: def test_without_the_run_content_hash_the_loaded_operator_raises( store_kind: str, nullability: str, read: str, tmp_path: pathlib.Path ) -> None: - """A node that cannot name its run's rows never shows the stored rows as none.""" + """A node that cannot name its run's rows never shows the stored rows as none: its + stored view raises.""" store = _store(store_kind, tmp_path) job = _run(store, nullability) node = _load_replaying(_save(job, tmp_path, _drop_both), store).nodes["joined"] with pytest.raises(RuntimeError, match=UNKNOWN_KEY) as excinfo: - result = getattr(node, read)() + result = getattr(node.as_source(), read)() if read == "iter_data": _ = list(result) assert "holds 2 stored rows, none of them under the content hash" in str( @@ -279,23 +286,24 @@ def test_without_the_run_content_hash_an_absent_table_raises_too( saved = _save(_run(InMemoryArrowDatabase(), "non_null"), tmp_path, _drop_both) node = _load_replaying(saved, InMemoryArrowDatabase()).nodes["joined"] with pytest.raises(RuntimeError, match=UNKNOWN_KEY) as excinfo: - _ = node.as_table() + _ = node.as_source().as_table() assert "is absent from the attached store" in str(excinfo.value) @pytest.mark.parametrize("store_kind", STORES) -def test_without_the_run_content_hash_head_says_why_it_shows_no_rows( +def test_without_the_run_content_hash_head_never_raises_and_the_stored_view_says_why( store_kind: str, tmp_path: pathlib.Path ) -> None: - """``head()`` never raises on a loaded node: it shows no rows and names the cause.""" + """``head()`` never raises on a loaded node: it has not run (ITL-1095). Its stored + view raises and names the cause.""" store = _store(store_kind, tmp_path) job = _run(store, "nullable_tag") node = _load_replaying(_save(job, tmp_path, _drop_both), store).nodes["joined"] head = node.head() - assert head.status is ViewStatus.UNAVAILABLE + assert head.status is ViewStatus.NOT_COMPUTED assert head.table.num_rows == 0 - assert "does not record its run's content hash" in head.header() - assert "ITL-1048" in head.header() + with pytest.raises(RuntimeError, match=UNKNOWN_KEY): + _ = node.as_source().head() # --------------------------------------------------------------------------- @@ -317,16 +325,19 @@ def _live_replaying(store: ArrowDatabaseProtocol, job: PipelineJob) -> Any: def _assert_reads_as_live_with_no_rows( loaded_node: Any, live_node: Any, *, same_types: bool = True ) -> None: - """The loaded node reads what a running node reads from the same store: no rows. + """The loaded node's stored view reads what a running node's reads from the same + store: no rows. ``same_types=False`` where the loaded node's input slots are unbound: a running node types its zero-row table from its sources (ITL-912), while the loaded node has only their Python schema, so only the user columns are compared (ITL-1089). """ user_columns = [*live_node.keys()[0], *live_node.keys()[1]] + loaded_stored = loaded_node.as_source() + live_stored = live_node.as_source() for config in ({}, {"all_info": True}): - table = loaded_node.as_table(**config) - live = live_node.as_table(**config) + table = loaded_stored.as_table(**config) + live = live_stored.as_table(**config) assert table.num_rows == 0 if same_types: assert table.schema.equals(live.schema), (table.schema, live.schema) @@ -334,12 +345,10 @@ def _assert_reads_as_live_with_no_rows( assert [n for n in table.column_names if n in user_columns] == [ n for n in live.column_names if n in user_columns ] - assert list(loaded_node.iter_data()) == [] - head = loaded_node.head() - assert head.status is ViewStatus.NOTHING_STORED + assert list(loaded_stored.iter_data()) == [] + head = loaded_stored.head() assert head.table.num_rows == 0 - assert "the store holds no rows for its run" in head.header() - assert live_node.head().status is ViewStatus.NOTHING_STORED + assert head.status is live_stored.head().status def test_a_store_holding_only_other_runs_rows_gives_an_empty_result( @@ -412,8 +421,9 @@ def test_an_absent_cache_table_gives_the_running_nodes_types( empty_store = InMemoryArrowDatabase() node = _load_replaying(_save(job, tmp_path), empty_store).nodes["joined"] live = _live_replaying(empty_store, job) - with pytest.warns(StoreContentWarning, match="records 2 rows .* holds 0"): - table = node.as_table(all_info=True) + # Neither node has run: both read their zero-row tables, and neither reads the + # store (ITL-1095), so the job file's row count is not checked here. + table = node.as_table(all_info=True) assert table.schema.equals(live.as_table(all_info=True).schema) @@ -596,7 +606,8 @@ def source(column: str, values: list[int]) -> ArrowTableSource: def test_a_format_0_1_0_file_finds_its_join_table_and_never_shows_its_rows_as_none() -> ( None ): - """The real 0.1.0 file: the run's path, ``head()`` says why, ``as_table()`` raises.""" + """The real 0.1.0 file: the run's path; the node has not run (ITL-1095) and its + stored view raises.""" store = InMemoryArrowDatabase() run_join = _store_the_fixture_join(store) loaded = PipelineJob.load(str(FORMAT_0_1_0), database=store) @@ -607,13 +618,11 @@ def test_a_format_0_1_0_file_finds_its_join_table_and_never_shows_its_rows_as_no ) assert node.pipeline_hash() == run_join.pipeline_hash() assert node.node_identity_path == run_join.node_identity_path - head = node.head() - assert head.status is ViewStatus.UNAVAILABLE - assert "ITL-1048" in head.header() + assert node.head().status is ViewStatus.NOT_COMPUTED with pytest.raises( RuntimeError, match=r"'Join': the job file does not record .*ITL-1048" ): - _ = node.as_table() + _ = node.as_source().as_table() # --------------------------------------------------------------------------- @@ -636,14 +645,15 @@ def test_get_cached_output_of_a_loaded_node_reads_its_runs_rows( def test_get_cached_output_of_a_loaded_node_never_caches_unreadable_rows_as_none( tmp_path: pathlib.Path, ) -> None: - """``get_cached_output()`` raises as ``as_table()`` does; later reads raise too.""" + """``get_cached_output()`` raises as the stored view does; the node keeps nothing.""" store = InMemoryArrowDatabase() job = _run(store, "nullable_tag") node = _load_replaying(_save(job, tmp_path, _drop_both), store).nodes["joined"] with pytest.raises(RuntimeError, match="the job file does not record"): _ = node.get_cached_output() with pytest.raises(RuntimeError, match="the job file does not record"): - _ = node.as_table() + _ = node.as_source().as_table() + assert node.as_table().num_rows == 0 # --------------------------------------------------------------------------- @@ -680,11 +690,12 @@ def drop_src_a(data: dict[str, Any]) -> None: def _read_all(job: PipelineJob, label: str) -> pa.Table: - """Read a loaded node three ways and return its table.""" - node = job.nodes[label] - _ = list(node.iter_data()) - _ = node.head() - return node.as_table() + """Read a loaded node's stored view three ways and return its table (the node itself + is empty until it runs, ITL-1095).""" + stored = job.nodes[label].as_source() + _ = list(stored.iter_data()) + _ = stored.head() + return stored.as_table() def _loaded(saved: pathlib.Path, store: ArrowDatabaseProtocol) -> PipelineJob: @@ -757,7 +768,7 @@ def test_matching_counts_and_zero_expected_zero_found_do_not_warn( "joined": 0 } node = _load_replaying(empty_saved, InMemoryArrowDatabase()).nodes["joined"] - assert node.as_table().num_rows == 0 + assert node.as_source().as_table().num_rows == 0 def test_deleting_the_section_by_hand_loads_identically_without_a_check( diff --git a/tests/test_pipeline/test_async_error_policy.py b/tests/test_pipeline/test_async_error_policy.py index 5d8cb2d13..e05ea7952 100644 --- a/tests/test_pipeline/test_async_error_policy.py +++ b/tests/test_pipeline/test_async_error_policy.py @@ -202,8 +202,9 @@ def ks(job: PipelineJob, label: str) -> list[str]: def stored(store: Any, build: Callable[[Any, list[int]], PipelineJob], label: str): - """The rows a fresh job over ``store`` reads for ``label`` without running.""" - return build(store, xs_failing_at(0)).nodes[label].as_table() + """Every row ``store`` holds for ``label``: the node's stored view (``as_source()``; + a node itself shows only its current run, ITL-1095).""" + return build(store, xs_failing_at(0)).nodes[label].as_source().as_table() @pytest.fixture(params=list(ORCHESTRATORS)) @@ -355,10 +356,10 @@ def test_fail_fast_leaves_the_nodes_with_no_run( ) -> None: """After a fail_fast raise, the failing node and the node after it hold no run. - A run that raised defines no rows (ITL-1093): each node reads as a node with no run, - every row stored under its content hash, also the downstream node the async run - cancelled. Another job stores rows of other values first, so a node still holding - its previous run's rows reads fewer rows than a node with no run. + A run that raised defines no rows (ITL-1093): each node it ran reads as a node with no + run, which is empty (ITL-1095), also the downstream node the async run cancelled; what + was stored stays readable through ``as_source()``. Another job stores rows of other + values first, so the stored view holds both jobs' rows. """ store = InMemoryArrowDatabase() run_job(build(store, [100 + i for i in range(N_ROWS)]), orchestrator, "continue") @@ -372,10 +373,11 @@ def test_fail_fast_leaves_the_nodes_with_no_run( run_job(job, orchestrator, "fail_fast") for label in ("f", "down"): - no_run = rows_of(build(store, xs).nodes[label].as_table()) - assert rows_of(job.nodes[label].as_table()) == no_run, label - # Also the other job's rows: not the previous run's rows only. - assert len(no_run) == 2 * N_ROWS - 1, label + node = job.nodes[label] + assert node.as_table().num_rows == 0, label + assert node.is_stale, label + # Every stored row stays: the other job's rows and this job's. + assert len(rows_of(node.as_source().as_table())) == 2 * N_ROWS - 1, label # The row ``sibling`` leaves uncomputed until the run that is cancelled. @@ -407,7 +409,8 @@ def test_a_node_cancelled_on_another_branch_holds_no_run() -> None: """A node the async run cancels, not downstream of the failure, holds no run either. ``g`` waits at its one uncomputed row until the failure of ``f`` cancels it. Its - previous run's rows are not held: it reads as a node with no run (ITL-1093). + previous run's rows are not held: it reads as a node with no run (ITL-1093), which is + empty (ITL-1095); its stored rows stay readable through ``as_source()``. """ store = InMemoryArrowDatabase() other = [100 + i for i in range(N_ROWS)] @@ -425,9 +428,10 @@ def test_a_node_cancelled_on_another_branch_holds_no_run() -> None: finally: sibling_mode["mode"] = "fail" - no_run = rows_of(branches_job(store, xs).nodes["g"].as_table()) - assert rows_of(job.nodes["g"].as_table()) == no_run - assert len(no_run) == 2 * N_ROWS - 1 + g = job.nodes["g"] + assert g.as_table().num_rows == 0 + assert g.is_stale + assert len(rows_of(g.as_source().as_table())) == 2 * N_ROWS - 1 def test_an_observer_failing_at_node_end_is_called_once(orchestrator: str) -> None: diff --git a/tests/test_pipeline/test_current_run_review_fixes.py b/tests/test_pipeline/test_current_run_review_fixes.py new file mode 100644 index 000000000..429ccde6a --- /dev/null +++ b/tests/test_pipeline/test_current_run_review_fixes.py @@ -0,0 +1,893 @@ +"""Review fixes for the current-run rule and load-only runs (ITL-1095, PR #380 review). + +Each test names the guarantee it protects: a run that raised (in its observer, its store, +its commit or its output) leaves no node holding its rows; a load-only run writes nothing, +not even pending writes of others; an operator filled by a load-only run says so; the +freshness check stays linear on a DAG of shared ancestors. +""" + +from __future__ import annotations + +import asyncio +import json +from pathlib import Path +from typing import Any, Literal +from unittest import mock + +import pyarrow as pa +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.nodes import OperatorJobNode +from orcapod.core.nodes.source_node import SourceJobNode +from orcapod.core.operators import Join, MapData, SelectDataColumns +from orcapod.core.streams import ArrowTableStream, ViewStatus +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.errors import StoreWriteError +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.pipeline.observer import NoOpObserver +from orcapod.side_effects import SideEffectPod, SideEffectPodConfig, side_effect_pod +from orcapod.sources import ArrowTableSource, DictSource +from orcapod.types import CacheMode + +CALLS: list[str] = [] +FAIL = {"tap": False} + + +@function_pod(output_keys="y") +def inc(x: int) -> int: + """Return ``x + 1``.""" + CALLS.append("inc") + return x + 1 + + +@function_pod(output_keys="z") +def dbl(y: int) -> int: + """Return ``2 * y``.""" + CALLS.append("dbl") + return 2 * y + + +@function_pod(output_keys="w") +def neg(z: int) -> int: + """Return ``-z``.""" + CALLS.append("neg") + return -z + + +@function_pod(output_keys="z") +def fails_on_21(y: int) -> int: + """Return ``2 * y``; raise for ``y == 21``.""" + if y == 21: + raise ValueError("y is 21") + return 2 * y + + +@function_pod(output_keys="total") +def total(y: list[int]) -> int: + """Return the sum of a group's values.""" + return sum(y) + + +def _deliver(y: int, ctx: Any) -> None: + """A side effect that fails while ``FAIL['tap']`` is set.""" + if FAIL["tap"]: + raise RuntimeError("delivery failed") + + +ROWS = [{"g": i % 2, "k": i, "x": 10 * i} for i in range(6)] +ORCHESTRATORS = {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator} + + +@pytest.fixture(autouse=True) +def _reset() -> None: + """Start every test with no recorded calls and a delivery that succeeds.""" + CALLS.clear() + FAIL["tap"] = False + + +def _source(data: list[dict[str, int]] = ROWS) -> DictSource: + """An in-memory source of ``data`` with tags ``g`` and ``k``.""" + return DictSource(data, tag_columns=["g", "k"], source_id="s") + + +def _grouped_job(data: list[dict[str, int]], db: Any) -> PipelineJob: + """``rowwise`` -> ``grouped`` (GroupBy) -> ``summed``.""" + job = PipelineJob(name="j", database=db) + with job: + rows: Any = inc.pod(_source(data), label="rowwise") + total.pod(rows.group_by("g", label="grouped"), label="summed") + return job + + +def _chain_job(data: list[dict[str, int]], db: Any) -> PipelineJob: + """``a`` (inc) -> ``b`` (dbl) -> ``c`` (neg).""" + job = PipelineJob(name="c", database=db) + with job: + neg.pod(dbl.pod(inc.pod(_source(data), label="a"), label="b"), label="c") + return job + + +class FailsAtNodeEnd(NoOpObserver): + """An observer whose ``on_node_end`` raises for one node.""" + + def __init__(self, label: str) -> None: + """Keep the label of the node this observer acts at.""" + self.label = label + + def on_node_end(self, node_label: str, node_hash: str) -> None: + """Raise at the node's end, as a broken observer would.""" + if node_label == self.label: + raise RuntimeError("observer failed") + + def contextualize(self, *identity_path: str) -> FailsAtNodeEnd: + """Keep this observer for every node.""" + return self + + +# --------------------------------------------------------------------------- +# A run that raised leaves no node holding its rows +# --------------------------------------------------------------------------- + + +def test_an_observer_failing_at_the_end_of_a_load_only_node_leaves_it_empty() -> None: + """An observer raising at a load-only node's end leaves the node empty and stale.""" + db = InMemoryArrowDatabase() + _grouped_job(ROWS, db).run() + job = _grouped_job(ROWS, db) + with pytest.raises(RuntimeError, match="observer failed"): + job.run(load_only=True, observer=FailsAtNodeEnd("rowwise")) + assert job.nodes["rowwise"].as_table().num_rows == 0 + assert job.nodes["rowwise"].is_stale + + +def test_a_failed_commit_of_a_node_run_alone_leaves_it_empty() -> None: + """A node run alone whose commit fails holds nothing afterwards.""" + from orcapod.core.nodes import FunctionJobNode + + table = pa.table( + {"k": [0, 1], "x": [1, 2]}, + schema=pa.schema( + [ + pa.field("k", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + node = FunctionJobNode( + inc.pod, # pyright: ignore[reportArgumentType] # the decorator's pod type is not declared as FunctionPodProtocol + ArrowTableStream(table, tag_columns=["k"]), + pipeline_database=InMemoryArrowDatabase(), + ) + + def fail(self: Any) -> None: + """A flush that always fails.""" + raise OSError("disk full") + + with ( + mock.patch.object(InMemoryArrowDatabase, "flush", fail), + pytest.raises(StoreWriteError), + ): + node.run() + assert node.as_table().num_rows == 0 + assert node.is_stale + + +def test_a_failed_run_clears_every_node_it_ran() -> None: + """A downstream failure (fail_fast) leaves the upstream node that ran empty too. + + Sync only: the asynchronous orchestrator does not stop for ``fail_fast`` (ITL-879). + """ + job = PipelineJob(name="f", database=InMemoryArrowDatabase()) + with job: + fails_on_21.pod(inc.pod(_source(), label="up"), label="down") + with pytest.raises(ValueError, match="y is 21"): + job.run(orchestrator=SyncPipelineOrchestrator(error_policy="fail_fast")) + assert job.nodes["up"].as_table().num_rows == 0 + assert job.nodes["up"].is_stale + + +def test_a_failed_cascade_clears_every_node_it_ran() -> None: + """A node.run() cascade whose commit fails clears every node it ran.""" + job = PipelineJob(name="f", database=InMemoryArrowDatabase()) + with job: + fails_on_21.pod(inc.pod(_source(), label="up"), label="down") + + def fail(self: Any) -> None: + """A flush that always fails.""" + raise OSError("disk full") + + with ( + mock.patch.object(InMemoryArrowDatabase, "flush", fail), + pytest.raises(StoreWriteError), + ): + job.nodes["down"].run() + for label in ("up", "down"): + assert job.nodes[label].as_table().num_rows == 0, label + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_failed_operator_store_leaves_the_operator_empty(orchestrator: str) -> None: + """An operator whose LOG write fails holds nothing afterwards.""" + db = InMemoryArrowDatabase() + job = PipelineJob(name="o", database=db) + with job: + SelectDataColumns(["x"])(_source(), label="sel") + job.nodes["sel"].attach_databases( + pipeline_database=db.at("o"), cache_mode=CacheMode.LOG + ) + with ( + mock.patch.object( + InMemoryArrowDatabase, "add_records", side_effect=OSError("disk full") + ), + pytest.raises((StoreWriteError, ExceptionGroup)), + ): + job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert job.nodes["sel"].as_table().num_rows == 0 + assert job.nodes["sel"].is_stale + + +def _tap_job(db: Any) -> PipelineJob: + """``rows`` -> ``tap`` (side effect, raises while FAIL is set) -> ``down``.""" + tap = side_effect_pod(_deliver, config=SideEffectPodConfig(track_completion=False)) + assert isinstance(tap, SideEffectPod) + job = PipelineJob(name="t", database=db) + with job: + tapped: Any = tap.process(inc.pod(_source(), label="rows"), label="tap") + dbl.pod(tapped.select_data_columns("y"), label="down") + return job + + +def test_a_failed_side_effect_run_makes_the_nodes_below_it_stale() -> None: + """A failed side-effect run makes it and every node below it stale.""" + job = _tap_job(InMemoryArrowDatabase()) + job.run() + assert job.nodes["down"].as_table().num_rows == 6 + job.nodes["tap"].clear_cache() + FAIL["tap"] = True + with pytest.raises(RuntimeError, match="delivery failed"): + job.nodes["tap"].run() + assert job.nodes["tap"].is_stale + assert job.nodes["down"].is_stale + assert job.nodes["down"].as_table().num_rows == 0 + + +def test_a_side_effect_node_whose_output_cannot_close_holds_nothing() -> None: + """A side-effect run whose output close fails holds no rows.""" + from orcapod.channels import Channel + + job = _tap_job(InMemoryArrowDatabase()) + job.run() + node = job.nodes["tap"] + node.clear_cache() + + class CloseFails: + """An output writer that keeps what it is sent and raises on close.""" + + async def send(self, item: Any) -> None: + """Accept ``item``.""" + + async def close(self) -> None: + """Raise, as a broken channel would.""" + raise OSError("output cannot be closed") + + async def run() -> None: + """Run the node's channel entry point.""" + inputs: Channel[Any] = Channel(buffer_size=16) + for item in job.nodes["rows"].iter_data(): + await inputs.writer.send(item) + await inputs.writer.close() + # The node's channel entry point is what this test is about. + await node.async_execute([inputs.reader], CloseFails()) # pyright: ignore[reportArgumentType] # a test double of the writer + + with pytest.raises(OSError, match="cannot be closed"): + asyncio.run(run()) + assert node.as_table().num_rows == 0 + assert node.is_stale + + +# --------------------------------------------------------------------------- +# A load-only run writes nothing, pending writes of others included +# --------------------------------------------------------------------------- + + +def _pending(db: InMemoryArrowDatabase) -> dict[str, int]: + """The store's pending batches with their row counts (private on purpose: what a + load-only run must leave as it found it).""" + return {key: table.num_rows for key, table in db._pending_batches.items()} # pyright: ignore[reportPrivateUsage] # the pending batches are what this test is about + + +def _add_pending_write(db: InMemoryArrowDatabase) -> None: + """Leave one uncommitted write of another table in ``db``.""" + db.add_records( + ("elsewhere",), + pa.table({"__rid": [b"r1"], "v": [1]}), + record_id_column="__rid", + ) + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_load_only_run_leaves_pending_writes_as_it_found_them( + orchestrator: str, +) -> None: + """A load-only job run and cascade flush nothing: pending writes stay pending.""" + db = InMemoryArrowDatabase() + _grouped_job(ROWS, db).run() + _add_pending_write(db) + before = _pending(db) + assert before + job = _grouped_job(ROWS, db) + with mock.patch.object( + InMemoryArrowDatabase, "flush", side_effect=AssertionError("flushed") + ): + job.run(orchestrator=ORCHESTRATORS[orchestrator](), load_only=True) + cascade = _grouped_job(ROWS, db) + cascade.nodes["summed"].run(load_only=True) + assert _pending(db) == before + assert job.nodes["summed"].as_table().num_rows == 2 + + +# --------------------------------------------------------------------------- +# Operators: the standalone LOG write is committed; load-only output is marked +# --------------------------------------------------------------------------- + + +def _source_table() -> ArrowTableSource: + """A three-row Arrow source with tag ``k`` and data ``x``, non-null.""" + table = pa.table( + {"k": [0, 1, 2], "x": [1, 2, 3]}, + schema=pa.schema( + [ + pa.field("k", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["k"], source_id="t") + + +def test_a_standalone_log_run_commits_its_write(tmp_path: Path) -> None: + """A standalone LOG operator run commits its write before returning.""" + node = OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[_source_table()], + pipeline_database=DeltaTableDatabase(tmp_path), + cache_mode=CacheMode.LOG, + ) + node.run() + reopened = OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[_source_table()], + pipeline_database=DeltaTableDatabase(tmp_path), + cache_mode=CacheMode.LOG, + ) + stored = reopened.get_all_records() + assert stored is not None and stored.num_rows == 3 + + +def test_a_standalone_log_run_after_a_load_only_run_stores(tmp_path: Path) -> None: + """A full standalone run after a load-only run computes and stores.""" + db = InMemoryArrowDatabase() + node = OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[_source_table()], + pipeline_database=db, + cache_mode=CacheMode.LOG, + ) + node.run(load_only=True) + assert node.get_all_records() is None + node.run() + stored = node.get_all_records() + assert stored is not None and stored.num_rows == 3 + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_an_operator_filled_by_a_load_only_run_says_so(orchestrator: str) -> None: + """An operator filled by a load-only run shows ``loaded_only``.""" + db = InMemoryArrowDatabase() + _grouped_job(ROWS, db).run() + job = _grouped_job(ROWS, db) + job.run(orchestrator=ORCHESTRATORS[orchestrator](), load_only=True) + grouped = job.nodes["grouped"] + assert grouped.head().status is ViewStatus.LOADED_ONLY + assert grouped.run(load_only=True).status is ViewStatus.LOADED_ONLY + + +def test_an_unrun_loaded_operator_without_its_operator_keeps_its_columns( + tmp_path: Path, +) -> None: + """An operator node rebuilt without its operator keeps its saved columns.""" + db = InMemoryArrowDatabase() + job = PipelineJob(name="l", database=db) + with job: + Join()( + DictSource([{"k": 0, "a": 1}], tag_columns=["k"], source_id="a"), + DictSource([{"k": 0, "b": 2}], tag_columns=["k"], source_id="b"), + label="joined", + ) + job.run() + job.save(tmp_path / "job.json") + nodes = json.loads((tmp_path / "job.json").read_text())["pipeline"]["nodes"] + (descriptor,) = [d for d in nodes.values() if d.get("label") == "joined"] + # A node rebuilt without its operator is what this test is about. + stub = OperatorJobNode.from_descriptor( + descriptor, operator=None, input_streams=(), databases={} + ) + table = stub.as_table() + assert table.num_rows == 0 + assert sorted(table.column_names) == ["a", "b", "k"] + + +# --------------------------------------------------------------------------- +# Staleness and summaries +# --------------------------------------------------------------------------- + + +def test_a_bound_source_with_no_modified_time_makes_its_slot_unknown() -> None: + """A source slot bound to a stream with no modified time has none.""" + stream = ArrowTableStream(pa.table({"k": [0], "x": [1]}), tag_columns=["k"]) + slot = SourceJobNode(name="s", bound_source=stream) + assert slot.last_modified is not None + stream._set_modified_time(None) # pyright: ignore[reportPrivateUsage] # an unknown modified time is what this test is about + assert slot.last_modified is None + + +def test_freshness_of_shared_ancestors_is_checked_once_per_traversal() -> None: + """Five split-and-join stages: the check visits each node once, not once per path. + + (Five stages: building a deeper job of this shape is itself slow, independently of + this check.) + """ + from orcapod.core.streams.base import StreamBase + + job = PipelineJob(name="d", database=InMemoryArrowDatabase()) + with job: + current: Any = inc.pod(_source(), label="start") + for stage in range(5): + left = MapData({"y": f"l{stage}"})(current, label=f"left{stage}") + right = MapData({"y": f"r{stage}"})(current, label=f"right{stage}") + joined = Join()(left, right, label=f"join{stage}") + picked = SelectDataColumns([f"l{stage}"])(joined, label=f"pick{stage}") + current = MapData({f"l{stage}": "y"})(picked, label=f"back{stage}") + job.run() + original = StreamBase.is_stale.fget + assert original is not None + calls = 0 + + def counting(self: Any) -> bool: + """Count the staleness checks, then check.""" + nonlocal calls + calls += 1 + return original(self) + + with mock.patch.object(StreamBase, "is_stale", property(counting)): + assert job.nodes["back4"].holds_current_run() + # One own-staleness check per job node of the ancestry (26 nodes), not per path. + assert calls <= 26 + + +def test_a_held_dependency_keeps_its_incomplete_inputs_in_the_summary() -> None: + """A -> B -> C, load-only: A skipped inputs; C cleared and loaded again still names A.""" + db = InMemoryArrowDatabase() + _chain_job(ROWS[:3], db).run() + job = _chain_job(ROWS, db) + job.run(load_only=True) + job.nodes["c"].clear_cache() + summary = job.run(load_only=True) + assert summary.nodes["c"].inputs_incomplete == ("a",) + + +def test_a_run_with_every_node_held_says_so() -> None: + """The headline of a run with every node held says there was nothing to do.""" + job = _grouped_job(ROWS, InMemoryArrowDatabase()) + job.run() + summary = job.run() + assert repr(summary).splitlines()[0] == ( + "Run of job 'j': nothing to do; every node holds a current run." + ) + + +# --------------------------------------------------------------------------- +# A loaded job missing a node (ITL-1094) fails load-only with a clear error +# --------------------------------------------------------------------------- + + +def _deliver_x(x: int, ctx: Any) -> None: + """A side effect over the source's own column.""" + + +def test_a_loaded_job_without_its_side_effect_node_names_it(tmp_path: Path) -> None: + """A loaded job missing a side-effect node fails load-only naming ITL-1094.""" + tap = side_effect_pod(_deliver_x) + assert isinstance(tap, SideEffectPod) + db = InMemoryArrowDatabase() + job = PipelineJob(name="m", database=db) + with job: + tapped: Any = tap.process(_source(), label="delivered") + inc.pod(tapped, label="down") + job.run() + job.save(tmp_path / "job.json") + loaded = PipelineJob.load(tmp_path / "job.json", database=db) + loaded.bind(sources={"s": _source()}) + with pytest.raises(RuntimeError, match=r"'down'.*ITL-1094"): + loaded.run(load_only=True) + + +# --------------------------------------------------------------------------- +# Second review round +# --------------------------------------------------------------------------- + +FLAKY = {"fail": True} + + +@function_pod(output_keys="y") +def flaky(x: int) -> int: + """Return ``x + 1``; raise for ``x == 20`` while ``FLAKY['fail']`` is set.""" + CALLS.append("flaky") + if x == 20 and FLAKY["fail"]: + raise ValueError("transient failure") + return x + 1 + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_run_with_failed_rows_is_retried_by_the_next_run(orchestrator: str) -> None: + """A run that left failed rows is never current: the next run computes exactly + the failed rows and reads the rest; a third, unchanged run runs nothing.""" + FLAKY["fail"] = True + job = PipelineJob(name="r", database=InMemoryArrowDatabase()) + with job: + flaky.pod(_source(), label="node") + with pytest.warns(Warning): + job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert job.nodes["node"].as_table().num_rows == 5 + CALLS.clear() + FLAKY["fail"] = False + summary = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert CALLS == ["flaky"] + assert (summary.nodes["node"].computed, summary.nodes["node"].reused) == (1, 5) + assert job.nodes["node"].as_table().num_rows == 6 + CALLS.clear() + summary = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert CALLS == [] + assert summary.nodes["node"].held == 6 + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_an_unchanged_rerun_touches_no_store(orchestrator: str) -> None: + """A re-run with nothing to run flushes nothing.""" + db = InMemoryArrowDatabase() + job = _grouped_job(ROWS, db) + job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + _add_pending_write(db) + before = _pending(db) + with mock.patch.object( + InMemoryArrowDatabase, "flush", side_effect=AssertionError("flushed") + ): + job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert _pending(db) == before + + +def test_a_held_load_only_node_keeps_its_missing_inputs_in_the_summary() -> None: + """A held load-only node still reports the inputs it skipped.""" + db = InMemoryArrowDatabase() + _grouped_job(ROWS[:3], db).run() + job = _grouped_job(ROWS, db) + job.run(load_only=True) + summary = job.run(load_only=True) + rowwise = summary.nodes["rowwise"] + assert (rowwise.held, rowwise.not_stored) == (3, 3) + assert "held from the previous run (3 inputs without a stored result)" in repr( + summary + ) + + +class _Stop(BaseException): + """An interruption that is not an ``Exception``, as a cancellation is.""" + + +class StopsAtNodeEnd(NoOpObserver): + """An observer that interrupts the run at the end of one node.""" + + def __init__(self, label: str) -> None: + """Keep the label of the node this observer acts at.""" + self.label = label + + def on_node_end(self, node_label: str, node_hash: str) -> None: + """Interrupt at the node's end.""" + if node_label == self.label: + raise _Stop() + + def contextualize(self, *identity_path: str) -> StopsAtNodeEnd: + """Keep this observer for every node.""" + return self + + +def test_an_interrupted_run_clears_the_nodes_it_ran() -> None: + """A run interrupted by a non-Exception clears the nodes it ran.""" + job = _grouped_job(ROWS, InMemoryArrowDatabase()) + with pytest.raises(_Stop): + job.run(observer=StopsAtNodeEnd("summed")) + assert job.nodes["rowwise"].as_table().num_rows == 0 + assert job.nodes["rowwise"].is_stale + + +def test_a_loaded_operator_that_lost_an_input_names_it(tmp_path: Path) -> None: + """A loaded operator whose saved input was not restored names ITL-1094.""" + tap = side_effect_pod(_deliver_x) + assert isinstance(tap, SideEffectPod) + db = InMemoryArrowDatabase() + job = PipelineJob(name="m2", database=db) + with job: + tapped: Any = tap.process(_source(), label="delivered") + Join()( + tapped, + DictSource( + [{"g": r["g"], "k": r["k"], "w": r["x"]} for r in ROWS], + tag_columns=["g", "k"], + source_id="w", + ), + label="joined", + ) + job.run() + job.save(tmp_path / "job.json") + loaded = PipelineJob.load(tmp_path / "job.json", database=db) + loaded.bind( + sources={ + "s": _source(), + "w": DictSource( + [{"g": r["g"], "k": r["k"], "w": r["x"]} for r in ROWS], + tag_columns=["g", "k"], + source_id="w", + ), + } + ) + with pytest.raises(RuntimeError, match=r"'joined'.*ITL-1094"): + loaded.run(load_only=True) + + +def test_an_async_cache_only_node_holds_the_rows_it_sent() -> None: + """An async CACHE_ONLY node holds the stored rows it sent.""" + from orcapod.channels import Channel + from orcapod.core.nodes import FunctionJobNode + from orcapod.pipeline.serialization import LoadStatus + + table = pa.table( + {"k": [0, 1], "x": [1, 2]}, + schema=pa.schema( + [ + pa.field("k", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + db = InMemoryArrowDatabase() + + def node() -> Any: + """A function node over the two-row table, on the shared store.""" + return FunctionJobNode( + inc.pod, # pyright: ignore[reportArgumentType] # the decorator's pod type is not declared as FunctionPodProtocol + ArrowTableStream(table, tag_columns=["k"]), + pipeline_database=db, + ) + + node().run() + cached = node() + # A node whose input is unavailable (as from_descriptor makes it) is the case here. + cached._load_status = LoadStatus.CACHE_ONLY # pyright: ignore[reportPrivateUsage] # the load status is what this test is about + + async def run() -> list[Any]: + """Run the node's channel entry point.""" + inputs: Channel[Any] = Channel(buffer_size=4) + output: Channel[Any] = Channel(buffer_size=4) + await inputs.writer.close() + # The channel entry point is what this test is about. + await cached.async_execute(inputs.reader, output.writer) + return await output.reader.collect() + + sent = asyncio.run(run()) + assert len(sent) == 2 + assert cached.as_table().num_rows == 2 + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_every_node_a_load_only_run_fills_says_so(orchestrator: str) -> None: + """Function, operator and side-effect nodes filled by a load-only run show + ``loaded_only``; a source node is read as in any run and shows its rows.""" + db = InMemoryArrowDatabase() + _tap_job(db).run() + job = _tap_job(db) + job.run(orchestrator=ORCHESTRATORS[orchestrator](), load_only=True) + for label in ("rows", "tap", "down"): + assert job.nodes[label].head().status is ViewStatus.LOADED_ONLY, label + assert job.nodes["tap"].run(load_only=True).status is ViewStatus.LOADED_ONLY + assert job.nodes["s"].head().status is ViewStatus.ROWS + + +# --------------------------------------------------------------------------- +# Third review round +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) +def test_a_direct_async_run_drains_the_inputs_of_a_held_node( + error_policy: Literal["continue", "fail_fast"], +) -> None: + """A held node's incoming channel is read even when the caller passes the whole + graph: a source with more rows than the channel buffer does not block the run.""" + from orcapod.pipeline.dag import OrcaDAG + + rows = [{"g": i % 2, "k": i, "x": i} for i in range(64)] + job = _grouped_job(rows, InMemoryArrowDatabase()) + job.run() + rowwise, grouped, summed = (job.nodes[n] for n in ("rowwise", "grouped", "summed")) + source = rowwise.upstreams[0] + dag: OrcaDAG[Any] = OrcaDAG() + for upstream, downstream in ( + (source, rowwise), + (rowwise, grouped), + (grouped, summed), + ): + dag.add_edge(upstream, downstream) + held = {rowwise: list(rowwise.iter_data())} + summed.clear_cache() + grouped.clear_cache() + orchestrator = AsyncPipelineOrchestrator(buffer_size=2, error_policy=error_policy) + + async def run() -> None: + """Run the whole graph with ``rowwise`` held, under a time limit.""" + await asyncio.wait_for(orchestrator.run_async(dag, held_rows=held), timeout=30) + + asyncio.run(run()) + assert summed.as_table().num_rows == 2 + + +def test_a_loaded_job_runs_its_bound_branch_beside_a_broken_unbound_one( + tmp_path: Path, +) -> None: + """The ITL-1094 check covers only the nodes the run includes: a branch below an + unbound source and a dropped node is left out, the bound branch loads.""" + tap = side_effect_pod(_deliver_x) + assert isinstance(tap, SideEffectPod) + db = InMemoryArrowDatabase() + job = PipelineJob(name="m3", database=db) + with job: + tapped: Any = tap.process(_source(), label="delivered") + inc.pod(tapped, label="broken") + dbl.pod( + inc.pod( + DictSource( + [{"g": 0, "k": 9, "x": 5}], + tag_columns=["g", "k"], + source_id="other", + ), + label="up2", + ), + label="bound_branch", + ) + job.run() + job.save(tmp_path / "job.json") + loaded = PipelineJob.load(tmp_path / "job.json", database=db) + loaded.bind( + sources={ + "other": DictSource( + [{"g": 0, "k": 9, "x": 5}], tag_columns=["g", "k"], source_id="other" + ) + } + ) + loaded.run(load_only=True) + assert loaded.nodes["bound_branch"].as_table().num_rows == 1 + assert loaded.nodes["broken"].as_table().num_rows == 0 + + +def test_a_cache_only_node_keeps_its_rows_over_an_input_with_no_modified_time() -> None: + """``as_table`` judges a ``CACHE_ONLY`` node as ``head`` and ``iter_data`` do: its + unavailable input does not make the rows it holds stale.""" + from orcapod.channels import Channel + from orcapod.core.nodes import FunctionJobNode + from orcapod.pipeline.serialization import LoadStatus + + table = pa.table( + {"k": [0, 1], "x": [1, 2]}, + schema=pa.schema( + [ + pa.field("k", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + db = InMemoryArrowDatabase() + stream = ArrowTableStream(table, tag_columns=["k"]) + FunctionJobNode( + inc.pod, # pyright: ignore[reportArgumentType] # the decorator's pod type is not declared as FunctionPodProtocol + stream, + pipeline_database=db, + ).run() + cached = FunctionJobNode( + inc.pod, # pyright: ignore[reportArgumentType] # the decorator's pod type is not declared as FunctionPodProtocol + stream, + pipeline_database=db, + ) + # An unavailable input (as from_descriptor makes it) is the case here. + cached._load_status = LoadStatus.CACHE_ONLY # pyright: ignore[reportPrivateUsage] # the load status is what this test is about + + async def run() -> None: + """Feed nothing: a CACHE_ONLY node serves what is stored.""" + inputs: Channel[Any] = Channel(buffer_size=4) + output: Channel[Any] = Channel(buffer_size=4) + await inputs.writer.close() + await cached.async_execute(inputs.reader, output.writer) + _ = await output.reader.collect() + + asyncio.run(run()) + stream._set_modified_time(None) # pyright: ignore[reportPrivateUsage] # an input with no modified time is what this test is about + assert cached.as_table().num_rows == 2 + assert len(list(cached.iter_data())) == 2 + + +def test_a_commit_interrupted_by_a_non_exception_clears_the_nodes_it_ran() -> None: + """An interruption during the end-of-run commit (the last flush of the run) leaves + its outcome unknown: the nodes the run ran hold nothing.""" + original = InMemoryArrowDatabase.flush + calls = 0 + + def counting(self: Any) -> None: + """Count the flushes of a successful run.""" + nonlocal calls + calls += 1 + original(self) + + with mock.patch.object(InMemoryArrowDatabase, "flush", counting): + _grouped_job(ROWS, InMemoryArrowDatabase()).nodes["summed"].run() + total, calls = calls, 0 + + def interrupt_last(self: Any) -> None: + """Interrupt the final flush, as a ``KeyboardInterrupt`` would.""" + nonlocal calls + calls += 1 + if calls == total: + raise _Stop() + original(self) + + job = _grouped_job(ROWS, InMemoryArrowDatabase()) + with ( + mock.patch.object(InMemoryArrowDatabase, "flush", interrupt_last), + pytest.raises(_Stop), + ): + job.nodes["summed"].run() + for label in ("rowwise", "grouped", "summed"): + assert job.nodes[label].as_table().num_rows == 0, label + + +def test_a_job_run_whose_final_commit_is_interrupted_clears_the_nodes_it_ran() -> None: + """An interruption during ``job.run()``'s end-of-run commit (its last flush) leaves + the outcome unknown: the nodes the run ran hold nothing, as for ``node.run()``.""" + original = InMemoryArrowDatabase.flush + calls = 0 + + def counting(self: Any) -> None: + """Count the flushes of a successful run.""" + nonlocal calls + calls += 1 + original(self) + + with mock.patch.object(InMemoryArrowDatabase, "flush", counting): + _grouped_job(ROWS, InMemoryArrowDatabase()).run() + total, calls = calls, 0 + + def interrupt_last(self: Any) -> None: + """Interrupt the final flush, as a ``KeyboardInterrupt`` would.""" + nonlocal calls + calls += 1 + if calls == total: + raise _Stop() + original(self) + + job = _grouped_job(ROWS, InMemoryArrowDatabase()) + with ( + mock.patch.object(InMemoryArrowDatabase, "flush", interrupt_last), + pytest.raises(_Stop), + ): + job.run() + for label in ("rowwise", "grouped", "summed"): + assert job.nodes[label].as_table().num_rows == 0, label + assert job.nodes[label].is_stale, label diff --git a/tests/test_pipeline/test_current_run_view_zero_row_types.py b/tests/test_pipeline/test_current_run_view_zero_row_types.py new file mode 100644 index 000000000..90ef12dbf --- /dev/null +++ b/tests/test_pipeline/test_current_run_view_zero_row_types.py @@ -0,0 +1,209 @@ +"""A job node's current-run view has one Arrow schema in every state (ITL-1095, ITL-912). + +ITL-1095: a job node shows its current run, and reads zero rows before a run, after +``clear_cache()``, after a run that raised and while stale. ITL-912: a table with no rows +has exactly the Arrow schema of the node's rows (types, nullability, system tags, source +info, column order), derived from the input schemas. Together: ``as_table(all_info=True)`` +of a function node and of operator nodes (``group_by``, ``join``) has the same schema in +each state, over inputs whose types are not the defaults (an ``int8`` non-null tag, an +``int32`` data column). +""" + +from __future__ import annotations + +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.sources import ArrowTableSource +from orcapod.system_constants import constants + +ORCHESTRATORS = {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator} +NODES = ("inc", "grouped", "joined") +FAIL = {"on": False} + + +@function_pod(output_keys="y") +def inc(x: int) -> int: + """Return ``x + 1``; raise while ``FAIL["on"]`` is set.""" + if FAIL["on"]: + raise RuntimeError("inc failed on purpose") + return x + 1 + + +@pytest.fixture(autouse=True) +def _reset_failure() -> None: + """Let ``inc`` succeed at the start of each test.""" + FAIL["on"] = False + + +def _source(data: str, source_id: str) -> ArrowTableSource: + """Tags ``g``, ``id`` (``int8`` non-null) and one ``int32`` non-null data column.""" + schema = pa.schema( + [ + pa.field("g", pa.int8(), nullable=False), + pa.field("id", pa.int8(), nullable=False), + pa.field(data, pa.int32(), nullable=False), + ] + ) + table = pa.table( + {"g": [0, 1, 0, 1], "id": [0, 1, 2, 3], data: [10, 20, 30, 40]}, schema=schema + ) + return ArrowTableSource(table, tag_columns=["g", "id"], source_id=source_id) + + +def _job(db: InMemoryArrowDatabase | None = None) -> PipelineJob: + """``inc`` (function) -> ``grouped`` (GroupBy on ``g``); ``inc`` joined with ``b``.""" + job = PipelineJob(name="j", database=db or InMemoryArrowDatabase()) + with job: + rowwise: Any = inc.pod(_source("x", "a"), label="inc") + _ = rowwise.group_by("g", label="grouped") + _ = rowwise.join(_source("z", "b"), label="joined") + return job + + +def _assert_reads_with( + job: PipelineJob, expected: dict[str, pa.Schema], *, rows: bool +) -> None: + """Each node reads rows (or none) and has exactly the ``expected`` schema.""" + for label in NODES: + table = job.nodes[label].as_table(all_info=True) + assert (table.num_rows > 0) is rows, label + assert table.schema.equals(expected[label]), ( + label, + table.schema, + expected[label], + ) + + +def _assert_empty_with(job: PipelineJob, expected: dict[str, pa.Schema]) -> None: + """Each node reads zero rows and has exactly the ``expected`` schema.""" + _assert_reads_with(job, expected, rows=False) + + +def _run_schemas(orchestrator: str) -> dict[str, pa.Schema]: + """The schema of each node's populated table after a run of a fresh job.""" + job = _job() + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + schemas: dict[str, pa.Schema] = {} + for label in NODES: + table = job.nodes[label].as_table(all_info=True) + assert table.num_rows > 0, label + schemas[label] = table.schema + return schemas + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +class TestOneSchemaInEveryState: + """``as_table(all_info=True)`` of every node keeps its run's schema in each state.""" + + def test_the_populated_tables_keep_the_narrow_input_types( + self, orchestrator: str + ) -> None: + """The reference schemas carry the input's types and the system tags.""" + schemas = _run_schemas(orchestrator) + for label in ("inc", "joined"): + assert schemas[label].field("id").type == pa.int8(), label + assert not schemas[label].field("id").nullable, label + assert schemas["grouped"].field("g").type == pa.int8() + assert schemas["joined"].field("z").type == pa.int32() + for label in NODES: + assert any( + n.startswith(constants.SYSTEM_TAG_PREFIX) for n in schemas[label].names + ), label + + def test_before_a_run(self, orchestrator: str) -> None: + """A fresh job: every node reads zero rows with its run's schema.""" + _assert_empty_with(_job(), _run_schemas(orchestrator)) + + def test_after_a_run_and_after_clear_cache(self, orchestrator: str) -> None: + """A run fills each node; ``clear_cache()`` empties it with the same schema.""" + expected = _run_schemas(orchestrator) + job = _job() + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + _assert_reads_with(job, expected, rows=True) + for label in NODES: + job.nodes[label].clear_cache() + _assert_empty_with(job, expected) + + def test_after_a_failed_run(self, orchestrator: str) -> None: + """A run that raised leaves every node empty, with its run's schema.""" + expected = _run_schemas(orchestrator) + job = _job() + FAIL["on"] = True + with pytest.raises(RuntimeError, match="inc failed on purpose"): + _ = job.run( + orchestrator=ORCHESTRATORS[orchestrator](error_policy="fail_fast") + ) + _assert_empty_with(job, expected) + + def test_while_stale(self, orchestrator: str) -> None: + """A rebound source makes every node stale and empty, with the same schema.""" + expected = _run_schemas(orchestrator) + job = _job() + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + # A new source object of the same schema makes every node below it stale. + slot = next(name for name, src in job.sources.items() if "x" in src.keys()[1]) + job.bind(sources={slot: _source("x", "a")}) + for label in NODES: + assert job.nodes[label].is_stale, label + _assert_empty_with(job, expected) + + def test_after_a_load_only_run(self, orchestrator: str) -> None: + """A load-only run over a store filled by a run shows that run's schema.""" + expected = _run_schemas(orchestrator) + db = InMemoryArrowDatabase() + _ = _job(db).run(orchestrator=ORCHESTRATORS[orchestrator]()) + job = _job(db) + FAIL["on"] = True # a load-only run never calls the function + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator](), load_only=True) + _assert_reads_with(job, expected, rows=True) + + def test_after_a_load_only_run_with_nothing_stored(self, orchestrator: str) -> None: + """A load-only run over an empty store fills every node with zero rows. + + The join's zero-row result has its populated table's fields in another order + (ITL-1131), so the fields are compared by name here; the exact + schema is ``test_a_join_with_no_input_rows_keeps_its_column_order``. + """ + expected = _run_schemas(orchestrator) + job = _load_only_over_an_empty_store(orchestrator) + for label in NODES: + table = job.nodes[label].as_table(all_info=True) + assert table.num_rows == 0, label + assert _fields_by_name(table.schema) == _fields_by_name(expected[label]) + if label != "joined": + assert table.schema.equals(expected[label]), label + + @pytest.mark.xfail( + strict=True, + raises=AssertionError, + reason=( + "ITL-1131: a join's zero-row result lists its data columns before " + "its system tags; its populated table lists them after" + ), + ) + def test_a_join_with_no_input_rows_keeps_its_column_order( + self, orchestrator: str + ) -> None: + """The join that received no rows has its populated schema, in order.""" + expected = _run_schemas(orchestrator)["joined"] + job = _load_only_over_an_empty_store(orchestrator) + assert job.nodes["joined"].as_table(all_info=True).schema.equals(expected) + + +def _load_only_over_an_empty_store(orchestrator: str) -> PipelineJob: + """A job after a load-only run over an empty store: every node holds zero rows.""" + job = _job() + FAIL["on"] = True # a load-only run never calls the function + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator](), load_only=True) + return job + + +def _fields_by_name(schema: pa.Schema) -> dict[str, "pa.Field[Any]"]: + """The fields of ``schema`` by name, so that their order is not compared.""" + return {field.name: field for field in schema} diff --git a/tests/test_pipeline/test_hash_step_combined.py b/tests/test_pipeline/test_hash_step_combined.py index 7459a4bad..c8880a3b6 100644 --- a/tests/test_pipeline/test_hash_step_combined.py +++ b/tests/test_pipeline/test_hash_step_combined.py @@ -226,7 +226,10 @@ def test_a_table_written_before_v03_that_appears_between_runs_is_refused(self): # A log of the same pod at the layout Orcapod v0.2 wrote, under another hash. old_path = _add_table_written_before_v03(store, "tdb_v2", "tdb_v1") - # The same job object runs again: the check is not remembered from the first run. + # The same job object runs the sink again (a node holding a current run is not + # run, ITL-1095, so it is cleared first): the check is not remembered from the + # first run. + job.nodes["sink"].clear_cache() with pytest.raises(SideEffectLogDataContextError) as info: _ = job.run() assert old_path in info.value.record_paths @@ -239,11 +242,13 @@ def test_a_function_pod_with_ctx_arg_checks_before_every_run_too(self): _ = job.run() assert sorted(delivered) == [1, 2] delivered.clear() + job.nodes["notify"].clear_cache() # a current node is not run again (ITL-1095) _ = job.run() # nothing new in the store: runs, and delivers nothing again assert delivered == [] old_path = _add_table_written_before_v03(store, "pdb_v2", "pdb_v1") + job.nodes["notify"].clear_cache() with pytest.raises(SideEffectLogDataContextError) as info: _ = job.run() assert old_path in info.value.record_paths diff --git a/tests/test_pipeline/test_job_node_current_run_view.py b/tests/test_pipeline/test_job_node_current_run_view.py new file mode 100644 index 000000000..e9ad8c4b1 --- /dev/null +++ b/tests/test_pipeline/test_job_node_current_run_view.py @@ -0,0 +1,208 @@ +"""A job node shows the current run; ``node.as_source()`` shows what is stored (ITL-1095). + +Developer, November 13, 2026: a job node itself (``as_table``, ``iter_data``, the dataframe +methods, ``flow``, ``head``, the preview, ``run``) is the current run view; ``as_source()`` +is the view of everything stored for the node. Before a run in this process a node is empty +and stale, whatever its store holds; a run fills it. + +One store holds the results of two earlier runs (6 source rows, then 3). A fresh job over +that store, and a job loaded from a saved file, read empty from every node until they run. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.streams.view import ViewStatus +from orcapod.databases import InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.sources import DictSource +from orcapod.types import CacheMode + +CALLS: list[str] = [] + + +@function_pod(output_keys="y") +def inc(x: int) -> int: + """Return ``x + 1``.""" + CALLS.append("inc") + return x + 1 + + +@function_pod(output_keys="total") +def total(y: list[int]) -> int: + """Return the sum of a group's values.""" + CALLS.append("total") + return sum(y) + + +ROWS = [{"g": i % 2, "k": i, "x": 10 * i} for i in range(6)] +ORCHESTRATORS = {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator} +JOB_NODES = ("rowwise", "grouped", "summed") + + +@pytest.fixture(autouse=True) +def _reset_calls() -> None: + CALLS.clear() + + +def _build( + data: list[dict[str, int]], + db: Any, + cache_mode: CacheMode | None = None, +) -> PipelineJob: + """``rowwise`` (function) -> ``grouped`` (GroupBy operator) -> ``summed`` (function).""" + src = DictSource(data, tag_columns=["g", "k"], source_id="s") + job = PipelineJob(name="j", database=db) + with job: + rowwise: Any = inc.pod(src, label="rowwise") + grouped = rowwise.group_by("g", label="grouped") + total.pod(grouped, label="summed") + if cache_mode is not None: + job.nodes["grouped"].attach_databases( + pipeline_database=db.at("j"), cache_mode=cache_mode + ) + return job + + +def _populated_store( + orchestrator: str = "sync", cache_mode: CacheMode | None = None +) -> InMemoryArrowDatabase: + """A store holding the results of a run over 6 rows and a run over 3 rows.""" + db = InMemoryArrowDatabase() + for data in (ROWS, ROWS[:3]): + _build(data, db, cache_mode).run(orchestrator=ORCHESTRATORS[orchestrator]()) + CALLS.clear() + return db + + +def _assert_empty(node: Any, columns: list[str]) -> None: + """Every read of ``node`` returns zero rows with the node's columns.""" + table = node.as_table() + assert table.num_rows == 0 + assert table.column_names == columns + assert list(node.iter_data()) == [] + assert node.as_polars_df().height == 0 + assert node.head().table.num_rows == 0 + assert node.head().table.column_names == columns + + +def _assert_points_to_run_and_stored_view(node: Any) -> None: + """The head of a node that has not run says so and names both ways in.""" + head = node.head() + assert head.status is ViewStatus.NOT_COMPUTED + header = head.header() + assert "not run" in header + assert "run()" in header + assert "as_source()" in header + assert "not run" in node._repr_html_() + + +class TestFreshJobOverAPopulatedStore: + """A job built over a store that holds results reads empty until it runs.""" + + @pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) + def test_every_read_of_every_node_is_empty_before_a_run( + self, orchestrator: str + ) -> None: + db = _populated_store(orchestrator) + job = _build(ROWS[:3], db) + expected_columns = { + label: _build(ROWS[:3], db).nodes[label].as_table().column_names + for label in JOB_NODES + } + for label in JOB_NODES: + node = job.nodes[label] + _assert_empty(node, expected_columns[label]) + _assert_points_to_run_and_stored_view(node) + assert CALLS == [] + + def test_the_stored_view_shows_every_stored_record(self) -> None: + db = _populated_store() + job = _build(ROWS[:3], db) + # Both earlier runs' rows: 6 rowwise results, 2 + 2 group totals. + assert job.nodes["rowwise"].as_source().as_table().num_rows == 6 + assert job.nodes["summed"].as_source().as_table().num_rows == 4 + assert job.nodes["rowwise"].as_table().num_rows == 0 + + def test_every_node_of_a_fresh_job_is_stale(self) -> None: + job = _build(ROWS[:3], _populated_store()) + for label in JOB_NODES: + assert job.nodes[label].is_stale, label + + @pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) + def test_a_run_fills_the_nodes_with_this_runs_rows(self, orchestrator: str) -> None: + db = _populated_store(orchestrator) + job = _build(ROWS[:3], db) + job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert CALLS == [] # every result was stored + assert job.nodes["rowwise"].as_table().num_rows == 3 + assert sorted( + (r["g"], r["total"]) for r in job.nodes["summed"].as_table().to_pylist() + ) == [(0, 22), (1, 11)] + for label in JOB_NODES: + assert not job.nodes[label].is_stale, label + assert job.nodes[label].head().status is ViewStatus.ROWS, label + + +@pytest.mark.parametrize("mode", [CacheMode.OFF, CacheMode.LOG, CacheMode.REPLAY]) +def test_an_operator_node_reads_empty_before_a_run_in_every_cache_mode( + mode: CacheMode, +) -> None: + """REPLAY included: before a run it no longer shows every logged run.""" + # LOG runs fill the operator cache, so REPLAY has stored rows to (not) show. + db = _populated_store(cache_mode=CacheMode.LOG) + job = _build(ROWS[:3], db, cache_mode=mode) + node = job.nodes["grouped"] + _assert_empty(node, ["g", "y"]) + _assert_points_to_run_and_stored_view(node) + assert node.is_stale + + +class TestAfterARun: + """What empties a node that held a run: clearing it, or a change upstream.""" + + @pytest.mark.parametrize("label", list(JOB_NODES)) + def test_clear_cache_empties_the_node_and_makes_it_stale(self, label: str) -> None: + job = _build(ROWS[:3], InMemoryArrowDatabase()) + job.run() + node = job.nodes[label] + columns = node.as_table().column_names + node.clear_cache() + assert node.is_stale + _assert_empty(node, columns) + + def test_a_rebound_source_empties_its_dependent_nodes(self) -> None: + job = _build(ROWS[:3], InMemoryArrowDatabase()) + job.run() + source_name = next(iter(job.sources)) # the one source slot + job.bind( + sources={ + source_name: DictSource(ROWS, tag_columns=["g", "k"], source_id="s") + } + ) + for label in JOB_NODES: + node = job.nodes[label] + assert node.is_stale, label + assert node.as_table().num_rows == 0, label + assert node.head().status is ViewStatus.NOT_COMPUTED, label + + +def test_a_loaded_job_reads_empty_until_it_runs(tmp_path: Path) -> None: + """A loaded job's nodes are empty and do not raise; ``as_source()`` reads the store.""" + db = InMemoryArrowDatabase() + job = _build(ROWS, db) + job.run() + job.save(tmp_path / "job.json") + loaded = PipelineJob.load(tmp_path / "job.json", database=db) + for label in JOB_NODES: + node = loaded.nodes[label] + assert node.as_table().num_rows == 0, label + assert list(node.iter_data()) == [], label + assert node.head().status is ViewStatus.NOT_COMPUTED, label + assert "not run" in node.head().header(), label + assert loaded.nodes["rowwise"].as_source().as_table().num_rows == 6 diff --git a/tests/test_pipeline/test_job_node_rows_after_run.py b/tests/test_pipeline/test_job_node_rows_after_run.py index defb0b1f5..76b9b1341 100644 --- a/tests/test_pipeline/test_job_node_rows_after_run.py +++ b/tests/test_pipeline/test_job_node_rows_after_run.py @@ -5,8 +5,8 @@ the rows of that run's inputs through ``as_table``, ``iter_data``, ``head`` and ``run()``, and so does a consumer built over the node afterwards; the two orchestrators agree row for row. Stored results of other inputs stay stored and reusable. A run that raised defines no -rows: the node then reads as a node with no run in this process (every row stored under -its content hash). +rows: the node then reads as a node with no run in this process, which is empty (ITL-1095); +what was stored stays readable through ``as_source()``. """ from __future__ import annotations @@ -21,6 +21,7 @@ from orcapod.databases import InMemoryArrowDatabase from orcapod.errors import StoreWriteError from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.core.streams import ViewStatus from orcapod.sources import DictSource @@ -293,14 +294,7 @@ def add_one(x: int) -> int: class TestRaisedRun: - """A run that raised defines no rows: the node reads as a node with no run.""" - - @staticmethod - def _no_run_read(fn: Any, data: list[dict[str, int]], db: Any) -> list[Any]: - """Return the ``(k, y)`` rows a compiled, unrun job over ``db`` reads.""" - fresh = _single(fn.pod, data, db) - fresh.compile() - return _values(fresh.nodes["node"], lambda r: (r["k"], r["y"])) + """A run that raised defines no rows: the node reads as a node with no run (empty).""" @pytest.mark.parametrize( ("kind", "fn", "policy"), @@ -320,12 +314,13 @@ def test_raised_run_reads_as_no_run(self, kind: str, fn: Any, policy: str) -> No with pytest.raises((ValueError, StoreWriteError)): job.run(orchestrator=_orchestrator(kind, policy)) node = job.nodes["node"] - expected = self._no_run_read(fn, ROWS[:4], db) - assert _values(node, lambda r: (r["k"], r["y"])) == expected + assert _values(node, lambda r: (r["k"], r["y"])) == [] reads = _reads(node) - assert reads["head"] == reads["as_table"] - # Nothing stored was dropped: the stored rows of the earlier run are among them. - assert {(3, 31), (4, 41), (5, 51)} <= set(expected) + assert reads["head"] == reads["as_table"] == [] + assert node.is_stale + # Nothing stored was dropped: the stored view holds the earlier run's rows. + stored = {(r["k"], r["y"]) for r in node.as_source().as_table().to_pylist()} + assert {(3, 31), (4, 41), (5, 51)} <= stored class TestDuplicateInputs: @@ -471,10 +466,14 @@ def on_node_end(self, node_label: str, node_hash: str) -> None: job.run(orchestrator=_orchestrator(kind)) node = job.nodes["node"] assert _values(node, lambda r: r["k"]) == [0, 1, 2] + # A node holding a current run is not run again (ITL-1095): clear it first. + node.clear_cache() with pytest.raises(RuntimeError, match="^observer failed$"): job.run(orchestrator=_orchestrator(kind), observer=FailsAtNodeEnd()) - # Every stored row (the 6-row run's), read from the store. - assert _values(node, lambda r: r["k"]) == [0, 1, 2, 3, 4, 5] + # The node holds no run (it reads empty, ITL-1095); the store keeps every row. + assert _values(node, lambda r: r["k"]) == [] + stored = node.as_source().as_table().to_pylist() + assert sorted(r["k"] for r in stored) == [0, 1, 2, 3, 4, 5] class TestAsyncOutputCloseFails: @@ -535,6 +534,8 @@ async def run() -> None: with pytest.raises(OSError, match="cannot be closed"): asyncio.run(run()) assert len(output.sent) == 2 - # No run is held: the rows are read from the store. - assert node.head(10).origin == "stored" - assert _values(node, lambda r: (r["k"], r["y"])) == [(0, 6), (1, 7)] + # No run is held: the node reads empty (ITL-1095); the store keeps the rows. + assert node.head(10).status is ViewStatus.NOT_COMPUTED + assert _values(node, lambda r: (r["k"], r["y"])) == [] + stored = node.as_source().as_table().to_pylist() + assert sorted((r["k"], r["y"]) for r in stored) == [(0, 6), (1, 7)] diff --git a/tests/test_pipeline/test_job_run_state_rule.py b/tests/test_pipeline/test_job_run_state_rule.py new file mode 100644 index 000000000..a59a6ae5d --- /dev/null +++ b/tests/test_pipeline/test_job_run_state_rule.py @@ -0,0 +1,240 @@ +"""``job.run()`` follows the state rule of ``node.run()`` (ITL-1095). + +Developer, November 14, 2026 ("let's align now"): ``job.run()`` and ``run_async()`` apply +the same rule as ``node.run()``. A node that holds a current completed run (not stale; for +a full run, not filled by a load-only run) is not run again: its rows feed the nodes after +it. Never-run, stale and (for a full run) load-only-filled nodes run. So a second +``job.run()`` of an unchanged job computes nothing, reads and writes nothing in the store +and delivers no side effect. +""" + +from __future__ import annotations + +import asyncio +from typing import Any, Literal +from unittest import mock + +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.errors import StoreWriteError +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.side_effects import SideEffectPod, tap_pod +from orcapod.sources import DictSource + +CALLS: list[str] = [] +DELIVERED: list[int] = [] + + +@function_pod(output_keys="y") +def inc(x: int) -> int: + """Return ``x + 1``.""" + CALLS.append("inc") + return x + 1 + + +@function_pod(output_keys="total") +def total(y: list[int]) -> int: + """Return the sum of a group's values.""" + CALLS.append("total") + return sum(y) + + +def _deliver(y: int, ctx: Any) -> None: + """A side effect: record the delivered value.""" + DELIVERED.append(y) + + +ROWS = [{"g": i % 2, "k": i, "x": 10 * i} for i in range(6)] +ORCHESTRATORS = {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator} + + +@pytest.fixture(autouse=True) +def _reset() -> None: + """Start every test with no recorded calls or deliveries.""" + CALLS.clear() + DELIVERED.clear() + + +def _source(data: list[dict[str, int]]) -> DictSource: + """An in-memory source of ``data`` with tags ``g`` and ``k``.""" + return DictSource(data, tag_columns=["g", "k"], source_id="s") + + +def _build(data: list[dict[str, int]], db: Any) -> PipelineJob: + """``rowwise`` -> ``tap`` (side effect, no completion tracking) -> ``grouped`` -> + ``summed``.""" + job = PipelineJob(name="j", database=db) + with job: + rows = inc.pod(_source(data), label="rowwise") + tap = tap_pod(_deliver) + assert isinstance(tap, SideEffectPod) + tapped: Any = tap.process(rows, label="tap") + grouped = tapped.group_by("g", label="grouped") + total.pod(grouped, label="summed") + return job + + +def _run(job: PipelineJob, orchestrator: str, **kwargs: Any) -> Any: + """Run ``job`` on the named orchestrator.""" + return job.run(orchestrator=ORCHESTRATORS[orchestrator](), **kwargs) + + +def _totals(job: PipelineJob) -> list[tuple[int, int]]: + """The ``(g, total)`` rows of ``summed``, sorted.""" + return sorted( + (r["g"], r["total"]) for r in job.nodes["summed"].as_table().to_pylist() + ) + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_second_run_of_an_unchanged_job_does_nothing(orchestrator: str) -> None: + """No computation, store access, source read or delivery; every node held.""" + job = _build(ROWS, InMemoryArrowDatabase()) + _run(job, orchestrator) + assert len(DELIVERED) == 6 + CALLS.clear() + DELIVERED.clear() + with ( + mock.patch.object( + InMemoryArrowDatabase, + "get_all_records", + side_effect=AssertionError("store read"), + ), + mock.patch.object( + InMemoryArrowDatabase, + "add_records", + side_effect=AssertionError("store write"), + ), + mock.patch.object( + DictSource, "iter_data", side_effect=AssertionError("source read") + ), + ): + summary = _run(job, orchestrator) + assert CALLS == [] + assert DELIVERED == [] + assert _totals(job) == [(0, 63), (1, 93)] + rowwise = summary.nodes["rowwise"] + assert (rowwise.computed, rowwise.reused, rowwise.held) == (0, 0, 6) + assert summary.nodes["summed"].held == 2 + assert "rowwise: 6 rows: held from the previous run" in repr(summary) + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_cleared_node_runs_again_with_the_nodes_below_it(orchestrator: str) -> None: + """Clearing ``summed`` runs it alone, from the rows ``grouped`` holds.""" + job = _build(ROWS, InMemoryArrowDatabase()) + _run(job, orchestrator) + CALLS.clear() + DELIVERED.clear() + job.nodes["summed"].clear_cache() + summary = _run(job, orchestrator) + assert CALLS == [] # its results are stored: read, not computed + assert DELIVERED == [] + assert summary.nodes["summed"].reused == 2 + assert summary.nodes["rowwise"].held == 6 + assert _totals(job) == [(0, 63), (1, 93)] + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_clearing_a_side_effect_node_delivers_it_again(orchestrator: str) -> None: + """``clear_cache()`` on a side-effect node makes the next run deliver again.""" + job = _build(ROWS, InMemoryArrowDatabase()) + _run(job, orchestrator) + DELIVERED.clear() + job.nodes["tap"].clear_cache() + _run(job, orchestrator) + assert len(DELIVERED) == 6 + assert CALLS.count("inc") == 6 # only the first run computed + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_rebound_source_runs_every_node_below_it(orchestrator: str) -> None: + """Binding a new source object makes every node below it run; new inputs compute.""" + job = _build(ROWS[:3], InMemoryArrowDatabase()) + _run(job, orchestrator) + CALLS.clear() + job.bind(sources={next(iter(job.sources)): _source(ROWS)}) + _run(job, orchestrator) + assert CALLS.count("inc") == 3 + assert _totals(job) == [(0, 63), (1, 93)] + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_full_run_after_a_load_only_run_runs_the_load_only_nodes( + orchestrator: str, +) -> None: + """Load-only rows are never current for a full run: it computes what was skipped.""" + db = InMemoryArrowDatabase() + _run(_build(ROWS[:3], db), orchestrator) + CALLS.clear() + DELIVERED.clear() + job = _build(ROWS, db) + _run(job, orchestrator, load_only=True) + assert CALLS == [] and DELIVERED == [] + _run(job, orchestrator) + assert CALLS.count("inc") == 3 + assert len(DELIVERED) == 6 + assert _totals(job) == [(0, 63), (1, 93)] + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_load_only_run_after_a_full_run_does_nothing(orchestrator: str) -> None: + """A full run's rows are current for a load-only run too.""" + job = _build(ROWS, InMemoryArrowDatabase()) + _run(job, orchestrator) + CALLS.clear() + summary = _run(job, orchestrator, load_only=True) + assert CALLS == [] + assert summary.nodes["rowwise"].held == 6 + + +def test_run_async_follows_the_rule() -> None: + """``run_async`` holds current nodes as ``run`` does.""" + job = _build(ROWS, InMemoryArrowDatabase()) + asyncio.run(job.run_async()) + CALLS.clear() + DELIVERED.clear() + summary = asyncio.run(job.run_async()) + assert CALLS == [] and DELIVERED == [] + assert summary.nodes["summed"].held == 2 + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_failed_commit_clears_only_the_nodes_that_ran(orchestrator: str) -> None: + """A failed end-of-run commit clears the nodes that ran; held nodes keep rows.""" + job = _build(ROWS, InMemoryArrowDatabase()) + _run(job, orchestrator) + job.nodes["summed"].clear_cache() + + def fail(self: Any) -> None: + """A flush that always fails.""" + raise OSError("disk full") + + with ( + mock.patch.object(InMemoryArrowDatabase, "flush", fail), + pytest.raises(StoreWriteError), + ): + _run(job, orchestrator) + assert job.nodes["summed"].as_table().num_rows == 0 + assert job.nodes["rowwise"].as_table().num_rows == 6 + assert job.nodes["grouped"].as_table().num_rows == 2 + + +@pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_partly_held_run_under_each_error_policy( + orchestrator: str, error_policy: Literal["continue", "fail_fast"] +) -> None: + """Held nodes feed the nodes that run, on both orchestrators and error policies.""" + job = _build(ROWS, InMemoryArrowDatabase()) + kind = ORCHESTRATORS[orchestrator] + job.run(orchestrator=kind(error_policy=error_policy)) + CALLS.clear() + job.nodes["summed"].clear_cache() + summary = job.run(orchestrator=kind(error_policy=error_policy)) + assert CALLS == [] + assert summary.nodes["rowwise"].held == 6 + assert summary.nodes["summed"].reused == 2 + assert _totals(job) == [(0, 63), (1, 93)] diff --git a/tests/test_pipeline/test_load_only_run.py b/tests/test_pipeline/test_load_only_run.py new file mode 100644 index 000000000..f3c1ff5c9 --- /dev/null +++ b/tests/test_pipeline/test_load_only_run.py @@ -0,0 +1,232 @@ +"""``job.run(load_only=True)``: fill the nodes from what is stored, compute nothing (ITL-1095). + +Developer, November 13, 2026: "it might help for run to have an argument ... load_only=True +where it would fill the pipeline only based on what can be loaded ... and does not trigger +computation of any missing entries." A load-only run reads the sources, runs operators on +the rows that reach them, and fills each function node with the results already stored for +this run's inputs. No user function is called, no side effect is delivered, nothing is +written to the store. Inputs without a stored result are skipped and counted. +""" + +from __future__ import annotations + +import asyncio +from typing import Any + +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.side_effects import SideEffectPod, tap_pod +from orcapod.sources import DictSource +from orcapod.types import CacheMode + +CALLS: list[str] = [] +DELIVERED: list[int] = [] + + +@function_pod(output_keys="y") +def inc(x: int) -> int: + """Return ``x + 1``.""" + CALLS.append("inc") + return x + 1 + + +@function_pod(output_keys="total") +def total(y: list[int]) -> int: + """Return the sum of a group's values.""" + CALLS.append("total") + return sum(y) + + +def _deliver(y: int, ctx: Any) -> None: + """A side effect: record the delivered value.""" + DELIVERED.append(y) + + +ROWS = [{"g": i % 2, "k": i, "x": 10 * i} for i in range(6)] +ORCHESTRATORS = {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator} + + +@pytest.fixture(autouse=True) +def _reset() -> None: + CALLS.clear() + DELIVERED.clear() + + +def _build( + data: list[dict[str, int]], + db: Any, + cache_mode: CacheMode | None = None, + with_tap: bool = False, +) -> PipelineJob: + """``rowwise`` (function) -> ``grouped`` (GroupBy) -> ``summed`` (function). + + With ``with_tap``, a side-effect node ``tap`` sits between ``rowwise`` and the group. + """ + src = DictSource(data, tag_columns=["g", "k"], source_id="s") + job = PipelineJob(name="j", database=db) + with job: + rows: Any = inc.pod(src, label="rowwise") + if with_tap: + tap = tap_pod(_deliver) + assert isinstance(tap, SideEffectPod) + rows = tap.process(rows, label="tap") + grouped = rows.group_by("g", label="grouped") + total.pod(grouped, label="summed") + if cache_mode is not None: + job.nodes["grouped"].attach_databases( + pipeline_database=db.at("j"), cache_mode=cache_mode + ) + return job + + +def _snapshot(db: InMemoryArrowDatabase) -> dict[tuple[str, ...], int]: + """Every table of the store with its row count.""" + db.flush() + counts: dict[tuple[str, ...], int] = {} + for path in db.list_table_paths(): + records = db.get_all_records(path) + counts[path] = 0 if records is None else records.num_rows + return counts + + +def _run(job: PipelineJob, orchestrator: str, **kwargs: Any) -> Any: + return job.run(orchestrator=ORCHESTRATORS[orchestrator](), **kwargs) + + +def _totals(job: PipelineJob) -> list[tuple[int, int]]: + return sorted( + (r["g"], r["total"]) for r in job.nodes["summed"].as_table().to_pylist() + ) + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_load_only_run_over_a_full_store_equals_a_run_and_computes_nothing( + orchestrator: str, +) -> None: + db = InMemoryArrowDatabase() + _run(_build(ROWS, db), orchestrator) + expected = _build(ROWS, InMemoryArrowDatabase()) + _run(expected, orchestrator) + CALLS.clear() + before = _snapshot(db) + + job = _build(ROWS, db) + summary = _run(job, orchestrator, load_only=True) + + assert CALLS == [] + assert _snapshot(db) == before + assert job.nodes["rowwise"].as_table().num_rows == 6 + assert _totals(job) == _totals(expected) + assert summary.load_only + rowwise = summary.nodes["rowwise"] + assert (rowwise.computed, rowwise.reused, rowwise.not_stored) == (0, 6, 0) + assert rowwise.inputs_incomplete == () + assert summary.nodes["summed"].inputs_incomplete == () + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_inputs_without_a_stored_result_are_skipped_and_counted( + orchestrator: str, +) -> None: + """The store holds a 3-row run; the source now has 6 rows.""" + db = InMemoryArrowDatabase() + _run(_build(ROWS[:3], db), orchestrator) + CALLS.clear() + before = _snapshot(db) + + job = _build(ROWS, db) + summary = _run(job, orchestrator, load_only=True) + + assert CALLS == [] + assert _snapshot(db) == before + assert sorted(r["k"] for r in job.nodes["rowwise"].as_table().to_pylist()) == [ + 0, + 1, + 2, + ] + rowwise = summary.nodes["rowwise"] + assert (rowwise.computed, rowwise.reused, rowwise.not_stored) == (0, 3, 3) + # The groups below hold only the loaded rows: those of the earlier 3-row run, whose + # totals are stored. The summary says their inputs are incomplete. + assert _totals(job) == [(0, 22), (1, 11)] + summed = summary.nodes["summed"] + assert (summed.reused, summed.not_stored) == (2, 0) + assert summed.inputs_incomplete == ("rowwise",) + text = repr(summary) + assert "rowwise: 6 rows: 3 loaded, 3 not stored (skipped)" in text + assert "summed: 2 rows: 2 loaded (inputs incomplete: rowwise)" in text + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_load_only_run_over_an_empty_store_fills_nothing_and_writes_nothing( + orchestrator: str, +) -> None: + db = InMemoryArrowDatabase() + job = _build(ROWS, db) + summary = _run(job, orchestrator, load_only=True) + assert CALLS == [] + assert _snapshot(db) == {} + for label in ("rowwise", "grouped", "summed"): + assert job.nodes[label].as_table().num_rows == 0, label + assert summary.nodes["rowwise"].not_stored == 6 + assert summary.nodes["summed"].reused == 0 + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_load_only_run_delivers_no_side_effect_and_passes_the_rows_on( + orchestrator: str, +) -> None: + db = InMemoryArrowDatabase() + _run(_build(ROWS, db, with_tap=True), orchestrator) + assert len(DELIVERED) == 6 + DELIVERED.clear() + CALLS.clear() + before = _snapshot(db) + + job = _build(ROWS, db, with_tap=True) + _run(job, orchestrator, load_only=True) + + assert DELIVERED == [] + assert CALLS == [] + assert _snapshot(db) == before + assert _totals(job) == [(0, 63), (1, 93)] + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_a_log_operator_writes_nothing_in_a_load_only_run(orchestrator: str) -> None: + db = InMemoryArrowDatabase() + _run(_build(ROWS[:3], db, cache_mode=CacheMode.LOG), orchestrator) + before = _snapshot(db) + + job = _build(ROWS[:3], db, cache_mode=CacheMode.LOG) + _run(job, orchestrator, load_only=True) + + assert _snapshot(db) == before + assert job.nodes["grouped"].as_table().num_rows == 2 + + +def test_run_async_takes_load_only() -> None: + db = InMemoryArrowDatabase() + _run(_build(ROWS, db), "sync") + CALLS.clear() + job = _build(ROWS, db) + summary = asyncio.run(job.run_async(load_only=True)) + assert CALLS == [] + assert summary.load_only + assert job.nodes["rowwise"].as_table().num_rows == 6 + + +def test_load_only_is_keyword_only() -> None: + job = _build(ROWS, InMemoryArrowDatabase()) + with pytest.raises(TypeError): + job.run(None, None, True, 100, True) # pyright: ignore[reportCallIssue] # load_only is keyword-only + + +def test_a_full_run_summary_is_not_load_only() -> None: + summary = _build(ROWS, InMemoryArrowDatabase()).run() + assert not summary.load_only + assert summary.nodes["rowwise"].not_stored == 0 + assert summary.nodes["rowwise"].inputs_incomplete == () diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index a12320d24..bd8f143eb 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -111,10 +111,12 @@ def static_process_over_no_rows( op_html = job.nodes["sel"]._repr_html_() assert "could not read" not in op_html assert CALLS == [] - assert fn_head.status is ViewStatus.NOTHING_STORED + # A node that has not run says so and names both ways in (ITL-1095). + for head in (fn_head, op_head): + assert head.status is ViewStatus.NOT_COMPUTED + assert "not run" in head.header() + assert "as_source()" in head.header() assert fn_head.table.column_names == ["id", "y"] - assert op_head.status is ViewStatus.NOT_COMPUTED - assert "cache mode OFF" in op_head.header() @pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) @@ -142,26 +144,33 @@ def test_nodes_after_a_run_show_their_first_rows( @pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) -def test_a_new_job_over_a_filled_store_reads_stored_rows_without_computing( +def test_a_new_job_over_a_filled_store_shows_rows_once_its_run_reads_the_store( store_kind: str, tmp_path: Path ) -> None: - """A new job over a filled store shows stored rows without computing.""" + """A new job over a filled store shows no rows until it runs (ITL-1095); its run + reads every result from the store and computes nothing.""" store = _store(store_kind, tmp_path) _ = _job(store).run() CALLS.clear() - node = _job(store).nodes["triple"] + job = _job(store) + node = job.nodes["triple"] + assert node.head(4).status is ViewStatus.NOT_COMPUTED + # The stored view shows the stored rows meanwhile. + assert node.as_source().head(4).status is ViewStatus.ROWS + _ = job.run() head = node.head(4) assert CALLS == [] assert head.status is ViewStatus.ROWS - # The whole stored slice is read, so the header states the exact count and - # makes no claim of a bounded read. - assert head.header() == "FunctionJobNode[triple] — first 4 of 25 stored rows" + assert head.header() == "FunctionJobNode[triple] — first 4 of 25 rows" assert head.table.equals(node.as_table().slice(0, 4)) -def test_a_loaded_job_shows_stored_rows_and_never_raises(tmp_path: Path) -> None: - """A loaded job's nodes show stored rows, or say why there are none.""" +def test_a_loaded_job_shows_no_rows_until_it_runs_and_never_raises( + tmp_path: Path, +) -> None: + """A loaded job's nodes show no rows before a run (ITL-1095) and say where the + stored rows are; ``as_source()`` shows them.""" store = DeltaTableDatabase(base_path=tmp_path / "store") job = _job(store) _ = job.run() @@ -173,10 +182,12 @@ def test_a_loaded_job_shows_stored_rows_and_never_raises(tmp_path: Path) -> None fn_head = loaded.nodes["triple"].head() op_head = loaded.nodes["sel"].head() assert CALLS == [] - assert fn_head.status is ViewStatus.ROWS - assert fn_head.header().endswith("first 10 of 25 stored rows") - assert op_head.status is ViewStatus.UNAVAILABLE - assert "cache mode OFF" in op_head.header() + for head in (fn_head, op_head): + assert head.status is ViewStatus.NOT_COMPUTED + assert "as_source()" in head.header() + stored = loaded.nodes["triple"].as_source().head() + assert stored.status is ViewStatus.ROWS + assert stored.header().endswith("first 10 of 25 stored rows") def test_blueprint_nodes_are_unavailable() -> None: @@ -220,27 +231,27 @@ def _operator_node( def test_operator_node_cache_modes() -> None: - """Operator nodes show stored rows in REPLAY and none in LOG before a run.""" + """Before a run an operator node shows no rows in every cache mode (ITL-1095); a + REPLAY node shows the stored rows once it has run.""" store = InMemoryArrowDatabase() assert _operator_node(CacheMode.REPLAY, store).head().status is ( - ViewStatus.NOTHING_STORED + ViewStatus.NOT_COMPUTED ) seed = _operator_node(CacheMode.LOG, store) _ = seed.execute(_source()) replay = _operator_node(CacheMode.REPLAY, store) + assert replay.head(3).status is ViewStatus.NOT_COMPUTED + _ = replay.run() head = replay.head(3) assert head.status is ViewStatus.ROWS - assert ( - head.header() - == "OperatorJobNode[SelectDataColumns] — first 3 of 25 stored rows" - ) + assert head.header() == "OperatorJobNode[SelectDataColumns] — first 3 of 25 rows" assert head.table.equals(replay.as_table().slice(0, 3)) log = _operator_node(CacheMode.LOG, store) log_head = log.head() assert log_head.status is ViewStatus.NOT_COMPUTED - assert "not replayed" in log_head.header() + assert "as_source()" in log_head.header() @pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) @@ -281,7 +292,7 @@ def test_head_keeps_the_tag_arrow_type_of_the_input( """A node's head keeps an ``int8`` tag ``int8`` and is the prefix of ``as_table``. ``head`` and ``as_table`` share one table build (ITL-928 typing), in memory after a - run and from the store in a new job. + run and in a new job whose run reads its rows from the store. """ table = pa.table( {"id": pa.array(range(12), pa.int8()), "x": pa.array(range(12), pa.int64())}, @@ -306,7 +317,10 @@ def build(store: ArrowDatabaseProtocol) -> PipelineJob: store = _store(store_kind, tmp_path) job = build(store) _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) - for node in (job.nodes["triple"], build(store).nodes["triple"]): + # The new job shows rows once its run has read them from the store (ITL-1095). + fresh = build(store) + _ = fresh.run(orchestrator=ORCHESTRATORS[orchestrator]()) + for node in (job.nodes["triple"], fresh.nodes["triple"]): head = node.head(5, all_info=True) assert head.status is ViewStatus.ROWS assert head.table.schema.field("id").type == pa.int8() diff --git a/tests/test_pipeline/test_node_head_sorting.py b/tests/test_pipeline/test_node_head_sorting.py index 42dccf48f..bc4e38793 100644 --- a/tests/test_pipeline/test_node_head_sorting.py +++ b/tests/test_pipeline/test_node_head_sorting.py @@ -121,7 +121,8 @@ def _held_order(node: Any) -> list[int]: def test_a_function_nodes_head_is_the_prefix_of_as_table_in_any_completion_order( run_kind: str, store_kind: str, n: int, config: str, tmp_path: Path ) -> None: - """After a run, and from the store in a new job, head(n, cfg) == as_table(cfg)[:n].""" + """After a run, and in a new job whose run reads every row from the store, + head(n, cfg) == as_table(cfg)[:n].""" store = _store(store_kind, tmp_path) job = _run(run_kind, store) assert _held_order(job.nodes["node"]) != sorted(_held_order(job.nodes["node"])) @@ -131,6 +132,9 @@ def test_a_function_nodes_head_is_the_prefix_of_as_table_in_any_completion_order store, list(range(N_ROWS)), ) + # A node shows its current run only (ITL-1095): the new job's run reads its rows from + # the store, in the store's order, and computes nothing. + fresh.run() for node in (job.nodes["node"], fresh.nodes["node"]): head = node.head(n, **CONFIGS[config]) assert head.status is ViewStatus.ROWS @@ -175,11 +179,14 @@ def node(mode: CacheMode) -> OperatorJobNode: ran = node(CacheMode.LOG) ran.execute(_source(order)) - for each in (ran, node(CacheMode.REPLAY)): + # A REPLAY node shows the stored rows once it has run (ITL-1095). + replayed = node(CacheMode.REPLAY) + replayed.run() + for each in (ran, replayed): head = each.head(3, **CONFIGS[config]) assert head.status is ViewStatus.ROWS assert head.table.equals(each.as_table(**CONFIGS[config]).slice(0, 3)) - sorted_ids = node(CacheMode.REPLAY).head(3, all_info=True).table.column("id") + sorted_ids = replayed.head(3, all_info=True).table.column("id") assert sorted_ids.to_pylist() == [0, 1, 2] diff --git a/tests/test_pipeline/test_node_run_cascade.py b/tests/test_pipeline/test_node_run_cascade.py new file mode 100644 index 000000000..5fd93ffff --- /dev/null +++ b/tests/test_pipeline/test_node_run_cascade.py @@ -0,0 +1,415 @@ +"""``node.run()`` runs the job nodes it depends on, by their state (ITL-1095). + +Developer, November 13, 2026: "given that the initially loaded job is empty, I think it's +fair that node.run will cascade node.run on all nodes it depends on recursively. +Critically, node.run(load_only=True) would trigger the upstream node.run also with +load_only=True." And: "if it's not stale, it shouldn't even trigger a run and simply return +what's in the stream cached already." A node filled by a load-only run has its own status +and is never treated as complete by a full run. + +The cascade runs the node's ancestors and the node on the synchronous orchestrator, as +``job.run()`` runs that sub-graph: same rows, same stored records, same counts +(``node.last_run``), and the same end-of-run commit. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any +from unittest import mock + +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.operators import Join +from orcapod.core.streams import ViewStatus +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.errors import RowFailuresWarning +from orcapod.side_effects import SideEffectPod, tap_pod +from orcapod.sources import DictSource +from orcapod.types import CacheMode + +CALLS: list[str] = [] +DELIVERED: list[int] = [] + + +@function_pod(output_keys="y") +def inc(x: int) -> int: + """Return ``x + 1``.""" + CALLS.append("inc") + return x + 1 + + +@function_pod(output_keys="total") +def total(y: list[int]) -> int: + """Return the sum of a group's values.""" + CALLS.append("total") + return sum(y) + + +@function_pod(output_keys="z") +def dbl(y: int) -> int: + """Return ``2 * y``.""" + CALLS.append("dbl") + return 2 * y + + +@function_pod(output_keys="w") +def neg(y: int) -> int: + """Return ``-y``.""" + CALLS.append("neg") + return -y + + +@function_pod(output_keys="y") +def fails_on_20(x: int) -> int: + """Return ``x + 1``; raise for ``x == 20``.""" + if x == 20: + raise ValueError("x is 20") + return x + 1 + + +def _deliver(y: int, ctx: Any) -> None: + """A side effect: record the delivered value.""" + DELIVERED.append(y) + + +ROWS = [{"g": i % 2, "k": i, "x": 10 * i} for i in range(6)] + + +@pytest.fixture(autouse=True) +def _reset() -> None: + CALLS.clear() + DELIVERED.clear() + + +def _source(data: list[dict[str, int]]) -> DictSource: + return DictSource(data, tag_columns=["g", "k"], source_id="s") + + +def _build( + data: list[dict[str, int]], + db: Any, + *, + cache_mode: CacheMode | None = None, + with_tap: bool = False, +) -> PipelineJob: + """``rowwise`` -> [``tap``] -> ``grouped`` (GroupBy) -> ``summed``.""" + job = PipelineJob(name="j", database=db) + with job: + rows: Any = inc.pod(_source(data), label="rowwise") + if with_tap: + tap = tap_pod(_deliver) + assert isinstance(tap, SideEffectPod) + rows = tap.process(rows, label="tap") + grouped = rows.group_by("g", label="grouped") + total.pod(grouped, label="summed") + if cache_mode is not None: + job.nodes["grouped"].attach_databases( + pipeline_database=db.at("j"), cache_mode=cache_mode + ) + return job + + +def _snapshot(db: Any) -> dict[tuple[str, ...], int]: + db.flush() + counts: dict[tuple[str, ...], int] = {} + for path in db.list_table_paths(): + records = db.get_all_records(path) + counts[path] = 0 if records is None else records.num_rows + return counts + + +def _totals(table: Any) -> list[tuple[int, int]]: + return sorted((r["g"], r["total"]) for r in table.to_pylist()) + + +class TestCascade: + """``node.run()`` first runs every job node it depends on, each once.""" + + def test_a_node_of_a_fresh_job_runs_its_inputs_first(self) -> None: + job = _build(ROWS, InMemoryArrowDatabase()) + view = job.nodes["summed"].run() + assert sorted(CALLS) == ["inc"] * 6 + ["total"] * 2 + assert _totals(view.as_table()) == [(0, 63), (1, 93)] + # Every node of the cascade holds its run, as after job.run(). + for label in ("rowwise", "grouped", "summed"): + assert job.nodes[label].head().status is ViewStatus.ROWS, label + expected = _build(ROWS, InMemoryArrowDatabase()) + expected.run() + for label in ("rowwise", "summed"): + assert ( + job.nodes[label] + .as_table() + .sort_by("g") + .equals(expected.nodes[label].as_table().sort_by("g")) + ), label + + def test_the_cascade_reports_its_counts_on_the_node(self) -> None: + db = InMemoryArrowDatabase() + _build(ROWS[:3], db).run() + CALLS.clear() + job = _build(ROWS, db) + _ = job.nodes["summed"].run() + summary = job.nodes["summed"].last_run + assert summary is not None + assert not summary.load_only + rowwise = summary.nodes["rowwise"] + assert (rowwise.computed, rowwise.reused) == (3, 3) + assert summary.nodes["summed"].computed == 2 + assert "summed" in repr(summary) + + def test_a_diamond_runs_each_ancestor_once(self) -> None: + job = PipelineJob(name="d", database=InMemoryArrowDatabase()) + with job: + top = inc.pod(_source(ROWS), label="top") + left = dbl.pod(top, label="left") + right = neg.pod(top, label="right") + Join()(left, right, label="joined") + view = job.nodes["joined"].run() + assert view.as_table().num_rows == 6 + assert sorted(CALLS) == sorted(["inc"] * 6 + ["dbl"] * 6 + ["neg"] * 6) + + def test_failed_rows_warn_as_a_job_run_does(self) -> None: + job = PipelineJob(name="f", database=InMemoryArrowDatabase()) + with job: + fails_on_20.pod(_source(ROWS), label="node") + with pytest.warns( + RowFailuresWarning, match="1 of 6 rows failed in node 'node'" + ): + view = job.nodes["node"].run() + assert view.as_table().num_rows == 5 + + +class TestState: + """A node that holds a current run is not run again; a stale one is.""" + + def test_a_second_run_of_an_unchanged_node_does_nothing(self) -> None: + db = InMemoryArrowDatabase() + job = _build(ROWS, db) + first = job.nodes["summed"].run() + CALLS.clear() + with ( + mock.patch.object( + InMemoryArrowDatabase, + "get_all_records", + side_effect=AssertionError("store read"), + ), + mock.patch.object( + InMemoryArrowDatabase, + "add_records", + side_effect=AssertionError("store write"), + ), + ): + second = job.nodes["summed"].run() + assert CALLS == [] + assert second.as_table().equals(first.as_table()) + + def test_a_held_ancestor_is_not_run_again(self) -> None: + job = _build(ROWS, InMemoryArrowDatabase()) + _ = job.nodes["rowwise"].run() + stamped = job.nodes["rowwise"].last_modified + CALLS.clear() + _ = job.nodes["summed"].run() + assert job.nodes["rowwise"].last_modified == stamped + assert sorted(CALLS) == ["total", "total"] + + def test_a_node_runs_again_after_its_source_changed(self) -> None: + job = _build(ROWS[:3], InMemoryArrowDatabase()) + _ = job.nodes["summed"].run() + CALLS.clear() + job.bind(sources={next(iter(job.sources)): _source(ROWS)}) + assert job.nodes["summed"].is_stale + view = job.nodes["summed"].run() + assert CALLS.count("inc") == 3 + assert _totals(view.as_table()) == [(0, 63), (1, 93)] + + +class TestLoadOnly: + """``node.run(load_only=True)`` loads through the cascade; its rows have a status.""" + + def test_a_load_only_cascade_computes_and_writes_nothing(self) -> None: + db = InMemoryArrowDatabase() + _build(ROWS[:3], db).run() + CALLS.clear() + before = _snapshot(db) + job = _build(ROWS, db) + view = job.nodes["summed"].run(load_only=True) + assert CALLS == [] + assert _snapshot(db) == before + assert _totals(view.as_table()) == [(0, 22), (1, 11)] + summary = job.nodes["summed"].last_run + assert summary is not None and summary.load_only + assert summary.nodes["rowwise"].not_stored == 3 + rowwise_head = job.nodes["rowwise"].head() + assert rowwise_head.status is ViewStatus.LOADED_ONLY + assert "3 loaded, 3 inputs without a stored result" in rowwise_head.header() + summed_head = job.nodes["summed"].head() + assert summed_head.status is ViewStatus.LOADED_ONLY + assert "inputs incomplete: rowwise" in summed_head.header() + + def test_a_full_run_after_a_load_only_run_computes_what_is_missing(self) -> None: + db = InMemoryArrowDatabase() + _build(ROWS[:3], db).run() + CALLS.clear() + job = _build(ROWS, db) + _ = job.nodes["summed"].run(load_only=True) + assert CALLS == [] + view = job.nodes["summed"].run() + assert CALLS.count("inc") == 3 + assert _totals(view.as_table()) == [(0, 63), (1, 93)] + assert job.nodes["rowwise"].head().status is ViewStatus.ROWS + + def test_a_load_only_run_of_a_load_only_node_returns_what_it_holds(self) -> None: + db = InMemoryArrowDatabase() + _build(ROWS[:3], db).run() + job = _build(ROWS, db) + first = job.nodes["summed"].run(load_only=True) + stamped = job.nodes["rowwise"].last_modified + second = job.nodes["summed"].run(load_only=True) + assert job.nodes["rowwise"].last_modified == stamped + assert second.as_table().equals(first.as_table()) + + def test_a_load_only_cascade_uses_a_fully_run_ancestor(self) -> None: + job = _build(ROWS, InMemoryArrowDatabase()) + _ = job.nodes["rowwise"].run() + stamped = job.nodes["rowwise"].last_modified + _ = job.nodes["summed"].run(load_only=True) + assert job.nodes["rowwise"].last_modified == stamped + + def test_a_load_only_run_that_found_everything_stays_loaded_only(self) -> None: + db = InMemoryArrowDatabase() + _build(ROWS, db).run() + job = _build(ROWS, db) + _ = job.nodes["rowwise"].run(load_only=True) + head = job.nodes["rowwise"].head() + assert head.status is ViewStatus.LOADED_ONLY + assert "6 loaded, 0 inputs without a stored result" in head.header() + + +class TestSideEffects: + def test_a_side_effect_node_follows_the_state_rule(self) -> None: + """Stale before a run; while it holds a current run nothing re-delivers it.""" + job = _build(ROWS, InMemoryArrowDatabase(), with_tap=True) + tap = job.nodes["tap"] + assert tap.is_stale + _ = job.nodes["summed"].run() + assert len(DELIVERED) == 6 + assert tap.holds_current_run() + _ = job.nodes["summed"].run() + _ = tap.run() + assert len(DELIVERED) == 6 + tap.clear_cache() + assert job.nodes["summed"].is_stale + _ = job.nodes["summed"].run() + assert len(DELIVERED) == 12 + + def test_a_load_only_run_of_a_side_effect_node_holds_its_rows(self) -> None: + db = InMemoryArrowDatabase() + _build(ROWS, db, with_tap=True).run() + DELIVERED.clear() + job = _build(ROWS, db, with_tap=True) + _ = job.nodes["summed"].run(load_only=True) + tap = job.nodes["tap"] + assert DELIVERED == [] + assert tap.as_table().num_rows == 6 + assert tap.holds_current_run(load_only=True) + assert not tap.holds_current_run() + + def test_a_full_cascade_delivers_a_load_only_cascade_does_not(self) -> None: + db = InMemoryArrowDatabase() + job = _build(ROWS, db, with_tap=True) + _ = job.nodes["summed"].run(load_only=True) + assert DELIVERED == [] + _ = job.nodes["summed"].run() + assert sorted(DELIVERED) == sorted(r["x"] + 1 for r in ROWS) + + +def test_the_cascade_commits_every_write(tmp_path: Path) -> None: + """A store opened again through a new object holds every result and record.""" + path = tmp_path / "store" + job = _build(ROWS, DeltaTableDatabase(path), cache_mode=CacheMode.LOG) + _ = job.nodes["summed"].run() + CALLS.clear() + probe = _build(ROWS, DeltaTableDatabase(path), cache_mode=CacheMode.LOG) + for label, rows in (("rowwise", 6), ("grouped", 2), ("summed", 2)): + records = probe.nodes[label].get_all_records() + assert records is not None and records.num_rows == rows, label + again = _build(ROWS, DeltaTableDatabase(path)) + summary = again.run() + assert CALLS == [] + assert summary.nodes["rowwise"].reused == 6 + assert summary.nodes["summed"].reused == 2 + + +class TestFailedFinalCommit: + """A run whose end-of-run commit fails leaves no node holding its rows.""" + + @staticmethod + def _flush_calls(run: Any) -> int: + """How many flushes ``run()`` makes on an in-memory store.""" + calls = 0 + original = InMemoryArrowDatabase.flush + + def counting(self: Any) -> None: + nonlocal calls + calls += 1 + original(self) + + with mock.patch.object(InMemoryArrowDatabase, "flush", counting): + run() + return calls + + @staticmethod + def _failing_last_flush(total_calls: int) -> Any: + """A flush that fails on call number ``total_calls`` (the final commit).""" + calls = 0 + original = InMemoryArrowDatabase.flush + + def flush(self: Any) -> None: + nonlocal calls + calls += 1 + if calls == total_calls: + raise OSError("disk full") + original(self) + + return mock.patch.object(InMemoryArrowDatabase, "flush", flush) + + @pytest.mark.parametrize("orchestrator", ["sync", "async"]) + def test_a_job_run_whose_commit_fails_leaves_every_node_empty( + self, orchestrator: str + ) -> None: + from orcapod.errors import StoreWriteError + from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator + + kinds = {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator} + total = self._flush_calls( + lambda: _build(ROWS, InMemoryArrowDatabase()).run( + orchestrator=kinds[orchestrator]() + ) + ) + job = _build(ROWS, InMemoryArrowDatabase()) + with self._failing_last_flush(total), pytest.raises(StoreWriteError): + job.run(orchestrator=kinds[orchestrator]()) + for label in ("rowwise", "grouped", "summed"): + node = job.nodes[label] + assert node.as_table().num_rows == 0, label + assert node.head().status is ViewStatus.NOT_COMPUTED, label + assert node.is_stale, label + job.run(orchestrator=kinds[orchestrator]()) + assert _totals(job.nodes["summed"].as_table()) == [(0, 63), (1, 93)] + + def test_a_cascade_whose_commit_fails_leaves_every_node_empty(self) -> None: + from orcapod.errors import StoreWriteError + + total = self._flush_calls( + lambda: _build(ROWS, InMemoryArrowDatabase()).nodes["summed"].run() + ) + job = _build(ROWS, InMemoryArrowDatabase()) + with self._failing_last_flush(total), pytest.raises(StoreWriteError): + job.nodes["summed"].run() + for label in ("rowwise", "grouped", "summed"): + assert job.nodes[label].as_table().num_rows == 0, label + assert job.nodes[label].is_stale, label + view = job.nodes["summed"].run() + assert _totals(view.as_table()) == [(0, 63), (1, 93)] diff --git a/tests/test_pipeline/test_node_staleness_after_run.py b/tests/test_pipeline/test_node_staleness_after_run.py index fae00b2dc..9c0e6a23a 100644 --- a/tests/test_pipeline/test_node_staleness_after_run.py +++ b/tests/test_pipeline/test_node_staleness_after_run.py @@ -325,8 +325,8 @@ def test_a_function_node_is_stamped_before_its_output_closes(with_store: bool) - pipeline_database=store, result_database=store, ) - constructed = node.last_modified - assert constructed is not None + # A node that has not run holds nothing and has no modified time (ITL-1095). + assert node.last_modified is None seen_at_close: list[Any] = [] async def run() -> None: @@ -353,4 +353,4 @@ async def read() -> None: asyncio.run(run()) (stamp,) = seen_at_close - assert stamp is not None and stamp > constructed + assert stamp is not None diff --git a/tests/test_pipeline/test_stored_data_context.py b/tests/test_pipeline/test_stored_data_context.py index 17f072b73..a3d678727 100644 --- a/tests/test_pipeline/test_stored_data_context.py +++ b/tests/test_pipeline/test_stored_data_context.py @@ -48,6 +48,7 @@ from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator from orcapod.side_effects import SideEffectPod, sink_pod from orcapod.sources import ArrowTableSource, DictSource +from orcapod.core.streams import ViewStatus from orcapod.types import CacheMode, NodeConfig Orchestrator = SyncPipelineOrchestrator | AsyncPipelineOrchestrator @@ -764,9 +765,15 @@ def test_alone_they_give_no_rows(self, new_store: Callable[[], Store]): def test_function_node_reads_current_rows_only( self, new_store: Callable[[], Store] ): - """Every read method of a function node returns only the rows of ``CURRENT``.""" + """Every read method of a function node returns only the rows of ``CURRENT``. + + A node shows rows only after a run (ITL-1095); the run reads them from the store. + """ store = both_contexts_store(new_store) - node = build_job(store).nodes["dbl"] + job = build_job(store) + _ = job.run() + assert calls == [] + node = job.nodes["dbl"] assert rows_of(node.as_table()) == [2, 4] assert rows_of(node.get_all_records()) == [2, 4] assert sorted(d.as_dict()["double"] for _, d in node.iter_data()) == [2, 4] @@ -980,7 +987,9 @@ def test_function_node_upstream_of_a_refused_sink_is_not_refused( _ = build_job(store, with_sink=True).run() assert delivered == [] - assert rows_of(build_job(store).nodes["dbl"].as_table()) == [2, 4] + # The function node's stored rows are readable (the stored view: a node shows + # rows only after a run, ITL-1095). + assert rows_of(build_job(store).nodes["dbl"].as_source().as_table()) == [2, 4] def test_completions_logged_by_this_version_are_not_delivered_again( self, store: Store, orchestrator: Orchestrator @@ -1258,9 +1267,10 @@ def test_refused_input_after_a_good_run_leaves_no_run_rows(self, store: Store): """A run refused for its input's data context defines no rows (ITL-1093). After a good run the node holds that run's rows; a later sync run refused for its - input's data context drops them, and the node reads its stored rows, as with no - run in this process. (On the async path the context checked is the node's own - input stream's, so a node refused there never had a run.) + input's data context drops them, and the node reads as with no run in this + process: empty (ITL-1095); its stored view keeps the rows. (On the async path the + context checked is the node's own input stream's, so a node refused there never + had a run.) """ table = pa.table( {"k": ["a", "b"], "x": [1, 2]}, @@ -1283,8 +1293,10 @@ def test_refused_input_after_a_good_run_leaves_no_run_rows(self, store: Store): with pytest.raises(InputValidationError): node.execute(_other_context_stream()) - assert node.head(10).origin == "stored" - assert sorted(row["double"] for row in node.as_table().to_pylist()) == [2, 4] + assert node.head(10).status is ViewStatus.NOT_COMPUTED + assert node.as_table().num_rows == 0 + stored = node.as_source().as_table().to_pylist() + assert sorted(row["double"] for row in stored) == [2, 4] def test_add_pipeline_record_refuses_an_input_of_another_data_context( self, store: Store diff --git a/tests/test_pipeline/test_sync_orchestrator.py b/tests/test_pipeline/test_sync_orchestrator.py index aaa4c9b68..7fc3b67e4 100644 --- a/tests/test_pipeline/test_sync_orchestrator.py +++ b/tests/test_pipeline/test_sync_orchestrator.py @@ -560,6 +560,9 @@ def create_data_logger(self, tag, data, **kwargs): def contextualize(self, *identity_path): return self + # A node holding a current run is not run again (ITL-1095): clear it, so its + # second run reads its result from the store. + job.nodes["doubler"].clear_cache() job.run(observer=Obs2()) assert events2 == [True] diff --git a/tests/test_torch/test_tensor_job_load_back.py b/tests/test_torch/test_tensor_job_load_back.py index 20b819ecd..2d1022c5c 100644 --- a/tests/test_torch/test_tensor_job_load_back.py +++ b/tests/test_torch/test_tensor_job_load_back.py @@ -85,7 +85,8 @@ def test_a_loaded_job_reads_its_stored_tensors( assert loaded.nodes["value"].output_schema()[1]["value"] is torch.Tensor assert loaded.nodes["joined"].load_status == LoadStatus.FULL assert loaded.nodes["joined"].output_schema()[1]["value"] is torch.Tensor - got = _values(loaded.nodes["value"]) + # A loaded node is empty until it runs (ITL-1095): read the stored view. + got = _values(loaded.nodes["value"].as_source()) assert sorted(got) == [1, 2] for row_id, n in ((1, 3), (2, 4)): expected = function(n)