Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -773,8 +773,11 @@ data. The rules:
by the user is never changed. SourceStreamBuilder.build has no bare-hash default: it
needs a source_id or a source_kind.
- ArrowTableStream requires at least one data column; raises ValueError otherwise.
- FunctionNode Phase 1 returns ALL records in the shared pipeline_path DB table.
Phase 2 skips inputs whose hash is already in the DB.
- After a run, a FunctionJobNode returns exactly the rows of that run's inputs, sync or
async (ITL-1093; a run that raised defines no rows). Read without a completed run in this
process (a fresh or loaded job), it returns every record stored under its content hash for
the current data context, not only its current inputs. A run skips inputs whose entry is
already stored.
- Empty data → ArrowTableSource raises ValueError("Table is empty").
- DerivedSource over a job node before run() → presents zero rows with the origin's columns
(no computed records; no error), in the plain form and with tags_as_data.
Expand Down
8 changes: 5 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -814,9 +814,11 @@ data. The rules:
by the user is never changed. `SourceStreamBuilder.build` has no bare-hash default: it
needs a `source_id` or a `source_kind`.
- `ArrowTableStream` requires at least one data column; raises `ValueError` otherwise.
- `FunctionNode.iter_data()` Phase 1 returns ALL records in the shared `pipeline_path`
DB table (not filtered to current inputs). Phase 2 skips inputs whose hash is already
in the DB.
- After a run, a `FunctionJobNode` returns exactly the rows of that run's inputs, sync or
async (ITL-1093; a run that raised defines no rows). Read without a completed run in this
process (a fresh or loaded job), it returns every record stored under its content hash for
the current data context, not only its current inputs. A run skips inputs whose entry is
already stored.
- Empty data → `ArrowTableSource` raises `ValueError("Table is empty")`.
- `DerivedSource` over a job node before `run()` → presents zero rows with the origin's columns
(no computed records; no error), in the plain form and with `tags_as_data`.
Expand Down
3 changes: 3 additions & 0 deletions changelog.d/ITL-1093.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Fixed

- After `job.run()` on either orchestrator, a function job node's `as_table()`, `iter_data()`, `head()` and `run()`, and a stream built over the node, return exactly the rows of that run's inputs (ITL-1093). Before, after an async run a node returned every row stored under its key: a pipeline whose source rows changed between runs under one `source_id` showed the results of earlier runs too (for a function over `group_by`, the partial groups of an earlier run next to the full ones; rows of source rows no longer present). The sync orchestrator did the same when one job object ran again with its source rebound. Rows sent downstream during the run and the `RunSummary` were already right. Results of other inputs stay stored and are reused when those inputs come back; nothing stored, no hash and no table layout changes. A run that raised defines no rows: the node then reads as a node with no run in this process. A node read with no finished run in this process (a job built but not run, or a loaded job) still returns every row stored under its key.
26 changes: 26 additions & 0 deletions design_issues/ITL-1093-JR1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
## `src/orcapod/core/nodes/function_node.py`

### JR1 — After a run, a function job node returned rows of other runs' inputs
**Status:** resolved
**Severity:** high
**Issue:** ITL-1093

A `FunctionJobNode` kept one in-memory cache of results keyed by base entry id, and every read
(`iter_data`, and through it `as_table`, `run()` and any consumer; `head`) returned all of it.
Nothing recorded which inputs the run processed. `async_execute` loads every pipeline record
stored under the node's content hash into that cache before routing its inputs, so after an
async run the node returned every row ever stored for it: partial groups of an earlier run next
to full ones, rows of source rows no longer present. The sync `execute` loaded only its inputs'
entries, so it looked right with a new job per run, but a cache is add-only: running the same
node objects again (one job, its source rebound) accumulated rows on the sync path too. Rows sent
downstream during the run and the `RunSummary` were right on both paths.

**Fix:** The node records the base entry ids of its last completed run's inputs
(`_run_entry_counts`, set by `execute` and `async_execute`), and every read of its rows goes
through one helper, `_held_pairs`, which keeps only those inputs' rows (one per input, so two
equal inputs give two rows, as downstream received them). The cache stays a superset: stored
results of other inputs stay stored and reusable. A run that raised defines no rows: the node
clears its in-memory rows and reads as a node with no run in this process (every row stored
under its content hash), on both orchestrators. Nothing stored, no hash and no layout changed.
A read with no completed run in this process (a fresh or loaded job) still returns every row
stored under the node's content hash.
12 changes: 10 additions & 2 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,13 @@ database. Each recorded step becomes a node, available afterwards as `job.nodes[
listed under its `source_id`). A node is a
stream, so `as_table()`, `iter_data()` and `output_schema()` all work on it.

After `job.run()`, a function node returns exactly the rows of that run's inputs, on either
orchestrator. The database may hold more under the same node: results of an earlier run
over other rows of the source stay stored and are reused when those rows come back, but
they are not rows of this run. A run that raised defines no rows. A node read with no
finished run in this process (a job built but not run, or a loaded job) returns every row
stored under its key.

`InMemoryArrowDatabase` lasts only as long as the Python process. Use
`DeltaTableDatabase("some/directory")` from `orcapod.databases` to keep results on disk
between sessions.
Expand Down Expand Up @@ -696,8 +703,9 @@ file names each run's rows by the content hash the run stored them under, so a l
reads exactly what the same node built from your code would read from that store, and no
rows under that key is an empty result, not an error.

- A function node's `as_table()`, `iter_data()` and `get_all_records()` return the rows the
saved run stored, with the types they were stored with. Rows come in the order the store
- A function node's `as_table()`, `iter_data()` and `get_all_records()` return the rows
stored under the saved run's key, with the types they were stored with: those of every
run of that node over the same sources, not only the saved run's inputs. Rows come in the order the store
returns them; sort them if order matters.
- Operators (a join, a filter) store outputs only when caching is enabled, under `odb_v1`.
The saved file does not record an operator's cache mode (follow-up ITL-977), so attach a
Expand Down
Loading
Loading