Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
5b07349
docs(itl-1043): investigation scripts and findings for loaded operato…
Nov 13, 2026
85667e1
chore(itl-1043): type-clean the investigation scripts
Nov 13, 2026
51f3ab5
test(operator_node): loaded operator node reads under its run's path,…
Nov 13, 2026
c7e3e4d
fix(operator_node): a loaded operator node reads under its run's path…
Nov 13, 2026
19f424e
docs(itl-1043): changelog fragment, design issue LJ1, loaded operator…
Nov 13, 2026
128e8b7
test(pipeline): save/load round-trips every node's identity and schem…
Nov 13, 2026
8546d74
feat(pipeline): saved jobs record column nullability and operator run…
Nov 13, 2026
4f3f37a
docs(itl-1043): old saved files load lossy; upgrading them is ITL-1048
Nov 13, 2026
a4f78bb
fix(operator_node): a loaded node's head() shows why its stored rows …
Nov 13, 2026
02cd3ac
Merge remote-tracking branch 'origin/main' into eywalker/itl-1043-a-l…
Nov 13, 2026
8954207
test(pipeline): read the ITL-1026 format 0.1.0 fixture's join through…
Nov 13, 2026
8c58bcf
test(pipeline): keep the targeted pyright ignore on its line
Nov 13, 2026
0a3f776
fix(pipeline): loaded replay never caches unreadable rows; re-save ke…
Nov 13, 2026
f5e1967
fix(operator_node): the store is the source of truth for a loaded ope…
Nov 13, 2026
3ff559b
Merge remote-tracking branch 'origin/main' into eywalker/itl-1043-a-l…
Nov 13, 2026
87196d0
feat(pipeline): optional advisory run.expected_rows; a loaded node wa…
Nov 13, 2026
cb98a39
docs(design_issues): note on ITL-904 PJLOAD1: stored-records flag sup…
Nov 13, 2026
8ea8b5e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1043-a-l…
Nov 13, 2026
0c36142
Merge remote-tracking branch 'origin/main' into eywalker/itl-1043-a-l…
Nov 13, 2026
cc1038d
fix(pipeline): advisory check on a loaded operator's get_all_records;…
Nov 13, 2026
6811beb
Merge remote-tracking branch 'origin/main' into eywalker/itl-1043-a-l…
Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions changelog.d/ITL-1043.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
### Added

- **`run.expected_rows` and `StoreContentWarning`.** A saved job file holds an optional,
advisory section, `run.expected_rows`: for each function and operator node, how many rows
the store held under its run's key when the job was saved, written only when that is
certain (after a run that finished and committed; never invented when a loaded job is
saved again). A loaded node always returns what its store holds; when the store holds
another number of rows, the first read issues one `orcapod.errors.StoreContentWarning`
naming the node, both numbers and the store. Deleting the section from the file changes
nothing but this check. (ITL-1043)

### Changed

- **A loaded function node no longer raises when its store lacks the run's results** (it
returns what the store holds, with a `StoreContentWarning`). The store is the source of
truth: a loaded job shows what the store holds for it. This supersedes the
`run.node_has_stored_records` flag and its error described under ITL-904, both unreleased:
the flag is no longer written, and a file that still has it loads without a check. A
dangling reference, a pipeline record whose result is missing from the result store, is
logged and left out, for a loaded node as for a running one. (ITL-1043)

### Fixed

#### A loaded job keeps every node's identity, and a loaded operator reads its stored rows (ITL-1043)

A job saved and loaded with `PipelineJob.load` could not read the stored output of an operator
node. With a nullable tag or data column (the default for a table built with `pa.table` or
from pandas) the loaded node looked for its cache under another table path and reported "the
cache table is absent"; otherwise it found the table and returned zero rows, although rows
were stored.

- The saved file now records which columns of each schema may be null (`T | None`): the Arrow
type text it saves cannot say it, so `int | None` was saved as `int64` and loaded as `int`.
A node descriptor's `output_schema` gains `"nullable": {"tag": [...], "data": [...]}`, and a
source node's `source_config` and a source's identity config the same record. Loaded source,
function and operator nodes report the run's output schema, and `Pipeline.load` followed by
`PipelineJob.from_pipeline(sources=...)` reads and writes the run's operator cache table.
- The saved file records the run content hash of each operator node in
`run.node_job_content_hashes`, as it did for function nodes. A loaded operator node
attached in `CacheMode.REPLAY` reads exactly the rows its run stored.
- A loaded operator node takes the pipeline hash saved with the job, its run's, instead of
recomputing it from inputs rebuilt from the file.
- The store is the source of truth: a loaded operator node attached in `CacheMode.REPLAY`
reads exactly as the same node built from code does. When the store holds no rows under
its key (no table, or none of the table's rows), the result is empty, with the node's
columns, and `head()` reports `NOTHING_STORED`. A run whose result was empty stores
nothing and reads back the same way.
- A loaded operator node whose job file does not record its run's content hash cannot name
its run's rows. When the store holds none under the hash the loaded job computes,
`as_table()` and `iter_data()` raise a `RuntimeError` saying so, instead of returning zero
rows, and `head()` shows no rows with that reason (`UNAVAILABLE`) instead of raising.
`get_cached_output()` follows the same rule.
- Whether a saved file has the nullability and run-hash records is decided by the records,
not by the format version.

No hash, stored path or golden value of a running job changes: only what a loaded job
reconstructs.

### Known limitations

- A job or pipeline file written before these records (format `0.1.0`, or a `0.2.0` file
written before this change) lacks the `"nullable"` record in every `output_schema`,
`source_config` and source identity config, and lacks the operator nodes' entries in
`run.node_job_content_hashes`. It still loads, lossy: every column reads as non-null, and
a loaded operator node that finds no rows under the content hash it computes raises the
error above. Saving such a loaded
job again does not record what its file lacked. This change does not rewrite or upgrade
old files; that is the job of the migration tool after v0.3 (ITL-1048). (ITL-1043)
48 changes: 48 additions & 0 deletions design_issues/ITL-1043-LJ1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
## `src/orcapod/pipeline/job.py` — reading a loaded `PipelineJob`

### LJ1 — a saved job loses column nullability, and does not record the run content hash of an operator node
**Status:** resolved (cache mode not saved remains ITL-977)
**Severity:** high
**Issue:** ITL-1043

Found November 12, 2026. Evidence and scripts: `superpowers/specs/itl-1043/FINDINGS.md`.

- `serialize_schema` (`src/orcapod/pipeline/serialization.py`) writes each column as the Arrow
type of its Python type. An Arrow `DataType` has no field nullability, so `int | None` and
`int` are both saved as `"int64"` (nested nullability survives in the type text; only the
top level is lost). A loaded job's source nodes therefore have non-null schemas, a
different pipeline hash from the run's, and every loaded node's `output_schema()` drops
`| None`. An operator node rebuilt with its operator recomputed its pipeline hash from them
and looked for its cache under another table path; `Pipeline.load` followed by
`PipelineJob.from_pipeline(sources=...)` still does, so REPLAY reads no rows there and a
LOG run writes a second table under the wrong path.
- `PipelineJob.save` records the run content hash of function nodes only
(`run.node_job_content_hashes`). A loaded operator node over in-memory sources (not saved)
cannot reproduce the content hash its rows are stored under, so it cannot select them.
- The operator's `cache_mode` is not saved (ITL-977).

The running node's hashes are the intended ones: nullability is part of the schema hash
(`docs/reference/hashing.md`; golden `datagram.json` `arrow_source.int64` against
`arrow_source.int64_nullable`, `schema.json` `multi_field.nullable_type`). No live hash,
stored path or golden value is wrong; only what a loaded job reconstructs.

**Fix:** (ITL-1043; developer decision, November 12, 2026: record the information in the
saved file.) A node descriptor's `output_schema`, a source node's `source_config` and a
source's identity config record `"nullable": {"tag": [...], "data": [...]}`, restored as
`T | None` through the type converter's own rule (`serialization.serialize_nullability`,
`deserialize_output_schema`, `deserialize_schema(nullable=...)`). `PipelineJob.save`
records the run content hash of operator nodes in `run.node_job_content_hashes`; a loaded
operator node returns it, and the pipeline hash saved in its descriptor. The store is the
source of truth (developer, November 12, 2026): a loaded REPLAY node that knows its key
reads as the node built from code does, so no rows under its key (no table, or none of its
rows) is an empty result. A node whose file lacks the run content hash cannot name its
rows: with none under the hash it computes, `as_table()`/`iter_data()` raise and `head()`
shows no rows with the reason. A file without the records (`0.1.0`, or a `0.2.0` file
written before them) loads as before, lossy, and saving it again records nothing it did
not have. No hash, stored path or golden value of a running job changes. Facts about
stored data stay out of the definition: the job file holds only an optional, advisory
`run.expected_rows` section (rows under each node's run key at save time, written only
when certain); a loaded node returns what the store holds and issues one
`StoreContentWarning` when the number differs. A loaded function node no longer raises on
a store without its run's results (the ITL-904 `node_has_stored_records` flag is no longer
written or read).
9 changes: 9 additions & 0 deletions design_issues/ITL-904-PJLOAD1.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,3 +89,12 @@ latest-index selection (persistent wins equal indices). Loaded REPLAY operators
when their cache table is absent; loaded function `run()` rejects computation, and
loaded jobs report `is_runnable()` as false even after binding sources. Source-schema
restoration uses the saved context converter in both normal and fallback branches.

**Superseded in part (ITL-1043, developer decision, November 12, 2026):** the store is the
source of truth, with one rule for function and operator nodes. `run.node_has_stored_records`
is no longer written, and is ignored when an older file has it: a loaded function node no
longer raises on a store without its run's records; it returns what the store holds. The job
file's optional, advisory `run.expected_rows` makes a difference one `StoreContentWarning`
per node. Likewise, a loaded REPLAY operator whose job file records its run's content hash
reads an absent cache table as no rows instead of raising. See
`design_issues/ITL-1043-LJ1.md`.
42 changes: 35 additions & 7 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -691,23 +691,51 @@ A loaded job is for reading stored results. It cannot be run again: function pod
saved with the job, so `loaded.run()` raises. To compute again, build the job from your code
as before.

The store is the source of truth: a loaded job shows what the store holds for it. The job
file names each run's rows by the content hash the run stored them under, so a loaded node
reads exactly what the same node built from your code would read from that store, and no
rows under that key is an empty result, not an error.

- A function node's `as_table()`, `iter_data()` and `get_all_records()` return the rows the
saved run stored, with the types they were stored with. Rows come in the order the store
returns them; sort them if order matters.
- Operators (a join, a filter) store outputs only when caching is enabled, under `odb_v1`.
A loaded operator without a readable cache raises from `as_table()` and `iter_data()`.
Read its input nodes instead (an in-memory input source must be bound first, see below).
Saved descriptors do not yet preserve all runtime identity and cache configuration needed
for operator readback without reconstruction (follow-up ITL-977).
The saved file does not record an operator's cache mode (follow-up ITL-977), so attach a
loaded operator node in `CacheMode.REPLAY` to read the rows its run stored:
`loaded.nodes["joined"].attach_databases(pipeline_database=store.at("<job name>"),
cache_mode=CacheMode.REPLAY)`. A loaded operator node reads exactly as the same node built
from your code and attached in `CacheMode.REPLAY`; when the store holds no rows under the
node's key (no table, or none of its rows), the answer is an empty result, as for that
node. A loaded operator not attached in `REPLAY` has no readable cache and raises from
`as_table()` and `iter_data()`; read its input nodes instead (an in-memory input source
must be bound first, see below), or build the job from your code.
- A loaded node has the pipeline hash, content hash, table path and output schema its run
had (a source node whose data was not saved gets its run's content hash once its source is
bound). The file records which columns may be null (`T | None`), because the Arrow type text
of a schema cannot say it, and the content hash of each node's run, which the stored rows
carry. Whether a file has these records is decided by the records themselves, not by the
file's format version: a file without them (format `0.1.0`, or a `0.2.0` file written
before they existed) still loads, but lossy: its schemas read every column as non-null,
and a loaded operator node cannot name its run's rows. When the store holds none under the
content hash the loaded job computes, `as_table()` and `iter_data()` raise, and `head()`
shows no rows and says why, rather than presenting stored rows as none. An empty record is
not a missing one: it says that no column may be null. Saving such a loaded job again does
not record what its file did not. Upgrading such files is left to a migration tool after
v0.3.
- Sources that read files, such as `CSVSource` and `DeltaTableSource`, are rebuilt from their
paths. In-memory sources (`DictSource`, `ListSource`, `ArrowTableSource`,
`DataFrameSource`) keep their data only in memory, so they are not saved; loading logs
this at INFO level. This does not affect reading the other nodes. To read such a source
node itself, bind a source to it: `loaded.bind(sources={"mice": source})`.
- The results are read from the store recorded in the file. Pass
`PipelineJob.load(path, store=...)` if the store has moved. If the store you give lacks
results the saved run stored, reading the node raises an error naming the store, instead
of returning no rows.
`PipelineJob.load(path, store=...)` if the store has moved.
- The job file may also hold an optional, advisory section, `run.expected_rows`: for each
node, how many rows the store held under its run's key when the job was saved (written
only when that is certain: after a run that finished). Reading a loaded node always
returns what the store holds; when the store holds another number of rows, the read
issues one `StoreContentWarning` per node, naming the node, both numbers and the store:
the store may not be the one the run wrote, or it may have changed since. Deleting the
section from the file changes nothing but this check.

Logical types in saved schema text carry their complete identifier, written as
`extension<name;hex-metadata>`, including inside nested containers. The loader checks the
Expand Down
66 changes: 65 additions & 1 deletion src/orcapod/contexts/stored.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,15 @@
from __future__ import annotations

import logging
import warnings
from collections.abc import Collection
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, NamedTuple, Protocol, cast
from typing import TYPE_CHECKING, Any, NamedTuple, Protocol, cast

from orcapod.errors import (
InputValidationError,
SideEffectLogDataContextError,
StoreContentWarning,
TableListingNotSupportedError,
)
from orcapod.system_constants import (
Expand Down Expand Up @@ -185,6 +187,68 @@ def report_rows_of_other_data_contexts(
)


class _DescribedStore(Protocol):
"""The part of ``ArrowDatabaseProtocol`` that names a store (no import cycle)."""

@property
def base_path(self) -> tuple[str, ...]:
"""Path of this store's view from the root of the underlying store."""
...

def to_config(self) -> dict[str, Any]:
"""Return the store's configuration, with its location when it has one."""
...


def describe_store_table(
database: _DescribedStore, record_path: tuple[str, ...]
) -> str:
"""Name a stored table for a message: the store's location and the table's path.

Args:
database: The store, or a view of it.
record_path: Path of the table relative to ``database``.

Returns:
For example ``"store at '/data/store' (table 'job/node/...')"``.
"""
store_config = database.to_config()
location = store_config.get("root_uri")
store = (
f"store at {location!r}"
if location
else f"{store_config.get('type', 'attached')} store"
)
table = "/".join(database.base_path + record_path)
return f"{store} (table {table!r})"


def report_unexpected_row_count(
node: str, location: str, expected: int, found: int
) -> None:
"""Warn that a loaded node's store holds another number of rows than its run's.

The store is the source of truth: the read returns what it holds. The warning says
that it differs from what the job file recorded (ITL-1043).

Args:
node: The node, as messages name it (``"FunctionJobNode 'dbl'"``).
location: The store and table, from ``describe_store_table``.
expected: The rows the job file records for the node's run.
found: The rows the store holds under the node's run key.
"""
warnings.warn(
StoreContentWarning(
f"{node}: the job file records {expected} "
f"{'row' if expected == 1 else 'rows'} for its run, but the {location} "
f"holds {found} under the run's key. The read returns what the store "
"holds; the store may not be the one the run wrote, or it may have "
"changed since."
),
stacklevel=3,
)


def report_old_layout_rows(record_path: tuple[str, ...], rows: int) -> None:
"""Warn that a table written at an older table layout was found and is not used.

Expand Down
Loading
Loading