Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions changelog.d/ITL-904.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
### Fixed

#### A loaded job can read its results (ITL-904)

A job loaded with `PipelineJob.load` can read its results. A function node's `as_table()` and
`iter_data()` return the rows the saved run stored, with their stored types (they used to raise
"no function pod and no database attached" while `get_all_records()` worked). Without a store,
reading says to pass `PipelineJob.load(path, store=...)` or call `job.bind(store=...)`. `save()`
now records whether the store held records of each function node (`run.node_has_stored_records`,
an optional field; the format version is unchanged). A node saved with records raises, naming
the store, when the attached store holds none of them, instead of returning no rows. When an
entry has several records (it was recomputed), a function node now reads the one with the
greatest recomputation index, on a live node too; it used to keep whichever record the store
returned last. Loading a job in which an operator consumes a function node no longer crashes
with `InputValidationError`, and a loaded node's `output_schema()` reports Python types; a saved
type it cannot restore raises an error naming the column. Registered logical types such as
`Path` restore normally, so downstream operators remain available. Saved
`timestamp`, `duration`, `time` and `decimal` types are now read back. Reading a loaded operator
node without a readable cache raises instead of returning an empty table; operators can store
outputs when caching is enabled. In-memory sources (`DictSource`, `ListSource`,
`ArrowTableSource`, `DataFrameSource`) are
logged at INFO with the remedy instead of a "skipping" warning. A loaded job stays read-only and
cannot be run again. See *Saving and loading a job* in the function-pods guide. (ITL-904)

Stored-record expectations count only rows of the node's saved data context. Operator
descriptors without a reconstructed operator still lack the saved runtime identity needed
to find cached rows; the runtime-descriptor change is deferred to ITL-977. Direct execution
of loaded operator nodes is rejected. A newer unavailable ephemeral result yields an
`EmptyData` placeholder instead of exposing an older persistent result.

### Breaking changes and migration

- Saved pipeline/job schema text now includes the full logical type identifier as
`extension<name;hex-metadata>`, including nested logical types. Loading validates it
against the registered type; incompatible or malformed identifiers raise
`LogicalTypeIdentifierError` naming the node and column. Name-only extension text from
older saved files is rejected with an earlier-version error. Read those files with the
version that wrote them, and save a new job from the original pipeline under this version.
Stored table layouts, value decoding and hashes are unchanged. Unregistered logical types
are reported by node and column; affected operators load as read-only stubs. (ITL-904)

### Changed

- When persistent and ephemeral results exist for the same input, reads select the
greatest recomputation index across both stores; persistent rows retain precedence
for equal indices. Loaded REPLAY operators report an absent cache table explicitly,
loaded functions reject `run()`, and loaded jobs report `is_runnable()` as false.
Source schema restoration uses the saved data context's type converter. (ITL-904)
91 changes: 91 additions & 0 deletions design_issues/ITL-904-PJLOAD1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
## `src/orcapod/pipeline/job.py` — reading a loaded `PipelineJob`

### PJLOAD1 — a loaded job could not read its nodes, and could not load a pod feeding an operator
**Status:** resolved
**Severity:** high
**Issue:** ITL-904

Saving a job and loading it with `PipelineJob.load` failed in four ways:

- On a loaded function node, `as_table()` and `iter_data()` raised "no function pod and no
database attached", although the store was attached and `get_all_records()` read the rows.
The cause: `FunctionJobNode.from_descriptor` set `LoadStatus.UNAVAILABLE`, and attaching the
store never changed that status.
- A loaded node's `output_schema()` returned the serialized type strings (`'int64'`), and its
`data_context` dereferenced the missing pod. Loading a job in which an operator (a `Join`)
consumes a function node therefore crashed with `InputValidationError`.
- A loaded operator without a readable cache returned an empty table, as if the operator
had produced nothing. Operators can store outputs when caching is enabled (`odb_v1`).
- Loading logged every source it could not rebuild, in-memory ones included, as "Could not
reconstruct source … — skipping", giving neither the consequence nor a remedy.

**Fix:**

- Attaching a store makes a loaded function node `READ_ONLY`. `iter_data()` yields the stored
rows: those whose `_node_content_hash` is the content hash saved in
`run.node_job_content_hashes`, the same set `get_all_records()` returns. `as_table()` is an
`ArrowTableStream` over the stored Arrow records, so stored types are kept and `all_info`
includes system tags, source info and the data context.
- Without a store, reading names `PipelineJob.load(path, store=...)` and `job.bind(store=...)`.
- `save()` records `run.node_has_stored_records`: whether the store held at least one pipeline
record carrying each function node's content hash. This is an additive, optional field, and
the format version is unchanged. A node saved with records raises, naming the store and the
table, when the attached store holds none of them, instead of reading zero rows. Records
that other runs wrote to a shared table do not count. Files saved without the field read
zero rows, as before.
- A loaded node's `output_schema()` returns Python types (`deserialize_output_schema`), and its
data context comes from the descriptor.
- `parse_arrow_type_string` reads the parameterized types it used to reject: `timestamp[us, tz=UTC]`,
`duration`, `time32`, `time64`, `date64` and `decimal128`/`decimal256`. Commas inside `[...]` and
`(...)` no longer split struct fields.
- Logical schema types are saved with the full identifier defined by `arrow_type_identifier`:
`extension<name;hex-metadata>`, recursively inside containers. The reader resolves the
registered type and applies `require_readable`. Path, UUID, File, Directory, ndarray,
registered nominal classes and structural classes reconstruct normally. A missing
registration is reported by node and column, leaving affected operators as stubs.
Incompatible identifiers raise `LogicalTypeIdentifierError`; legacy name-only text raises
an earlier-version error. This narrow saved-file change was approved November 12, 2026.
- `get_all_records()` on a loaded node raises the same store errors as `as_table()`.
- Re-saving a loaded job keeps each node's saved expectation of stored records.
- One row per base entry, for `as_table()` and `iter_data()` alike. When an entry has several
records (it was recomputed), the record with the greatest recomputation index wins. This
also changes a live node's hot-load from the store: it used to keep whichever record the
join returned last.
- A loaded node whose results lived in an ephemeral store yields `EmptyData` placeholders from
`iter_data()`. Its `as_table()` raises an error saying the ephemeral store is not saved.
A newer missing ephemeral recomputation takes precedence over an older persistent row.
- A loaded operator node with nothing stored raises, names its input nodes, and says to bind an
unsaved input source first.
- Stored-record expectations constrain both the node content hash and the saved data context.
Rows of another data context cannot satisfy the expectation while reads omit those rows.
- Loading logs in-memory sources at INFO, and any other source or store failure at WARNING
with the exception. Each message states the consequence and the remedy.
- `PipelineJob.run()` rejects every loaded job, including an operator-only one whose
operators were rebuilt: a loaded job is read-only by design (developer, November 11,
2026).
- Direct operator `run`, `execute`, and `async_execute` also reject loaded nodes; async
rejection closes the output channel.

Regressions: `tests/test_pipeline/loaded_job/`, which saves in a separate process and loads
in the test process, for each source kind.

Open, for a follow-up:
- The saved type strings drop nullability (`float | None` is saved as `double`), so a loaded
node reports the non-null type.
- Map type text also omits custom item field names and item nullability; a non-null map
item reloads as nullable. Preserving these fields needs additional schema syntax and is
deferred to the pipeline save/load overhaul, ITL-977 (approved November 12, 2026).
- Unknown classes still require registration in the reader process.

The hash-step review also found that an operator built directly from a saved blueprint
descriptor cannot locate its stored rows: the descriptor lacks `pipeline_path` and carries
the definition content hash rather than the run's hash. The combined hash-step regression
still injects those values from the live node. The developer deferred this change to ITL-977;
the recommended approach is additive runtime descriptors in `run.node_descriptors`, preserving
blueprint identities and edges. The separate full logical-identifier gap is resolved here.

Review follow-ups: readable persistent and ephemeral records now both reach the
latest-index selection (persistent wins equal indices). Loaded REPLAY operators raise
when their cache table is absent; loaded function `run()` rejects computation, and
loaded jobs report `is_runnable()` as false even after binding sources. Source-schema
restoration uses the saved context converter in both normal and fallback branches.
66 changes: 66 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,72 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro
This is the contract for now. An opt-in mode that tolerates store failures may be added
later.

### Saving and loading a job

`job.save(path)` writes the job to a JSON file: the graph, the store's location and how each
source was configured. `PipelineJob.load(path)` reads it back, for example in a later Python
session, so that you can read the results the job stored:

```python
import tempfile
from pathlib import Path

from orcapod.databases import DeltaTableDatabase

workdir = Path(tempfile.mkdtemp()) # any directory you keep

job = PipelineJob(name="bmi_demo", store=DeltaTableDatabase(workdir / "store"))
with job:
source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice")
compute_bmi.pod(source, label="bmi")

job.run()
job.save(workdir / "bmi_job.json")

loaded = PipelineJob.load(workdir / "bmi_job.json")
print(loaded.nodes["bmi"].as_table().sort_by("subject_id").to_pandas())
# subject_id bmi
# 0 mouse_01 1756.944444
# 1 mouse_02 2210.000000
# 2 mouse_03 2297.520661
```

A loaded job is for reading stored results. It cannot be run again: function pods are not
saved with the job, so `loaded.run()` raises. To compute again, build the job from your code
as before.

- A function node's `as_table()`, `iter_data()` and `get_all_records()` return the rows the
saved run stored, with the types they were stored with. Rows come in the order the store
returns them; sort them if order matters.
- Operators (a join, a filter) store outputs only when caching is enabled, under `odb_v1`.
A loaded operator without a readable cache raises from `as_table()` and `iter_data()`.
Read its input nodes instead (an in-memory input source must be bound first, see below).
Saved descriptors do not yet preserve all runtime identity and cache configuration needed
for operator readback without reconstruction (follow-up ITL-977).
- Sources that read files, such as `CSVSource` and `DeltaTableSource`, are rebuilt from their
paths. In-memory sources (`DictSource`, `ListSource`, `ArrowTableSource`,
`DataFrameSource`) keep their data only in memory, so they are not saved; loading logs
this at INFO level. This does not affect reading the other nodes. To read such a source
node itself, bind a source to it: `loaded.bind(sources={"mice": source})`.
- The results are read from the store recorded in the file. Pass
`PipelineJob.load(path, store=...)` if the store has moved. If the store you give lacks
results the saved run stored, reading the node raises an error naming the store, instead
of returning no rows.

Logical types in saved schema text carry their complete identifier, written as
`extension<name;hex-metadata>`, including inside nested containers. The loader checks the
identifier against the registered type. Known types such as `Path`, `File`, UUID and arrays
keep downstream operators available; a user class must be registered in the loading process.
An unknown type is reported with its node and column, and affected operators become read-only
stubs. An incompatible identifier raises `LogicalTypeIdentifierError` instead of using a stub.
Stored data independently validates its full identifier when values are read.

Older files containing name-only extension type text cannot be loaded by this reader: the
error says the file was saved by an earlier version. Use that version to read the old file,
and save a new job from the original pipeline under this version. Files with only supported
primitive schema types still load; results from pre-v0.3 table layouts remain excluded as
described below. This changes saved schema text only, not stored tables or hashes.

### A store written by an older version

Every row Orcapod stores names the **data context** that wrote it, in the column
Expand Down
4 changes: 3 additions & 1 deletion docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -947,7 +947,9 @@ nodes, caches and cached sources used directly; inside a `PipelineJob` every inp
in a source node of the default data context, so the refusal is not reached there. (What
should eventually happen instead is tracked in ITL-951.) `std:v0.1:default` itself is not shipped
with v0.3 and does not resolve; a pipeline or job file saved by v0.2 loads under the current
data context with a warning.
data context with a warning when its schema types are supported. Saved files with name-only
logical extension types are rejected: current saved schema text requires the full logical
type identifier. See [Saving and loading a job](../concepts/function-pods.md#saving-and-loading-a-job).

`stored_data_contexts(database, record_path)` reads a stored table and counts its rows by
data context. A table without the column reports all its rows under `std:v0.1:default`.
Expand Down
15 changes: 8 additions & 7 deletions examples/save_and_load_pipelines.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Save a pipeline job to JSON, then load it back to inspect its stored results.
"""Save a pipeline job to JSON, then load it back to read its stored results.

Run from any directory: the database and the saved job are written to the
current working directory.
Expand Down Expand Up @@ -30,18 +30,19 @@
# Record the pipeline: nothing is computed inside the ``with`` block.
job = PipelineJob(name="my_pipeline", store=database)
with job:
take_sum.pod(source1.join(source2), label="sum")

Check warning on line 33 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Result of call expression is of type "StreamProtocol" and is not used; assign to variable "_" if this is intentional (reportUnusedCallResult)

job.run()

Check warning on line 35 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Result of call expression is of type "PipelineJob" and is not used; assign to variable "_" if this is intentional (reportUnusedCallResult)
print(job.nodes["sum"].as_table().to_pandas())

Check warning on line 36 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Argument type is Any   Argument corresponds to parameter "values" in function "print" (reportAny)

Check warning on line 36 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "to_pandas" is Any (reportAny)

Check warning on line 36 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "as_table" is Any (reportAny)

# Save the job (pipeline topology and store configuration) to JSON ...
# Save the job (graph, store location and source configuration) to JSON ...
path = "./pipeline.json"
job.save(path)

# ... and load it back, for example in a later session, to inspect the stored
# results. A loaded job is read-only: its nodes read the records in the store
# with get_all_records(), and the job cannot run again. In-memory sources such
# as DictSource are not saved, so loading logs that they were skipped.
# ... and load it back, for example in a later session, to read the stored
# results. A loaded job is for reading: it cannot be run again, because function
# pods are not saved with it. Its function nodes serve the rows the run stored.
# In-memory sources such as DictSource are not saved either (loading logs this at
# INFO level); that does not affect reading the other nodes.
job2 = PipelineJob.load(path)
print(job2.nodes["sum"].get_all_records().to_pandas())
print(job2.nodes["sum"].as_table().sort_by("id").to_pandas())

Check warning on line 48 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Argument type is Any   Argument corresponds to parameter "values" in function "print" (reportAny)

Check warning on line 48 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "to_pandas" is Any (reportAny)

Check warning on line 48 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "sort_by" is Any (reportAny)

Check warning on line 48 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "as_table" is Any (reportAny)
Loading
Loading