diff --git a/changelog.d/ITL-904.md b/changelog.d/ITL-904.md new file mode 100644 index 000000000..4354c4763 --- /dev/null +++ b/changelog.d/ITL-904.md @@ -0,0 +1,48 @@ +### Fixed + +#### A loaded job can read its results (ITL-904) + +A job loaded with `PipelineJob.load` can read its results. A function node's `as_table()` and +`iter_data()` return the rows the saved run stored, with their stored types (they used to raise +"no function pod and no database attached" while `get_all_records()` worked). Without a store, +reading says to pass `PipelineJob.load(path, store=...)` or call `job.bind(store=...)`. `save()` +now records whether the store held records of each function node (`run.node_has_stored_records`, +an optional field; the format version is unchanged). A node saved with records raises, naming +the store, when the attached store holds none of them, instead of returning no rows. When an +entry has several records (it was recomputed), a function node now reads the one with the +greatest recomputation index, on a live node too; it used to keep whichever record the store +returned last. Loading a job in which an operator consumes a function node no longer crashes +with `InputValidationError`, and a loaded node's `output_schema()` reports Python types; a saved +type it cannot restore raises an error naming the column. Registered logical types such as +`Path` restore normally, so downstream operators remain available. Saved +`timestamp`, `duration`, `time` and `decimal` types are now read back. Reading a loaded operator +node without a readable cache raises instead of returning an empty table; operators can store +outputs when caching is enabled. In-memory sources (`DictSource`, `ListSource`, +`ArrowTableSource`, `DataFrameSource`) are +logged at INFO with the remedy instead of a "skipping" warning. A loaded job stays read-only and +cannot be run again. See *Saving and loading a job* in the function-pods guide. (ITL-904) + +Stored-record expectations count only rows of the node's saved data context. Operator +descriptors without a reconstructed operator still lack the saved runtime identity needed +to find cached rows; the runtime-descriptor change is deferred to ITL-977. Direct execution +of loaded operator nodes is rejected. A newer unavailable ephemeral result yields an +`EmptyData` placeholder instead of exposing an older persistent result. + +### Breaking changes and migration + +- Saved pipeline/job schema text now includes the full logical type identifier as + `extension`, including nested logical types. Loading validates it + against the registered type; incompatible or malformed identifiers raise + `LogicalTypeIdentifierError` naming the node and column. Name-only extension text from + older saved files is rejected with an earlier-version error. Read those files with the + version that wrote them, and save a new job from the original pipeline under this version. + Stored table layouts, value decoding and hashes are unchanged. Unregistered logical types + are reported by node and column; affected operators load as read-only stubs. (ITL-904) + +### Changed + +- When persistent and ephemeral results exist for the same input, reads select the + greatest recomputation index across both stores; persistent rows retain precedence + for equal indices. Loaded REPLAY operators report an absent cache table explicitly, + loaded functions reject `run()`, and loaded jobs report `is_runnable()` as false. + Source schema restoration uses the saved data context's type converter. (ITL-904) diff --git a/design_issues/ITL-904-PJLOAD1.md b/design_issues/ITL-904-PJLOAD1.md new file mode 100644 index 000000000..699505c63 --- /dev/null +++ b/design_issues/ITL-904-PJLOAD1.md @@ -0,0 +1,91 @@ +## `src/orcapod/pipeline/job.py` — reading a loaded `PipelineJob` + +### PJLOAD1 — a loaded job could not read its nodes, and could not load a pod feeding an operator +**Status:** resolved +**Severity:** high +**Issue:** ITL-904 + +Saving a job and loading it with `PipelineJob.load` failed in four ways: + +- On a loaded function node, `as_table()` and `iter_data()` raised "no function pod and no + database attached", although the store was attached and `get_all_records()` read the rows. + The cause: `FunctionJobNode.from_descriptor` set `LoadStatus.UNAVAILABLE`, and attaching the + store never changed that status. +- A loaded node's `output_schema()` returned the serialized type strings (`'int64'`), and its + `data_context` dereferenced the missing pod. Loading a job in which an operator (a `Join`) + consumes a function node therefore crashed with `InputValidationError`. +- A loaded operator without a readable cache returned an empty table, as if the operator + had produced nothing. Operators can store outputs when caching is enabled (`odb_v1`). +- Loading logged every source it could not rebuild, in-memory ones included, as "Could not + reconstruct source … — skipping", giving neither the consequence nor a remedy. + +**Fix:** + +- Attaching a store makes a loaded function node `READ_ONLY`. `iter_data()` yields the stored + rows: those whose `_node_content_hash` is the content hash saved in + `run.node_job_content_hashes`, the same set `get_all_records()` returns. `as_table()` is an + `ArrowTableStream` over the stored Arrow records, so stored types are kept and `all_info` + includes system tags, source info and the data context. +- Without a store, reading names `PipelineJob.load(path, store=...)` and `job.bind(store=...)`. +- `save()` records `run.node_has_stored_records`: whether the store held at least one pipeline + record carrying each function node's content hash. This is an additive, optional field, and + the format version is unchanged. A node saved with records raises, naming the store and the + table, when the attached store holds none of them, instead of reading zero rows. Records + that other runs wrote to a shared table do not count. Files saved without the field read + zero rows, as before. +- A loaded node's `output_schema()` returns Python types (`deserialize_output_schema`), and its + data context comes from the descriptor. +- `parse_arrow_type_string` reads the parameterized types it used to reject: `timestamp[us, tz=UTC]`, + `duration`, `time32`, `time64`, `date64` and `decimal128`/`decimal256`. Commas inside `[...]` and + `(...)` no longer split struct fields. +- Logical schema types are saved with the full identifier defined by `arrow_type_identifier`: + `extension`, recursively inside containers. The reader resolves the + registered type and applies `require_readable`. Path, UUID, File, Directory, ndarray, + registered nominal classes and structural classes reconstruct normally. A missing + registration is reported by node and column, leaving affected operators as stubs. + Incompatible identifiers raise `LogicalTypeIdentifierError`; legacy name-only text raises + an earlier-version error. This narrow saved-file change was approved November 12, 2026. +- `get_all_records()` on a loaded node raises the same store errors as `as_table()`. +- Re-saving a loaded job keeps each node's saved expectation of stored records. +- One row per base entry, for `as_table()` and `iter_data()` alike. When an entry has several + records (it was recomputed), the record with the greatest recomputation index wins. This + also changes a live node's hot-load from the store: it used to keep whichever record the + join returned last. +- A loaded node whose results lived in an ephemeral store yields `EmptyData` placeholders from + `iter_data()`. Its `as_table()` raises an error saying the ephemeral store is not saved. + A newer missing ephemeral recomputation takes precedence over an older persistent row. +- A loaded operator node with nothing stored raises, names its input nodes, and says to bind an + unsaved input source first. +- Stored-record expectations constrain both the node content hash and the saved data context. + Rows of another data context cannot satisfy the expectation while reads omit those rows. +- Loading logs in-memory sources at INFO, and any other source or store failure at WARNING + with the exception. Each message states the consequence and the remedy. +- `PipelineJob.run()` rejects every loaded job, including an operator-only one whose + operators were rebuilt: a loaded job is read-only by design (developer, November 11, + 2026). +- Direct operator `run`, `execute`, and `async_execute` also reject loaded nodes; async + rejection closes the output channel. + +Regressions: `tests/test_pipeline/loaded_job/`, which saves in a separate process and loads +in the test process, for each source kind. + +Open, for a follow-up: +- The saved type strings drop nullability (`float | None` is saved as `double`), so a loaded + node reports the non-null type. +- Map type text also omits custom item field names and item nullability; a non-null map + item reloads as nullable. Preserving these fields needs additional schema syntax and is + deferred to the pipeline save/load overhaul, ITL-977 (approved November 12, 2026). +- Unknown classes still require registration in the reader process. + +The hash-step review also found that an operator built directly from a saved blueprint +descriptor cannot locate its stored rows: the descriptor lacks `pipeline_path` and carries +the definition content hash rather than the run's hash. The combined hash-step regression +still injects those values from the live node. The developer deferred this change to ITL-977; +the recommended approach is additive runtime descriptors in `run.node_descriptors`, preserving +blueprint identities and edges. The separate full logical-identifier gap is resolved here. + +Review follow-ups: readable persistent and ephemeral records now both reach the +latest-index selection (persistent wins equal indices). Loaded REPLAY operators raise +when their cache table is absent; loaded function `run()` rejects computation, and +loaded jobs report `is_runnable()` as false even after binding sources. Source-schema +restoration uses the saved context converter in both normal and fallback branches. diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 06a48dec6..5d828ea0d 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -376,6 +376,72 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro This is the contract for now. An opt-in mode that tolerates store failures may be added later. +### Saving and loading a job + +`job.save(path)` writes the job to a JSON file: the graph, the store's location and how each +source was configured. `PipelineJob.load(path)` reads it back, for example in a later Python +session, so that you can read the results the job stored: + +```python +import tempfile +from pathlib import Path + +from orcapod.databases import DeltaTableDatabase + +workdir = Path(tempfile.mkdtemp()) # any directory you keep + +job = PipelineJob(name="bmi_demo", store=DeltaTableDatabase(workdir / "store")) +with job: + source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + compute_bmi.pod(source, label="bmi") + +job.run() +job.save(workdir / "bmi_job.json") + +loaded = PipelineJob.load(workdir / "bmi_job.json") +print(loaded.nodes["bmi"].as_table().sort_by("subject_id").to_pandas()) +# subject_id bmi +# 0 mouse_01 1756.944444 +# 1 mouse_02 2210.000000 +# 2 mouse_03 2297.520661 +``` + +A loaded job is for reading stored results. It cannot be run again: function pods are not +saved with the job, so `loaded.run()` raises. To compute again, build the job from your code +as before. + +- A function node's `as_table()`, `iter_data()` and `get_all_records()` return the rows the + saved run stored, with the types they were stored with. Rows come in the order the store + returns them; sort them if order matters. +- Operators (a join, a filter) store outputs only when caching is enabled, under `odb_v1`. + A loaded operator without a readable cache raises from `as_table()` and `iter_data()`. + Read its input nodes instead (an in-memory input source must be bound first, see below). + Saved descriptors do not yet preserve all runtime identity and cache configuration needed + for operator readback without reconstruction (follow-up ITL-977). +- Sources that read files, such as `CSVSource` and `DeltaTableSource`, are rebuilt from their + paths. In-memory sources (`DictSource`, `ListSource`, `ArrowTableSource`, + `DataFrameSource`) keep their data only in memory, so they are not saved; loading logs + this at INFO level. This does not affect reading the other nodes. To read such a source + node itself, bind a source to it: `loaded.bind(sources={"mice": source})`. +- The results are read from the store recorded in the file. Pass + `PipelineJob.load(path, store=...)` if the store has moved. If the store you give lacks + results the saved run stored, reading the node raises an error naming the store, instead + of returning no rows. + +Logical types in saved schema text carry their complete identifier, written as +`extension`, including inside nested containers. The loader checks the +identifier against the registered type. Known types such as `Path`, `File`, UUID and arrays +keep downstream operators available; a user class must be registered in the loading process. +An unknown type is reported with its node and column, and affected operators become read-only +stubs. An incompatible identifier raises `LogicalTypeIdentifierError` instead of using a stub. +Stored data independently validates its full identifier when values are read. + +Older files containing name-only extension type text cannot be loaded by this reader: the +error says the file was saved by an earlier version. Use that version to read the old file, +and save a new job from the original pipeline under this version. Files with only supported +primitive schema types still load; results from pre-v0.3 table layouts remain excluded as +described below. This changes saved schema text only, not stored tables or hashes. + ### A store written by an older version Every row Orcapod stores names the **data context** that wrote it, in the column diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index e3ef8a1ab..b44a8320b 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -947,7 +947,9 @@ nodes, caches and cached sources used directly; inside a `PipelineJob` every inp in a source node of the default data context, so the refusal is not reached there. (What should eventually happen instead is tracked in ITL-951.) `std:v0.1:default` itself is not shipped with v0.3 and does not resolve; a pipeline or job file saved by v0.2 loads under the current -data context with a warning. +data context with a warning when its schema types are supported. Saved files with name-only +logical extension types are rejected: current saved schema text requires the full logical +type identifier. See [Saving and loading a job](../concepts/function-pods.md#saving-and-loading-a-job). `stored_data_contexts(database, record_path)` reads a stored table and counts its rows by data context. A table without the column reports all its rows under `std:v0.1:default`. diff --git a/examples/save_and_load_pipelines.py b/examples/save_and_load_pipelines.py index 46bfffeae..69f67bab3 100644 --- a/examples/save_and_load_pipelines.py +++ b/examples/save_and_load_pipelines.py @@ -1,4 +1,4 @@ -"""Save a pipeline job to JSON, then load it back to inspect its stored results. +"""Save a pipeline job to JSON, then load it back to read its stored results. Run from any directory: the database and the saved job are written to the current working directory. @@ -35,13 +35,14 @@ def take_sum(x: int, y: int) -> int: job.run() print(job.nodes["sum"].as_table().to_pandas()) -# Save the job (pipeline topology and store configuration) to JSON ... +# Save the job (graph, store location and source configuration) to JSON ... path = "./pipeline.json" job.save(path) -# ... and load it back, for example in a later session, to inspect the stored -# results. A loaded job is read-only: its nodes read the records in the store -# with get_all_records(), and the job cannot run again. In-memory sources such -# as DictSource are not saved, so loading logs that they were skipped. +# ... and load it back, for example in a later session, to read the stored +# results. A loaded job is for reading: it cannot be run again, because function +# pods are not saved with it. Its function nodes serve the rows the run stored. +# In-memory sources such as DictSource are not saved either (loading logs this at +# INFO level); that does not affect reading the other nodes. job2 = PipelineJob.load(path) -print(job2.nodes["sum"].get_all_records().to_pandas()) +print(job2.nodes["sum"].as_table().sort_by("id").to_pandas()) diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 078021ed7..888d0b0df 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -335,11 +335,34 @@ def output_schema( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> tuple[Schema, Schema]: - """Return output schema, using stored value in read-only mode.""" + """Return output schema, using stored value in read-only mode. + + Raises: + ValueError: If the node was loaded and a column's saved type cannot be + restored to a Python type (an unregistered logical type). + """ if self._function_pod is None: - tag = Schema(self._stored_schema.get("tag", {})) - data = Schema(self._stored_schema.get("data", {})) - return tag, data + from orcapod.pipeline.serialization import ( + deserialize_output_schema, + SavedSchemaVersionError, + ) + from orcapod.logical_types.identifier import LogicalTypeIdentifierError + + try: + return deserialize_output_schema( + self._stored_schema, + self.data_context.type_converter, + ) + except (LogicalTypeIdentifierError, SavedSchemaVersionError) as exc: + exc.args = ( + f"Cannot restore the output schema of loaded node {self.label!r}: {exc}", + ) + raise + except ValueError as exc: + raise ValueError( + f"Cannot restore the output schema of loaded node {self.label!r}: " + f"{exc}" + ) from exc tag_schema = self._input_stream.output_schema( columns=columns, all_info=all_info )[0] @@ -877,6 +900,9 @@ def __init__( # .node_config setter or PipelineJob.apply_node_config(). self._node_config: NodeConfig = NodeConfig() + # Set by from_descriptor() for a loaded node whose saved run had stored + # records; a live node computes its own results. + self._expects_stored_records: bool = False # Batched commits, set only while PipelineJob.run runs the node (ITL-919). self._commit_every_rows: int | None = None self._rows_since_commit = 0 @@ -1082,6 +1108,11 @@ def attach_databases( result_database=result_database, record_path=self._stored_result_record_path, ) + # With a store attached, the stub can serve the results its saved + # run stored, though it still cannot compute. + from orcapod.pipeline.serialization import LoadStatus + + self._load_status = LoadStatus.READ_ONLY self._pipeline_database = pipeline_database @@ -1105,6 +1136,7 @@ def from_descriptor( input_stream: StreamProtocol | None = None, databases: dict[str, Any] | None = None, job_content_hash: str | None = None, + expect_stored_records: bool = False, ) -> "FunctionJobNode": """Create a read-only ``FunctionJobNode`` stub from a serialized descriptor. @@ -1131,6 +1163,11 @@ def from_descriptor( column written to the DB at execution time. Required for ``get_all_records()`` to correctly filter DB rows when ``table_scope="pipeline_hash"`` (the default). + expect_stored_records: Whether the store held at least one pipeline + record of this node (one with this node's content hash) when the + job was saved. When ``True``, reading the node from a store with + none of them raises instead of returning zero rows, because the + store is not the one the job ran against. Returns: A ``FunctionJobNode`` in read-only stub mode @@ -1229,6 +1266,7 @@ def from_descriptor( # FunctionJobNode — orchestrator config node._node_config = NodeConfig() + node._expects_stored_records = expect_stored_records node._commit_every_rows = None node._rows_since_commit = 0 node._commit_failed = False @@ -1237,6 +1275,12 @@ def from_descriptor( node._pipeline_database = None node._cached_function_pod = None + # FunctionJobNode — ephemeral result store (same defaults as __init__). + # A loaded node has no function pod, so it never gets an ephemeral pod; + # _fetch_joined_records reads it to resolve ephemeral pipeline records. + node.ephemeral_result_store = None + node._ephemeral_cached_pod = None + # Wire databases if provided pipeline_db = (databases or {}).get("pipeline") result_db = (databases or {}).get("result") @@ -1302,6 +1346,94 @@ def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: # Internal helpers # ------------------------------------------------------------------ + def _no_store_error(self) -> RuntimeError: + """Return the error for reading a loaded node that has no store attached. + + Only a node loaded by ``PipelineJob.load`` can be ``UNAVAILABLE``: it has no + function pod, so its results can come only from the store. + """ + return RuntimeError( + f"FunctionJobNode {self.label!r} was loaded without a store, so its stored " + "results cannot be read. Pass the store to PipelineJob.load(path, " + "store=...) or call job.bind(store=...)." + ) + + def _cannot_compute_error(self) -> RuntimeError: + """Return the error for asking a loaded node to compute. + + A node loaded by ``PipelineJob.load`` has no function pod: it can serve + the results its saved run stored, never compute new ones. + """ + return RuntimeError( + f"FunctionJobNode {self.label!r} was loaded by PipelineJob.load and has no " + "function pod, so it cannot compute; read its stored results with " + "as_table() or iter_data()." + ) + + def _has_stored_records(self) -> bool: + """Return whether the attached store holds a pipeline record of this node. + + A record belongs to this node when its ``_node_content_hash`` is this + node's content hash: the rows ``iter_data()`` and ``get_all_records()`` + read back. Records of other runs or data contexts do not count. + """ + if self._pipeline_database is None: + return False + if not self._pipeline_database.table_exists(self._versioned_pipeline_path): + return False + own_records = self._pipeline_database.get_records_with_column_value( + self._versioned_pipeline_path, + { + constants.NODE_CONTENT_HASH_COL: self.content_hash().to_prefixed_digest(), + constants.DATA_CONTEXT_COL: self._stored_data_context_key, + }, + ) + return own_records is not None and own_records.num_rows > 0 + + def _require_readable_stub(self) -> None: + """Raise unless this loaded node can read its stored results. + + Raises: + RuntimeError: If no store is attached, or the saved run stored records + that the attached store does not have. + """ + from orcapod.pipeline.serialization import LoadStatus + + if self.load_status == LoadStatus.UNAVAILABLE: + raise self._no_store_error() + self._require_expected_records() + + def _require_expected_records(self) -> None: + """Raise if the saved run stored this node's records but the store has none. + + A loaded node reads only what its store holds. When the job was saved + with records of this node in the store and the attached store has none + of them, the store is not the one the job ran against; reading zero rows + would make the stored results look lost. + + Raises: + RuntimeError: If the saved run stored records for this node and the + attached store has none of them. + """ + if not self._expects_stored_records or self._has_stored_records(): + return + assert self._pipeline_database is not None # attached: status is READ_ONLY + store_config = self._pipeline_database.to_config() + location = store_config.get("root_uri") + store = ( + f"store at {location!r}" + if location + else f"{store_config.get('type', 'attached')} store" + ) + table = "/".join( + self._pipeline_database.base_path + self._versioned_pipeline_path + ) + raise RuntimeError( + f"FunctionJobNode {self.label!r}: the saved run stored results for this " + f"node, but the attached {store} has none of them (table {table!r}). Load " + "the job with the store it ran against: PipelineJob.load(path, store=...)." + ) + def _require_pipeline_database(self) -> ArrowDatabaseProtocol: """Return the attached pipeline database, raising if there is none. @@ -1546,12 +1678,17 @@ def execute( ``None`` outputs and failed data. Raises: + RuntimeError: If the node was loaded by ``PipelineJob.load`` and so + has no function pod to compute with. InputValidationError: If the input stream is of another data context than this node; raised before any row is processed, whatever the error policy. """ from orcapod.pipeline.observer import NoOpObserver + if self._function_pod is None: + raise self._cannot_compute_error() + self._begin_run() # A whole-input condition, not a row failure: the error policy does not apply. require_same_data_context( @@ -2213,7 +2350,14 @@ def get_all_records( A ``pa.Table`` of joined results, or ``None`` if no database is attached, either DB fetch returns ``None``, or the join produces no rows. + + Raises: + RuntimeError: If the node was loaded by ``PipelineJob.load`` and no + store is attached, or its saved run stored records that the + attached store does not have. """ + if self._function_pod is None: + self._require_readable_stub() fetched = self._fetch_joined_records() if fetched is None: return None @@ -2460,6 +2604,13 @@ def _fetch_joined_records( unmatched_df = ephemeral_taginfo_df for row in unmatched_df.iter_rows(named=True): base_eid = row[_PIPELINE_BASE_ENTRY_ID_COL] + previous = empty_taginfo_rows.get(base_eid) + if ( + previous is not None + and previous[_PIPELINE_RECOMPUTATION_INDEX_COL] + > row[_PIPELINE_RECOMPUTATION_INDEX_COL] + ): + continue # Use OUTPUT_DATA_HASH_COL so that the downstream result cache # lookup in _process_data_internal finds the correct entry. # The downstream's result cache is keyed by the INPUT to the @@ -2486,15 +2637,11 @@ def _fetch_joined_records( empty_taginfo_rows[base_eid] = row # ------------------------------------------------------------------ - # Merge with persistent priority (anti-join + concat) + # Keep both stores' rows until latest-recomputation selection. Persistent + # rows come last so they retain priority when recomputation indices tie. # ------------------------------------------------------------------ if ephemeral_df.height > 0 and persistent_df.height > 0: - ephemeral_only_df = ephemeral_df.join( - persistent_df.select([_PIPELINE_BASE_ENTRY_ID_COL]), - on=_PIPELINE_BASE_ENTRY_ID_COL, - how="anti", - ) - merged_df = pl.concat([persistent_df, ephemeral_only_df], how="diagonal") + merged_df = pl.concat([ephemeral_df, persistent_df], how="diagonal") elif ephemeral_df.height > 0: merged_df = ephemeral_df elif persistent_df.height > 0: @@ -2532,6 +2679,82 @@ def _fetch_joined_records( empty_taginfo_rows=empty_taginfo_rows, ) + def _stored_tag_keys(self, fetched: _JoinedRecords) -> tuple[str, ...]: + """Return the user tag columns of the records in *fetched*. + + Prefers the input stream's tag keys; falls back to excluding system columns + from the pipeline records for CACHE_ONLY / deserialized nodes. + """ + if self._input_stream is not None: + return self._input_stream.keys()[0] + return tuple( + c + for c in fetched.taginfo_columns + if not c.startswith(constants.META_PREFIX) + and not c.startswith(constants.SOURCE_PREFIX) + and not c.startswith(constants.SYSTEM_TAG_PREFIX) + and c != _PIPELINE_ENTRY_ID_COL + and c != constants.NODE_CONTENT_HASH_COL + ) + + def _stored_rows_stream( + self, fetched: _JoinedRecords + ) -> tuple[list[bytes], ArrowTableStream] | None: + """Return the persistent stored rows of *fetched* as a stream. + + The stream is built over the stored Arrow records with the internal + columns dropped, so every column keeps the type it was stored with. A + base entry with several records contributes only the one with the + greatest recomputation index. + + Returns: + The base entry ID of each row, in stream order, and the stream; or + ``None`` when no persistent row is stored. + """ + joined = fetched.table + if joined.num_rows == 0: + return None + all_base_entry_ids = cast( + list[bytes], joined.column(_PIPELINE_BASE_ENTRY_ID_COL).to_pylist() + ) + # One row per base entry ID: when a base entry has several records (it + # was recomputed), the one with the greatest recomputation index -- the + # latest result -- wins, in the position where the base entry first + # appears. Ties keep the later row. + recomputation_indices = cast( + list[int], joined.column(_PIPELINE_RECOMPUTATION_INDEX_COL).to_pylist() + ) + chosen_row: dict[bytes, int] = {} + for row, base_entry_id in enumerate(all_base_entry_ids): + current = chosen_row.get(base_entry_id) + if ( + current is None + or recomputation_indices[row] >= recomputation_indices[current] + ): + chosen_row[base_entry_id] = row + chosen_row = { + base_eid: row + for base_eid, row in chosen_row.items() + if base_eid not in fetched.empty_taginfo_rows + or recomputation_indices[row] + >= fetched.empty_taginfo_rows[base_eid][_PIPELINE_RECOMPUTATION_INDEX_COL] + } + if not chosen_row: + return None + base_entry_ids = list(chosen_row) + if len(base_entry_ids) < joined.num_rows: + joined = joined.take(pa.array(list(chosen_row.values()), type=pa.int64())) + drop_cols = [ + c + for c in joined.column_names + if c.startswith(constants.META_PREFIX) + or c == constants.NODE_CONTENT_HASH_COL + ] + stream = ArrowTableStream( + joined.drop(drop_cols), tag_columns=self._stored_tag_keys(fetched) + ) + return base_entry_ids, stream + def _load_cached_entries( self, base_entry_ids: list[bytes] | None = None, @@ -2566,43 +2789,20 @@ def _load_cached_entries( if fetched.table.num_rows == 0 and not fetched.empty_data_tokens: return {} - joined = fetched.table - - # Derive tag keys: prefer input_stream when available; fall back to - # taginfo column exclusion for CACHE_ONLY / deserialized nodes. - if self._input_stream is not None: - tag_keys = self._input_stream.keys()[0] - else: - tag_keys = tuple( - c - for c in fetched.taginfo_columns - if not c.startswith(constants.META_PREFIX) - and not c.startswith(constants.SOURCE_PREFIX) - and not c.startswith(constants.SYSTEM_TAG_PREFIX) - and c != _PIPELINE_ENTRY_ID_COL - and c != constants.NODE_CONTENT_HASH_COL - ) - + tag_keys = self._stored_tag_keys(fetched) loaded: dict[bytes, tuple[TagProtocol, DataProtocol]] = {} # --- Process normal (fully matched) rows --- - if joined.num_rows > 0: - base_entry_ids_col = joined.column(_PIPELINE_BASE_ENTRY_ID_COL).to_pylist() - drop_cols = [ - c - for c in joined.column_names - if c.startswith(constants.META_PREFIX) - or c == constants.NODE_CONTENT_HASH_COL - ] - data_table = joined.drop([c for c in drop_cols if c in joined.column_names]) - stream = ArrowTableStream(data_table, tag_columns=tag_keys) + stored = self._stored_rows_stream(fetched) + if stored is not None: + base_entry_ids_col, stream = stored for base_eid, (tag, data) in zip(base_entry_ids_col, stream.iter_data()): loaded[base_eid] = (tag, data) # --- Process EmptyData tokens (ephemeral misses) --- for base_eid, empty_data in fetched.empty_data_tokens.items(): if base_eid in loaded: - # A persistent result exists for the same entry — it wins. + # A matched result at least as recent as the missing one wins. continue raw_row = fetched.empty_taginfo_rows[base_eid] tag_data = {k: v for k, v in raw_row.items() if k in tag_keys} @@ -2655,6 +2855,44 @@ async def _async_execute_cache_only( # Iteration # ------------------------------------------------------------------ + def as_table( + self, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> "pa.Table": + """Return this node's results as a PyArrow table. + + A node loaded by ``PipelineJob.load`` has no function pod. Its table is + built from the Arrow records its saved run stored, so every column keeps + its stored type. Row order follows the store. With nothing stored, the + table has zero rows and the node's columns. + + Raises: + RuntimeError: If the node was loaded and no store is attached, its + saved run stored records the attached store does not have, or + some of its results were kept in an ephemeral store, which is + not saved with the job. + """ + if self._function_pod is None: + self._require_readable_stub() + fetched = self._fetch_joined_records() + if fetched is None: + return super().as_table(columns=columns, all_info=all_info) + stored = self._stored_rows_stream(fetched) + stored_ids = set(stored[0]) if stored is not None else set() + ephemeral = [k for k in fetched.empty_data_tokens if k not in stored_ids] + if ephemeral: + raise RuntimeError( + f"FunctionJobNode {self.label!r}: the saved run kept " + f"{len(ephemeral)} of its results in an ephemeral store, which " + "is not saved with the job, so they cannot be read into a " + "table. iter_data() yields them as EmptyData placeholders." + ) + if stored is not None: + return stored[1].as_table(columns=columns, all_info=all_info) + return super().as_table(columns=columns, all_info=all_info) + def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Yield all computed (tag, data) pairs for this node. @@ -2671,10 +2909,7 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: status = self.load_status if status == LoadStatus.UNAVAILABLE: - raise RuntimeError( - f"FunctionJobNode {self.label!r} is unavailable: " - "no function pod and no database attached." - ) + raise self._no_store_error() if status == LoadStatus.CACHE_ONLY: # Upstream unavailable; serve entirely from DB. @@ -2696,6 +2931,9 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: if self.is_stale: self.clear_cache() + if self._function_pod is None: + # A loaded node: its rows can come only from the store. + self._require_expected_records() if not self._cached_output_datas and self._cached_function_pod is not None: # Hot-load from DB on the first call when store is empty. loaded = self._load_cached_entries() @@ -2714,16 +2952,16 @@ def run(self) -> None: """Eagerly compute all input data, filling pipeline and result databases. Raises: - RuntimeError: If ``load_status`` is UNAVAILABLE (no pod, no DB). + RuntimeError: If the node was loaded without a function pod, or + ``load_status`` is UNAVAILABLE (no pod, no DB). """ from orcapod.pipeline.serialization import LoadStatus + if self._function_pod is None: + raise self._cannot_compute_error() self._begin_run() if self._load_status == LoadStatus.UNAVAILABLE: - raise RuntimeError( - f"FunctionJobNode {self.label!r} is unavailable: " - "no function pod and no database attached." - ) + raise self._no_store_error() if self._load_status in (LoadStatus.CACHE_ONLY, LoadStatus.READ_ONLY): # CACHE_ONLY: upstream unavailable; computation requires a live input stream. # READ_ONLY: function pod is a proxy placeholder — cannot compute. @@ -2763,6 +3001,11 @@ async def async_execute( observer: Optional execution observer for hooks. run_id: Optional pipeline run identifier threaded through to ``InvocationContext.pipeline_run_id``. + + Raises: + RuntimeError: If the node was loaded by ``PipelineJob.load``: without a + store it has nothing to serve, and it has no function pod to + compute with. *output* is closed first. """ from orcapod.pipeline.serialization import LoadStatus @@ -2779,10 +3022,11 @@ async def async_execute( if status == LoadStatus.UNAVAILABLE: await output.close() - raise RuntimeError( - f"FunctionJobNode {self.label!r} is unavailable: " - "no function pod and no database attached." - ) + raise self._no_store_error() + + if self._function_pod is None: + await output.close() + raise self._cannot_compute_error() from orcapod.pipeline.observer import NoOpObserver diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index b8bf9457e..36dcc906e 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -234,11 +234,34 @@ def output_schema( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> tuple[Schema, Schema]: - """Return output schema, using stored value in read-only mode.""" + """Return output schema, using stored value in read-only mode. + + Raises: + ValueError: If the node was loaded and a column's saved type cannot be + restored to a Python type (an unregistered logical type). + """ if self._operator is None: - tag = Schema(self._stored_schema.get("tag", {})) - data = Schema(self._stored_schema.get("data", {})) - return tag, data + from orcapod.pipeline.serialization import ( + deserialize_output_schema, + SavedSchemaVersionError, + ) + from orcapod.logical_types.identifier import LogicalTypeIdentifierError + + try: + return deserialize_output_schema( + self._stored_schema, + self.data_context.type_converter, + ) + except (LogicalTypeIdentifierError, SavedSchemaVersionError) as exc: + exc.args = ( + f"Cannot restore the output schema of loaded node {self.label!r}: {exc}", + ) + raise + except ValueError as exc: + raise ValueError( + f"Cannot restore the output schema of loaded node {self.label!r}: " + f"{exc}" + ) from exc return self._operator.output_schema( *self._input_streams, columns=columns, @@ -612,6 +635,10 @@ def __init__( self._pipeline_database: ArrowDatabaseProtocol | None = None self._cache_mode = CacheMode.OFF + # True for a node rebuilt by ``from_descriptor`` (a job loaded by + # ``PipelineJob.load``): its output was computed by an earlier run. + self._loaded = False + if pipeline_database is not None: self.attach_databases( pipeline_database=pipeline_database, @@ -748,6 +775,7 @@ def from_descriptor( ) node._descriptor = descriptor node._load_status = LoadStatus.FULL + node._loaded = True return node # Read-only mode: bypass __init__, set minimum required state @@ -781,12 +809,13 @@ def from_descriptor( # From OperatorNodeBase node._operator = None - node._input_streams = () + node._input_streams = tuple(input_streams) node.tracker_manager = DEFAULT_TRACKER_MANAGER # From OperatorJobNode node._cached_output_stream = None node._cached_output_table = None + node._loaded = True # DB persistence state node._pipeline_database = pipeline_db @@ -817,6 +846,28 @@ def from_descriptor( # Internal helpers # ------------------------------------------------------------------ + def _no_stored_results_error(self) -> RuntimeError: + """Return the error for reading a loaded operator node with nothing to serve. + + Operator outputs are stored only when caching is enabled. A loaded node + needs a readable cache; returning an empty table when none is available + would look like the operator produced nothing. + """ + inputs = ", ".join(repr(stream.label) for stream in self._input_streams) + return RuntimeError( + f"OperatorJobNode {self.label!r} has no stored results available for this read. " + "Operator outputs require a stored cache and replay configuration. Read the " + f"stored results of its input nodes ({inputs}) instead; an input source " + "whose data was not saved must be bound first with job.bind(sources=...)." + ) + + def _cannot_compute_error(self) -> RuntimeError: + """Describe why this loaded node cannot execute an operator.""" + return RuntimeError( + f"OperatorJobNode {self.label!r} was loaded by PipelineJob.load and " + "cannot compute; create a fresh PipelineJob to run the pipeline again." + ) + def _filter_by_content_hash(self, table: pa.Table) -> pa.Table: """Filter *table* to rows whose ``NODE_CONTENT_HASH_COL`` matches this node. @@ -1002,6 +1053,9 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": - Returns ``None`` if ``_pipeline_database is None``. - Returns ``None`` if ``_cache_mode != CacheMode.REPLAY`` (LOG/OFF modes may have stale historical records in the DB). + + Raises: + RuntimeError: If a loaded REPLAY node's cache table is absent. """ if self._pipeline_database is None: return None @@ -1009,6 +1063,12 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": return None records = self._read_stored_records() if records is None: + if self._loaded: + raise RuntimeError( + f"OperatorJobNode {self.label!r}: the cache table is absent " + f"({'/'.join(self._stored_table_path)}). Load with the store " + "that holds the saved operator results." + ) if self._operator is None: # Read-only (deserialized) node with no operator: cannot derive # schema without a live operator. Return None so callers fall @@ -1061,6 +1121,9 @@ def execute( """ from orcapod.pipeline.observer import NoOpObserver + if self._loaded: + raise self._cannot_compute_error() + node_label = self.label node_hash = self.content_hash().to_string() @@ -1152,6 +1215,8 @@ def run(self) -> None: - **LOG**: always compute, write results to DB. - **REPLAY**: skip computation, load from DB. """ + if self._loaded: + raise self._cannot_compute_error() if self.is_stale: self.clear_cache() @@ -1178,12 +1243,16 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: ``CacheMode.REPLAY`` with a database yields the rows stored there; in any other mode it yields nothing. Call ``node.is_stale`` before iterating if you need to detect outdated cached data. + + Loaded nodes raise when no readable stored cache is available. """ if self._cached_output_stream is not None: return self._cached_output_stream.iter_data() db_stream = self._load_cached_stream_from_db() if db_stream is not None: return db_stream.iter_data() + if self._loaded: + raise self._no_stored_results_error() return iter([]) def as_table( @@ -1199,6 +1268,8 @@ def as_table( memory (before a run, or after ``clear_cache()``), a node in ``CacheMode.REPLAY`` with a database returns the rows stored there; in any other mode it returns a zero-row table with the node's columns. + + Loaded nodes raise when no readable stored cache is available. """ if self._cached_output_stream is not None: return self._cached_output_stream.as_table( @@ -1207,6 +1278,8 @@ def as_table( db_stream = self._load_cached_stream_from_db() if db_stream is not None: return db_stream.as_table(columns=columns, all_info=all_info) + if self._loaded: + raise self._no_stored_results_error() # No cached or stored records yet: construct a zero-row table with the # correct schema and route it through ArrowTableStream so that the # ColumnConfig / all_info shaping is applied consistently. @@ -1345,6 +1418,10 @@ async def async_execute( """ from orcapod.pipeline.observer import NoOpObserver + if self._loaded: + await output.close() + raise self._cannot_compute_error() + obs = observer if observer is not None else NoOpObserver() node_label = self.label node_hash = self.content_hash().to_string() diff --git a/src/orcapod/pipeline/graph.py b/src/orcapod/pipeline/graph.py index 28a86289e..3bc2f386d 100644 --- a/src/orcapod/pipeline/graph.py +++ b/src/orcapod/pipeline/graph.py @@ -287,6 +287,31 @@ def load(cls, path: str | Path) -> "Pipeline": if descriptor is None: continue + from orcapod.logical_types.identifier import LogicalTypeIdentifierError + from orcapod.pipeline.serialization import ( + SavedSchemaVersionError, + deserialize_output_schema, + ) + + # Validate even a leaf node's saved identifiers before exposing the job. + context = resolve_saved_context(descriptor.get("data_context_key")) + try: + deserialize_output_schema( + descriptor.get("output_schema", {}), context.type_converter + ) + except (LogicalTypeIdentifierError, SavedSchemaVersionError) as exc: + exc.args = ( + f"Cannot restore the output schema of loaded node " + f"{descriptor.get('label')!r}: {exc}", + ) + raise + except ValueError as exc: + logger.warning( + "Cannot restore the output schema of loaded node %r: %s", + descriptor.get("label"), + exc, + ) + node_type = descriptor.get("node_type") source_config = descriptor.get("source_config") or {} @@ -300,17 +325,27 @@ def load(cls, path: str | Path) -> "Pipeline": node_name = descriptor.get("label") or "unknown" if "tag_schema" in source_config and "data_schema" in source_config: tag_schema = Schema( - deserialize_schema(source_config["tag_schema"]) + deserialize_schema( + source_config["tag_schema"], context.type_converter + ) ) data_schema = Schema( - deserialize_schema(source_config["data_schema"]) + deserialize_schema( + source_config["data_schema"], context.type_converter + ) ) else: tag_schema = Schema( - deserialize_schema(descriptor["output_schema"]["tag"]) + deserialize_schema( + descriptor["output_schema"]["tag"], + context.type_converter, + ) ) data_schema = Schema( - deserialize_schema(descriptor["output_schema"]["data"]) + deserialize_schema( + descriptor["output_schema"]["data"], + context.type_converter, + ) ) node = SourceNodeClass( name=node_name, @@ -365,6 +400,25 @@ def load(cls, path: str | Path) -> "Pipeline": op_config.get("class_name"), exc, ) + if operator is not None: + from orcapod.logical_types.identifier import ( + LogicalTypeIdentifierError, + ) + from orcapod.pipeline.serialization import SavedSchemaVersionError + + try: + for upstream in upstream_nodes: + upstream.output_schema() + except (LogicalTypeIdentifierError, SavedSchemaVersionError): + raise + except ValueError as exc: + logger.warning( + "Cannot reconstruct operator node %r because an input " + "schema cannot be restored; keeping a read-only stub: %s", + descriptor.get("label"), + exc, + ) + operator = None node = OperatorNode.from_descriptor( descriptor, operator=operator, diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 9fde57aa5..1a5e8a5cf 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -106,6 +106,8 @@ def __init__( self._sources_by_hash: dict[str, cp.StreamProtocol] = {} self._has_run: bool = False self._run_id: str | None = None + # True for a job built by load(): it reads stored results and never runs. + self._loaded: bool = False # Lazy cache for the corresponding Pipeline blueprint. # Set to None after compile() to force recomputation on next access. @@ -715,7 +717,9 @@ def is_complete(self) -> bool: return self._store is not None and not self.unbound_sources def is_runnable(self, node_label: str) -> bool: - """Return ``True`` if all upstream inputs of *node_label* are resolved. + """Return whether a live job has all upstream inputs of *node_label* resolved. + + A loaded job is read-only and always returns ``False``. Uses hash-based traversal of ``_hash_graph`` combined with ``_persistent_node_map`` to check whether every upstream source slot @@ -729,7 +733,7 @@ def is_runnable(self, node_label: str) -> bool: Returns: ``True`` if the node can be executed with current bindings. """ - if not self._compiled: + if self._loaded or not self._compiled: return False target = self._nodes.get(node_label) @@ -821,7 +825,8 @@ def run( Raises: ValueError: If no store is set, or ``commit_every_rows`` is not a positive integer smaller than the store's ``batch_size``. - RuntimeError: If no pipeline has been compiled. + RuntimeError: If no pipeline has been compiled, or the job was built + by ``PipelineJob.load`` (a loaded job is read-only). StoreWriteError: If a node's results or records could not be written to the store, whatever the orchestrator's ``error_policy``. The run stops at the first failed write. Writes that succeeded before it @@ -855,6 +860,16 @@ def run( "PipelineJob has no compiled pipeline. " "Record invocations inside a 'with job:' block first." ) + # A loaded job reads the results its saved run stored and never runs: + # function pods are not saved with it, and even an operator-only loaded + # job, whose operators are rebuilt, stays read-only. + if self._loaded: + raise RuntimeError( + "PipelineJob.run() cannot execute a loaded job: PipelineJob.load() " + "is for reading stored results only. To run a new computation, " + "create a fresh PipelineJob from the original pipeline definition." + ) + table_committing = as_table_committing(self._store) if ( table_committing is not None @@ -882,11 +897,16 @@ def run( # inspect results. from orcapod.pipeline.serialization import LoadStatus + # A loaded function node has no function pod, whatever its load status + # (READ_ONLY once a store is attached, so it can read its stored results). unavailable_labels = [ node.label for node in self._persistent_node_map.values() - if isinstance(node, (FunctionJobNode, OperatorJobNode)) - and node.load_status == LoadStatus.UNAVAILABLE + if ( + isinstance(node, (FunctionJobNode, OperatorJobNode)) + and node.load_status == LoadStatus.UNAVAILABLE + ) + or (isinstance(node, FunctionJobNode) and node.producer is None) ] if unavailable_labels: raise RuntimeError( @@ -1096,9 +1116,20 @@ def save(self, path: "str | Path") -> None: # stubs whose content_hash() matches the _node_content_hash column in the # DB — required for _filter_by_content_hash() in get_all_records(). node_job_content_hashes: dict[str, str] = {} + # Whether the store holds at least one pipeline record of each function + # node (a record with the node's content hash), so that load() can tell a + # store that holds this run's results from one that does not (a + # different store passed to load(), or a moved one). + node_has_stored_records: dict[str, bool] = {} for node in (self._persistent_node_map or {}).values(): if isinstance(node, FunctionJobNode): node_job_content_hashes[node.label] = node.content_hash().to_string() + # A loaded node keeps the expectation it was saved with: a store + # attached since then that lacks the table does not mean the + # original run stored nothing. + node_has_stored_records[node.label] = ( + node._expects_stored_records or node._has_stored_records() + ) output: dict[str, Any] = { "orcapod_pipeline_job_version": PIPELINE_JOB_FORMAT_VERSION, @@ -1107,6 +1138,7 @@ def save(self, path: "str | Path") -> None: "status": status, "unbound_sources": list(self.unbound_sources), "node_job_content_hashes": node_job_content_hashes, + "node_has_stored_records": node_has_stored_records, }, "pipeline": pipeline_data, "bindings": { @@ -1124,11 +1156,21 @@ def load( path: "str | Path", store: "ArrowDatabaseProtocol | None" = None, ) -> "PipelineJob": - """Deserialize a ``PipelineJob`` from a JSON file. - - The embedded ``Pipeline`` blueprint is always restored. Concrete source - bindings are restored for reconstructable source types. Pass *store* - explicitly to override any serialized store configuration. + """Deserialize a ``PipelineJob`` from a JSON file to read its stored results. + + A loaded job is for reading the results its saved run stored; it cannot be + run again, because function pods are not saved with the job. Read a + function node with ``job.nodes[label].as_table()``, ``iter_data()`` or + ``get_all_records()``. Operators can store outputs when caching is enabled; + reading a loaded operator without a readable cache raises ``RuntimeError``. + + The embedded ``Pipeline`` blueprint is always restored. Sources whose data + lives in files (CSV, Delta) are rebuilt from the saved config. In-memory + sources (``DictSource``, ``ListSource``, ``ArrowTableSource``, + ``DataFrameSource``) are not saved; loading logs that at INFO, and their + source nodes can be read after ``job.bind(sources=...)``. A source that + cannot be rebuilt for another reason (a moved file) is logged as a + warning. Neither affects reading the other nodes. Args: path: Path to the JSON file produced by ``save()``. @@ -1136,8 +1178,7 @@ def load( over any store configuration in the file. Returns: - A ``PipelineJob`` ready to run (call ``bind()`` if sources are - missing). + A read-only ``PipelineJob`` whose nodes serve their stored results. Raises: ValueError: If the file's format version is unsupported. @@ -1178,13 +1219,35 @@ def load( for spec_name, src_config in ( data.get("bindings", {}).get("sources", {}).items() ): + source_type = src_config.get("source_type") try: source = resolve_source_from_config(src_config) if source is not None: sources[spec_name] = source - except Exception: + except NotImplementedError: + # In-memory sources (DictSource, ListSource, ArrowTableSource, + # DataFrameSource) are not saved with the job, by design. + logger.info( + "Source %r (%s) keeps its data in memory and is not saved with the " + "job; the loaded job has no data for it. Stored function-node " + "results are still readable from the store. To read source node " + "%r itself, bind a source: job.bind(sources={%r: }).", + spec_name, + source_type, + spec_name, + spec_name, + ) + except Exception as exc: logger.warning( - "Could not reconstruct source %r from config — skipping.", spec_name + "Could not reconstruct source %r (%s) from its saved config: %s. " + "Stored function-node results are still readable from the store. " + "To read source node %r itself, bind a source: " + "job.bind(sources={%r: }).", + spec_name, + source_type, + exc, + spec_name, + spec_name, ) # Reconstruct store @@ -1198,9 +1261,12 @@ def load( ) effective_store = resolve_database_from_config(store_config) - except Exception: + except Exception as exc: logger.warning( - "Could not reconstruct store from config — skipping." + "Could not reconstruct the store from its saved config: %s. " + "Node results cannot be read until a store is given: " + "PipelineJob.load(path, store=...) or job.bind(store=...).", + exc, ) # Reconstruct the job directly without going through to_invocations(). @@ -1209,6 +1275,7 @@ def load( # path used by from_pipeline() would fail. Instead, attach the loaded # pipeline directly as the compiled blueprint and set up source bindings. job = cls(name=pipeline._name, store=effective_store) + job._loaded = True job._compiled_pipeline = pipeline # Mark compiled so save() / compiled_pipeline can be accessed. job._compiled = True @@ -1243,6 +1310,11 @@ def load( node_job_content_hashes: dict[str, str] = run_block.get( "node_job_content_hashes", {} ) + # Absent in files saved before ITL-904: such nodes read zero rows from a + # store that has none, as before. + node_has_stored_records: dict[str, bool] = run_block.get( + "node_has_stored_records", {} + ) for node_hash in _nx.topological_sort(job._hash_graph): bp_node = pipeline._persistent_node_map.get(node_hash) @@ -1273,6 +1345,9 @@ def load( descriptor, input_stream=input_stream, job_content_hash=live_content_hash, + expect_stored_records=node_has_stored_records.get( + bp_node.label, False + ), ) job._persistent_node_map[node_hash] = fjn diff --git a/src/orcapod/pipeline/serialization.py b/src/orcapod/pipeline/serialization.py index f12737f20..cb4653b72 100644 --- a/src/orcapod/pipeline/serialization.py +++ b/src/orcapod/pipeline/serialization.py @@ -8,6 +8,14 @@ from enum import Enum from typing import TYPE_CHECKING, Any +from orcapod.types import Schema +from orcapod.logical_types.identifier import ( + LogicalTypeIdentifier, + LogicalTypeIdentifierError, + arrow_type_identifier, + require_readable, +) + if TYPE_CHECKING: from orcapod.protocols.database_protocols import DatabaseRegistryProtocol @@ -582,14 +590,53 @@ def register_data_function(type_id: str, cls: type) -> None: # --------------------------------------------------------------------------- +class SavedSchemaVersionError(ValueError): + """A saved schema predates complete logical type identifiers.""" + + +def _serialize_arrow_type(arrow_type: Any) -> str: + """Render a schema type, preserving complete logical identifiers at every depth.""" + import pyarrow as pa + + if isinstance(arrow_type, pa.ExtensionType): + return f"extension<{arrow_type_identifier(arrow_type).text}>" + if pa.types.is_list(arrow_type) or pa.types.is_large_list(arrow_type): + field = arrow_type.value_field + child = _serialize_arrow_type(field.type) + nullable = "" if field.nullable else " not null" + outer = "large_list" if pa.types.is_large_list(arrow_type) else "list" + return f"{outer}<{field.name}: {child}{nullable}>" + if pa.types.is_fixed_size_list(arrow_type): + field = arrow_type.value_field + nullable = "" if field.nullable else " not null" + return ( + f"fixed_size_list<{field.name}: {_serialize_arrow_type(field.type)}" + f"{nullable}>[{arrow_type.list_size}]" + ) + if pa.types.is_struct(arrow_type): + fields = [ + f"{field.name}: {_serialize_arrow_type(field.type)}" + + ("" if field.nullable else " not null") + for field in arrow_type + ] + return "struct<" + ", ".join(fields) + ">" + if pa.types.is_map(arrow_type): + ordered = ", keys_sorted" if arrow_type.keys_sorted else "" + return ( + f"map<{_serialize_arrow_type(arrow_type.key_type)}, " + f"{_serialize_arrow_type(arrow_type.item_type)}{ordered}>" + ) + return str(arrow_type) + + def serialize_schema(schema: Any, type_converter: Any | None = None) -> dict[str, str]: """Convert a Schema mapping to JSON-serializable Arrow type strings. The result contains human-readable Arrow type strings for each field (e.g. ``"int64"``, ``"large_string"``, ``"list"``). - These strings follow Arrow's canonical format and can be parsed back - by `deserialize_schema` in any language that implements the - Arrow type grammar. + Primitive/container strings follow Arrow's format. Logical types use + ``extension`` with the canonical full identifier, including + nested positions; the reader validates it against the registered type. Args: schema: A Schema-like mapping from field name to data type. @@ -606,11 +653,16 @@ def serialize_schema(schema: Any, type_converter: Any | None = None) -> dict[str for k, v in schema.items(): try: arrow_type = type_converter.python_type_to_arrow_type(v) - result[k] = str(arrow_type) + result[k] = _serialize_arrow_type(arrow_type) except Exception: result[k] = str(v) return result - return {k: str(v) for k, v in schema.items()} + import pyarrow as pa + + return { + k: _serialize_arrow_type(v) if isinstance(v, pa.DataType) else str(v) + for k, v in schema.items() + } def deserialize_schema( @@ -642,14 +694,60 @@ def deserialize_schema( result: dict[str, Any] = {} for name, type_str in schema_dict.items(): try: - arrow_type = parse_arrow_type_string(type_str) + arrow_type = parse_arrow_type_string(type_str, type_converter) result[name] = type_converter.arrow_type_to_python_type(arrow_type) + except LogicalTypeIdentifierError as exc: + raise exc.for_column(name) from exc + except SavedSchemaVersionError: + raise except Exception: result[name] = type_str return result -def parse_arrow_type_string(type_str: str) -> Any: +def deserialize_output_schema( + output_schema: dict[str, dict[str, str]], + type_converter: Any, +) -> tuple[Schema, Schema]: + """Reconstruct a node's ``(tag, data)`` output schema from its saved descriptor. + + Args: + output_schema: The descriptor's ``output_schema`` entry: ``{"tag": {...}, + "data": {...}}``, each mapping a column name to an Arrow type string as + written by `serialize_schema`. + type_converter: The node's type converter, used to turn each Arrow type back + into the Python type that ``output_schema()`` reports. + + Returns: + The tag schema and the data schema, with Python types. + + Raises: + ValueError: If a column's saved type cannot be restored to a Python type, + such as an unregistered logical type. The message names + the column and its saved type string. + """ + + def restore(columns: dict[str, str]) -> Schema: + restored: dict[str, Any] = {} + for name, type_str in columns.items(): + try: + arrow_type = parse_arrow_type_string(type_str, type_converter) + restored[name] = type_converter.arrow_type_to_python_type(arrow_type) + except LogicalTypeIdentifierError as exc: + raise exc.for_column(name) from exc + except SavedSchemaVersionError as exc: + raise SavedSchemaVersionError(f"Column {name!r}: {exc}") from exc + except Exception as exc: + raise ValueError( + f"Column {name!r} has the saved type {type_str!r}, which cannot " + "be restored to a Python type." + ) from exc + return Schema(restored) + + return restore(output_schema.get("tag", {})), restore(output_schema.get("data", {})) + + +def parse_arrow_type_string(type_str: str, type_converter: Any | None = None) -> Any: """Parse an Arrow type string into a ``pa.DataType``. Handles both primitive types (``"int64"``, ``"large_string"``) and @@ -660,6 +758,8 @@ def parse_arrow_type_string(type_str: str) -> Any: Args: type_str: Arrow type string to parse. + type_converter: The saved node's converter for resolving logical identifiers. + If omitted, logical types use the default context's converter. Returns: The corresponding ``pa.DataType``. @@ -671,11 +771,57 @@ def parse_arrow_type_string(type_str: str) -> Any: type_str = type_str.strip() + if type_str.startswith("fixed_size_list<"): + import re + + match = re.fullmatch(r"fixed_size_list<(.*)>\[(\d+)\]", type_str) + if match is None: + raise ValueError(f"Malformed fixed-size list: {type_str!r}") + return pa.list_( + _parse_arrow_field(match.group(1), type_converter), int(match.group(2)) + ) + + if type_str.startswith("extension<"): + if not type_str.endswith(">"): + raise LogicalTypeIdentifierError(LogicalTypeIdentifier(type_str, b""), ()) + identifier = type_str[len("extension<") : -1] + name, separator, metadata_text = identifier.partition(";") + if not name: + raise LogicalTypeIdentifierError( + LogicalTypeIdentifier(name, metadata_text.encode()), () + ) + if not separator: + raise SavedSchemaVersionError( + "This file was saved by an earlier version without a full logical " + "type identifier; read it with that version and save it again with " + "the original pipeline under the current version." + ) + try: + metadata = bytes.fromhex(metadata_text) + except ValueError as exc: + error = LogicalTypeIdentifierError( + LogicalTypeIdentifier(name, metadata_text.encode()), () + ) + error.add_note("Saved logical type metadata is not hexadecimal.") + raise error from exc + if type_converter is None: + from orcapod.contexts import resolve_context + + type_converter = resolve_context(None).type_converter + logical_type = type_converter.get_logical_type_by_arrow_extension_name(name) + if logical_type is None: + raise ValueError(f"Unregistered extension type: {name!r}") + require_readable(logical_type, metadata) + return logical_type.get_arrow_extension_type() + # Primitive types — try direct lookup via pa.() if "<" not in type_str: factory = _get_arrow_primitive_types().get(type_str) if factory is not None: return factory() + parameterized = _parse_parameterized_type(type_str) + if parameterized is not None: + return parameterized raise ValueError(f"Unknown Arrow type: {type_str!r}") # Nested types — parse the outer type and recurse @@ -685,8 +831,7 @@ def parse_arrow_type_string(type_str: str) -> Any: if outer in ("list", "large_list"): # "list" or "list>" - child_type_str = _strip_field_name(inner) - child_type = parse_arrow_type_string(child_type_str) + child_type = _parse_arrow_field(inner, type_converter) return ( pa.large_list(child_type) if outer == "large_list" else pa.list_(child_type) ) @@ -694,28 +839,84 @@ def parse_arrow_type_string(type_str: str) -> Any: if outer == "struct": # "struct" — split on top-level commas fields = _split_struct_fields(inner) - pa_fields = [] - for field_str in fields: - colon_pos = field_str.index(":") - field_name = field_str[:colon_pos].strip() - field_type_str = field_str[colon_pos + 1 :].strip() - pa_fields.append( - pa.field(field_name, parse_arrow_type_string(field_type_str)) - ) + pa_fields = [ + _parse_arrow_field(field_str, type_converter) for field_str in fields + ] return pa.struct(pa_fields) if outer == "map": # "map" — split on first top-level comma - parts = _split_top_level(inner, ",", max_splits=1) + parts = _split_top_level(inner, ",") + keys_sorted = len(parts) == 3 and parts[-1].strip() == "keys_sorted" + if keys_sorted: + parts = parts[:-1] if len(parts) != 2: raise ValueError(f"Cannot parse map type: {type_str!r}") - key_type = parse_arrow_type_string(parts[0].strip()) - value_type = parse_arrow_type_string(parts[1].strip()) - return pa.map_(key_type, value_type) + key_type = parse_arrow_type_string(parts[0].strip(), type_converter) + value_type = parse_arrow_type_string(parts[1].strip(), type_converter) + if keys_sorted: + return pa.map_(key_type, value_type, True) + return pa.map_(key_type, value_type, False) raise ValueError(f"Unknown nested Arrow type: {type_str!r}") +def _parse_arrow_field(text: str, type_converter: Any | None) -> Any: + """Restore a nested field's type and nullability from schema text.""" + import pyarrow as pa + + name, separator, type_text = text.strip().partition(":") + if not separator: + name, type_text = "item", text + type_text = type_text.strip() + nullable = not type_text.endswith(" not null") + if not nullable: + type_text = type_text[: -len(" not null")] + return pa.field( + name.strip(), + parse_arrow_type_string(type_text, type_converter), + nullable=nullable, + ) + + +def _parse_parameterized_type(type_str: str) -> Any | None: + """Parse a parameterized non-nested Arrow type string, or return ``None``. + + Handles the canonical ``str()`` forms of ``timestamp[us]``, + ``timestamp[us, tz=UTC]``, ``duration[us]``, ``time32[ms]``, + ``time64[us]``, ``date64[ms]``, ``decimal128(10, 2)`` and + ``decimal256(40, 2)``. + + Args: + type_str: Arrow type string, stripped of surrounding whitespace. + + Returns: + The corresponding ``pa.DataType``, or ``None`` if *type_str* is not one + of these forms. + """ + import re + from typing import Literal, cast + + import pyarrow as pa + + # Each regex admits only the units Arrow accepts, so the casts below hold. + if match := re.fullmatch(r"timestamp\[(s|ms|us|ns)(?:, tz=(.+))?\]", type_str): + unit = cast(Literal["s", "ms", "us", "ns"], match.group(1)) + return pa.timestamp(unit, tz=match.group(2)) + if match := re.fullmatch(r"duration\[(s|ms|us|ns)\]", type_str): + return pa.duration(cast(Literal["s", "ms", "us", "ns"], match.group(1))) + if match := re.fullmatch(r"time32\[(s|ms)\]", type_str): + return pa.time32(cast(Literal["s", "ms"], match.group(1))) + if match := re.fullmatch(r"time64\[(us|ns)\]", type_str): + return pa.time64(cast(Literal["us", "ns"], match.group(1))) + if type_str == "date64[ms]": + return pa.date64() + if match := re.fullmatch(r"decimal(128|256)\((\d+), (-?\d+)\)", type_str): + factory = pa.decimal128 if match.group(1) == "128" else pa.decimal256 + return factory(int(match.group(2)), int(match.group(3))) + return None + + def _strip_field_name(s: str) -> str: """Strip the ``"item: "`` or ``"field_name: "`` prefix from a child type string.""" if ":" in s: @@ -728,7 +929,10 @@ def _strip_field_name(s: str) -> str: def _split_top_level(s: str, sep: str, max_splits: int = -1) -> list[str]: - """Split *s* on *sep*, but only at the top level (not inside ``<>``). + """Split *s* on *sep*, but only at the top level (not inside ``<>``, ``[]`` or ``()``). + + Brackets and parentheses count because parameterized types carry commas, + e.g. ``timestamp[us, tz=UTC]`` and ``decimal128(10, 2)``. Args: s: String to split. @@ -742,9 +946,9 @@ def _split_top_level(s: str, sep: str, max_splits: int = -1) -> list[str]: depth = 0 current: list[str] = [] for ch in s: - if ch == "<": + if ch in "<[(": depth += 1 - elif ch == ">": + elif ch in ">])": depth -= 1 if ch == sep and depth == 0 and (max_splits == -1 or len(parts) < max_splits): parts.append("".join(current)) diff --git a/superpowers/plans/2026-11-11-itl-904-loaded-job-readback.md b/superpowers/plans/2026-11-11-itl-904-loaded-job-readback.md new file mode 100644 index 000000000..162eae426 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-904-loaded-job-readback.md @@ -0,0 +1,237 @@ +# ITL-904 Loaded PipelineJob read-back Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** On a job loaded with `PipelineJob.load`, reading a node with `as_table()` or +`iter_data()` returns its stored rows, or raises an error that says what is wrong and what to +do. Loading never crashes on a valid saved job, and loading logs a clear message for each +source it cannot rebuild. + +**Architecture:** A loaded function node is a stub with no function pod. It reads its +records through `_ResultDatabaseReader` once the store is attached. The work: +- `attach_databases` marks the stub readable (`LoadStatus.READ_ONLY`). +- The stub's `as_table()` builds an `ArrowTableStream` over the stored Arrow records. +- A stub's `output_schema()` and `data_context` come from the descriptor. +- A loaded operator node that has nothing stored raises. +- `PipelineJob.load` logs each source it cannot rebuild, with the consequence and the remedy. + +**Tech Stack:** Python 3.11+, PyArrow, Polars, Delta Lake (`DeltaTableDatabase`), pytest. + +**Linear:** ITL-904. **Branch:** `eywalker/itl-904-a-loaded-pipelinejob-cannot-read-its-nodes`. + +--- + +## Coordinator decisions (November 11, 2026) + +Approved contract (coordinator, answering the step-1 ask): + +- **A.** A loaded function node with a store attached serves the stored rows. A row is + stored for the node when its pipeline record carries `_node_content_hash` equal to the + live content hash saved in `run.node_job_content_hashes`. This is the same set + `get_all_records()` returns. + - `iter_data()` yields the stored `(tag, data)` pairs. + - `as_table()` is built from the stored Arrow records, honours `ColumnConfig`/`all_info`, + and re-infers no types. + - `load_status` is `READ_ONLY` while a store is attached and `UNAVAILABLE` without one. + - Nothing stored: zero rows with the node's schema. + - No store: `RuntimeError("FunctionJobNode 'dbl' was loaded without a store, so its stored + results cannot be read. Pass the store to PipelineJob.load(path, store=...) or call + job.bind(store=...).")` + - `PipelineJob.run()` still rejects loaded stubs. +- **B.** A stub's `output_schema()` returns deserialized Python types, not serialized + strings. This fixes `PipelineJob.load` crashing on pod-then-join. The `data_context` + AttributeError on stubs is fixed as well. +- **C.** A loaded operator node with no stored output raises: "OperatorJobNode 'joined' has + no stored results: a PipelineJob does not store operator outputs, so a loaded job cannot + show them. Read the stored results of its input nodes ('dbl', 'asrc') instead." It never + returns an empty table. Live, un-run operators keep returning empty. +- **D.** Source messages on load: + - INFO for in-memory kinds (`NotImplementedError`). + - WARNING with the exception for other failures. + - WARNING for a store that cannot be rebuilt. + - The exact wording is in Task 5. +- Mismatched-store detection, option (1): `run.node_has_stored_records`. This is an + additive, optional saved-file field and needs the **developer's** approval. Prepare it as + a separate, self-contained commit on top and **do not push it** until the coordinator + confirms. The base behaviour is (3): zero rows. +- Docs: add a short executed section "Saving and loading a job" in + `docs/concepts/function-pods.md` (in the PipelineJob part, after the store section). +- Example: edit main's `examples/save_and_load_pipelines.py` now; reconcile with PR #296 + when it lands. +- No saved-format, stored-schema or hash change in the base PR. The golden suite must pass + unchanged. +- CLAUDE.md's stale line "Phase 1 returns ALL records": propose a correction in the PR and + ask the coordinator before editing CLAUDE.md or `.zed/rules`. + +## Root causes (step 1) + +1. `FunctionJobNode.from_descriptor` (`src/orcapod/core/nodes/function_node.py`) sets + `LoadStatus.UNAVAILABLE`. `attach_databases` wires a `_ResultDatabaseReader` but never + updates the status, and `iter_data`, `run` and `async_execute` raise a hard-coded + "no function pod and no database attached". +2. `FunctionNodeBase.output_schema()` and `OperatorNodeBase.output_schema()` return + `Schema(self._stored_schema[...])`, whose values are serialized strings (`'int64'`). + `Pipeline.load` builds a full-mode operator over a function stub, and `Join` validation + then fails with "Types and int64 are not compatible". +3. The `data_context` of a stub dereferences `_function_pod` / `_operator`, which is `None`. +4. `PipelineJob` runs operators with `cache_mode=OFF`, so nothing is stored for them, and the + loaded node returns an empty table. +5. `PipelineJob.load` swallows every source reconstruction error into one WARNING. + In-memory sources are not serializable by design. + +## File map + +- Modify `src/orcapod/core/nodes/function_node.py`: + - `FunctionNodeBase.output_schema` and `data_context` for stubs. + - `FunctionJobNode.attach_databases` sets READ_ONLY. + - `iter_data` / `run` / `async_execute` use the new UNAVAILABLE message. + - New `as_table` override for stubs, plus a shared `_stored_records_stream` helper + used by `_load_cached_entries`. +- Modify `src/orcapod/core/nodes/operator_node.py`: + - Stub `output_schema` and `data_context`. + - A loaded marker; `iter_data` / `as_table` raise for a loaded operator with nothing to + serve. + - Keep `_input_streams` on read-only stubs. +- Modify `src/orcapod/pipeline/job.py`: + - Source and store log messages in `load`. + - The `run()` guard also rejects READ_ONLY function stubs. + - The `load` docstring. +- Create `tests/test_pipeline/test_loaded_job_readback.py`. It saves in a subprocess (fresh + process) and loads in the test process; it covers every source kind and reads each node. +- Modify `docs/concepts/function-pods.md`, `examples/save_and_load_pipelines.py`, + `CHANGELOG.md` and `DESIGN_ISSUES.md`. + +## Tasks + +### Task 1: Failing tests (public interface) + +`tests/test_pipeline/test_loaded_job_readback.py`. A module-scoped fixture writes a save +script into `tmp_path` and runs it with `subprocess.run([sys.executable, script, dir])`. +The script builds a job with: +- `DictSource` -> pod `dbl` -> `Join` with an `ArrowTableSource` (`joined`): the + pod-then-join shape; +- `CSVSource` -> pod `inc`; +- `DeltaTableSource` -> pod `half`; +- `ListSource` -> pod `sq`; +- a `Delta` store. + +It runs the job, writes each node's `as_table()` (sorted by tag) to parquet as the expected +values, then saves the job. Tests: + +- `test_load_does_not_crash_on_pod_then_join` +- `test_function_node_as_table_returns_stored_rows[dbl|inc|half|sq]`: equal to the live + table, sorted by tag; same schema. +- `test_function_node_as_table_all_info_matches_live`: same column set as live, all_info. +- `test_function_node_iter_data_yields_stored_rows` +- `test_function_node_load_status_read_only_with_store` +- `test_function_node_without_store_raises_clear_error`: load with a store that cannot be + rebuilt (edit the JSON store block to an unknown type) -> the message names + `PipelineJob.load(path, store=...)` and never says "no database attached". +- `test_function_node_nothing_stored_returns_zero_rows`: load with an empty + `InMemoryArrowDatabase` as `store=` -> zero rows, columns = tag + data. +- `test_loaded_operator_node_raises_no_stored_results` (`as_table` and `iter_data`). +- `test_loaded_job_run_still_rejected` +- `test_in_memory_sources_logged_at_info_with_remedy` (caplog): the message contains + `job.bind(sources={'dsrc': })`; no WARNING for in-memory kinds. +- `test_unreconstructable_file_source_logged_at_warning_with_reason`: delete the CSV + before loading. +- `test_csv_and_delta_source_nodes_readable_after_load` +- `test_in_memory_source_node_readable_after_bind` +- `test_stub_output_schema_is_python_types` + +Run `uv run pytest tests/test_pipeline/test_loaded_job_readback.py -v` -> expect failures. +Commit as a WIP test commit. + +### Task 2: Stub `output_schema` and `data_context` (B) + +In `FunctionNodeBase.output_schema` (stub branch), use +`deserialize_schema(self._stored_schema.get("tag", {}), type_converter=self.data_context.type_converter)`, +and the same for data. `data_context` returns `self._data_context` when +`_function_pod is None`; `from_descriptor` already sets `_data_context`. Make the same +change in `OperatorNodeBase` (`_operator is None`). Run the pod-then-join and schema tests. + +### Task 3: Function stub read-back (A) + +- `attach_databases`, stub path: after creating the `_ResultDatabaseReader`, set + `self._load_status = LoadStatus.READ_ONLY`. +- `_stored_records_stream(fetched) -> ArrowTableStream | None`: drop the `META_PREFIX` + columns and `NODE_CONTENT_HASH_COL`, then build + `ArrowTableStream(table, tag_columns=tag_keys)`. `_load_cached_entries` uses it (no + duplication). +- `FunctionJobNode.as_table` override: if `self._function_pod is None`, require + readability (the UNAVAILABLE error). If stored rows exist, return + `stream.as_table(columns=columns, all_info=all_info)` (cached). Otherwise fall back to + `super().as_table(...)`, which gives a zero-row table from `output_schema`. +- UNAVAILABLE message in `iter_data`, `run` and `async_execute`: the approved wording + (one private helper that builds the error). +- `PipelineJob.run()` guard: also list `FunctionJobNode`s whose `_function_pod is None`. + +### Task 4: Loaded operator raises (C) + +- `OperatorJobNode.from_descriptor`: set `node._loaded = True` on both paths (`False` in + `__init__`); keep `node._input_streams = tuple(input_streams)` on the read-only path. +- `iter_data` / `as_table`: when there is no cached stream and no DB stream, and + `self._loaded` is true, raise `RuntimeError` with the approved message. Input labels come + from `self._input_streams` (`label` of each). + +### Task 5: Load messages (D) + +In `PipelineJob.load`: + +```python +except NotImplementedError: + logger.info( + "Source %r (%s) keeps its data in memory and is not saved with the job; the " + "loaded job has no data for it. Stored results of all nodes are still readable. " + "To read source node %r itself, bind a source: job.bind(sources={%r: }).", + spec_name, source_type, spec_name, spec_name, + ) +except Exception as exc: + logger.warning( + "Could not reconstruct source %r (%s) from its saved config: %s. Stored results " + "of all nodes are still readable. To read source node %r itself, bind a source: " + "job.bind(sources={%r: }).", + spec_name, source_type, exc, spec_name, spec_name, + ) +``` + +The store message: "Could not reconstruct the store from its saved config: %s. Node results +cannot be read until a store is given: PipelineJob.load(path, store=...) or +job.bind(store=...)." The docstring says a loaded job is for reading stored results and +cannot be run again. + +### Task 6: Docs, example, CHANGELOG, DESIGN_ISSUES + +- `docs/concepts/function-pods.md`: add "### Saving and loading a job" with an executed + sample (`DeltaTableDatabase` in a temporary directory, save, `PipelineJob.load`, + `as_table()` printed sorted by tag). +- `examples/save_and_load_pipelines.py`: a `PipelineJob` with a Delta store; run, save, + load, print `as_table()`. +- CHANGELOG `[0.3.0] — unreleased` -> `### Fixed` entry. +- DESIGN_ISSUES.md: one appended entry, status resolved, with a **Fix:** note. + +### Task 7: Verification + +`uv run --with ruff ruff check .`, `ruff format --check .`, +`uv run --with basedpyright basedpyright` (exit code) and `uv run pytest tests/` (including +the golden and doc tests). Merge origin/main, push, open the PR (Fixes ITL-904), run +`orca linear attach` and `orca worktree set`, then do the CodeRabbit rounds. + +### Task 8 (separate, unpushed until confirmed): option (1) + +- `save()`: `run.node_has_stored_records[label] = pipeline_db.table_exists(node._versioned_pipeline_path)`. +- `load()`: pass the flag to the stub. If the flag is true and the attached store has no + table at the node's path, reading raises an error naming the store path. +- Test: an older file without the field still loads and gives zero rows. + +## Progress notes + +- [x] Step 1 investigation and ask; approved November 11, 2026. +- [x] Tasks 1–6 and 8 done and pushed. The developer approved option (1) (additive + `run.node_has_stored_records`), so it is in the PR. +- [x] Coordinator decision: the loaded `as_table(all_info=True)` follows the stream semantics + (`_context_key`, `_source_*` included). The live `FunctionJobNode.as_table()` drops those + columns; that is a separate bug, for the coordinator to file. +- [x] Merged origin/main (PR #296 landed). Conflicts in the example and in DESIGN_ISSUES.md + are resolved: both entries are kept. +- [ ] Task 7: full suite, PR, CodeRabbit. diff --git a/superpowers/plans/2026-11-12-itl-904-refresh.md b/superpowers/plans/2026-11-12-itl-904-refresh.md new file mode 100644 index 000000000..a7d04141e --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-904-refresh.md @@ -0,0 +1,91 @@ +# ITL-904 refresh assessment and implementation plan + +> Execute inline with sensei:executing-plans; coordinator approval gates apply before the real merge and any saved-format decision. + +**Goal:** Refresh PR #303 against current main, preserve stored rows and hash identities, and obtain CI and CodeRabbit review on the final head. +**Architecture:** Retain loaded function readback while integrating main's context filtering and versioned layouts. Investigate operator descriptor identity separately before selecting a saved-format solution. +**Tech Stack:** Python, uv, PyArrow, Delta, pytest, ruff, basedpyright. + +- [x] Read ITL-904 and both comments; recover the real PR branch from the inherited scratch merge. +- [x] Fetch main and PR history; main is b46549fd, original PR head is 10421990. +- [x] Assess main using PR loaded_job regressions with main library code; record exact failures. +- [x] Dry-run merge on scratch branch, record each library conflict and proposed resolution; ask coordinator before real merge. +- [x] Investigate graph.save versus run identity and descriptor loading; present stored-schema choices before implementation. +- [x] After approval, merge main (never rebase); retain context filtering and current layouts; migrate own changelog and design entries. +- [x] Write failing-first regressions for the approved scope; retain hash-step injection because runtime descriptors were deferred to ITL-977. +- [x] Run targeted loaded-job, context, layout, and unchanged golden tests. +- [x] Run full tests sequentially on Python 3.12, 3.11, 3.13; run ruff, format, basedpyright and both documentation-fragment checks. +- [ ] Check main again, update accurate PR body, commit and push once for the round. +- [ ] Inspect every review body and GraphQL review thread on final SHA; route nontrivial revisions through coordinator. +- [ ] Report reviewed SHA, resolved files, golden report, skips/failures and gaps with exactly one worker_done. + +The checkpoints below record the successive approvals and their superseded proposals. Final approved scope is recorded under "Final approved logical-schema scope". Existing implementation decisions from November 11 remain in the original plan; no existing commit is discarded. + +## Takeover checkpoint — November 12 + +- Coordinator confirmed exclusive ownership after stopping the other session. The inherited scratch merge is a fresh dry run, not the original interrupted one. +- Fetch confirmed `origin/main` and `MERGE_HEAD` both equal `b46549fd6e13059be795141684436a2e27e2771b`; `HEAD` remains `10421990d124298e0ab76afc25af7b23c73e56e6`. +- A fresh `git merge-tree --write-tree HEAD origin/main` confirms the same five conflicts: `DESIGN_ISSUES.md`, `docs/concepts/function-pods.md`, `src/orcapod/core/nodes/function_node.py`, `src/orcapod/core/nodes/operator_node.py`, `src/orcapod/pipeline/job.py`. +- Main baseline is archived under `.untracked/itl904-main` with the PR's `tests/test_pipeline/loaded_job` copied in. `SETUPTOOLS_SCM_PRETEND_VERSION=0.3.0 uv run --python 3.12 pytest tests/test_pipeline/loaded_job/ -q --tb=short` produces **39 failures in 21.12s**; complete log `.untracked/itl904-main-tests.log`. The original schema-restoration load crash remains (`int64` versus Python `int`). The version override is necessary only because the archive has no Git metadata. +- Linear ITL-904 and both comments read through Codex Linear tools. No status changes. Full GitHub history saved under `.untracked/pr303*.json` for inspection. +- Merge resolution proposal sent through Orca ask: combine batching and stored-record state, keep main context fallback docstrings, guard missing pods before computation setup, combine loaded-job run guard with batch validation. Preserve current layouts and context filtering. +- Saved-format decision pending: recommend separate additive runtime descriptors in the job's run block, retaining blueprint descriptors and edges unchanged; alternatives are runtime fields within blueprint descriptors or deferral. No schema decision or implementation made. +- No golden values changed by this worker. No library file edited, merge completed, commit made, or push performed. + +## Approved merge execution + +Coordinator approved the five-file merge resolutions. Saved-format changes remain deferred pending developer decision; retain the combined hash-step test injection and document the gap. + +Real merge started on the original PR branch after aborting the dry run and confirming `git pull --ff-only` is current. Conflict resolutions preserve main's context filtering and batching. Migrated PJ-LOAD1 and changelog text to per-entry files. Initial merged targeted run: **145 passed**. + +Contained integration fix: `_has_stored_records()` must constrain both `NODE_CONTENT_HASH_COL` and `DATA_CONTEXT_COL` to `self._stored_data_context_key`; otherwise foreign-context rows incorrectly satisfy the saved-record expectation. Added public save/load regression across all three read APIs, preserving field types while copying foreign-context records through database APIs. All three fail before the fix (no exception); the corrected operator error wording also fails before updating its message. Run log: `.untracked/context-red.log` (**4 failed, 38 passed**). Next: add the context lookup constraint, update inaccurate operator wording in code/docs, then rerun targeted and full verification. + +## Existing review findings + +Coordinator approved fixing two unanswered findings on the original PR head: latest unavailable ephemeral result must outrank an older persistent result; loaded operators must reject run/execute/async_execute. Five failing regressions (including the Path/Join investigation) are in `.untracked/review-red.log` (**5 failed, 42 passed**). + +- In `_fetch_joined_records`, retain the greatest recomputation index when multiple missing ephemeral rows share a base entry. +- In `_stored_rows_stream`, omit a selected persistent row if its recomputation index is lower than the corresponding missing ephemeral row. Existing placeholder processing then serves `EmptyData`, and table reads report the unavailable result. +- Guard all three operator execution entry points when `_loaded`; close async output before raising. Use a no-argument error factory that describes this node. +- Path/Join root cause: operator reconstruction validates upstream schemas, and serialized Path strings cannot be restored. Proposed no-format-change fix sent through ask: preflight upstream schema restoration during blueprint loading, falling back to an operator-less stub when schema restoration raises. Await approval for this third fix. +- Preliminary full 3.12 run interrupted for approved revisions after 2331 passed / 13 xfailed; not a completed full-suite result. Typecheck exit 0. Lint and format passed. Fragment validator requires entry id `PJLOAD1` rather than the legacy hyphenated `PJ-LOAD1`; migrated entry renamed accordingly. + +## Final approved logical-schema scope + +The coordinator relayed developer approval replacing the interim registered-name solution: saved logical schema text now uses the existing canonical `extension` identifier recursively, validates metadata through `require_readable`, and rejects legacy name-only text with an earlier-version error. This is the only approved saved-file change; no runtime descriptor fields, hash identities, table layouts, or golden values change. The saved runtime operator descriptor gap and the hash-step test injection remain deferred to ITL-977. Unknown classes produce contextual warnings and an operator stub; malformed/incompatible identifiers raise with node and column context. Primitive v0.2 saved jobs retain main's context warning behavior. + +Failing-first evidence: full-identifier regressions initially **10 failed, 1 passed**; standalone-node validation **2 failed, 11 passed**; recursive schema preservation **3 failed, 98 passed**. Final loaded-job/serialization/descriptor targeted suite: **173 passed** (`.untracked/schema-final-targeted.log`). Coverage includes Path, UUID, File, Directory, ndarray, registered dataclass and op.dataclass, Join reconstruction, metadata corruption, legacy schema text, and an unregistered class from a separate process. Broad golden/operator/descriptor run: **2269 passed, 11 xfailed**; golden files equal main. Main primitive v0.2 compatibility baseline: **4 passed**. + +The first completed Python 3.12 full run produced **9439 passed, 101 skipped, 19 xfailed, 5 errors**. All five errors shared the packaging build fixture: sandbox DNS prevented fetching hatch-vcs. Escalated `uv build` succeeded and populated the local cache; rerunning the full suite. Typecheck identified PyArrow's constrained literal boolean map argument, now passed as explicit literal branches. A read-only reviewer could not start because Orca rejected nested depth before creating any worker; local review plus final-head CodeRabbit review remains required. + +Merge remains in progress on the original PR branch; no commit or push yet. Updated changes also include context-aware stored-record expectations, latest-ephemeral selection, and loaded operator execution guards. Changelog/design/docs describe approved current behavior. Scratch and logs remain under `.untracked/`. + +Coordinator approved deferring map field names/item nullability to ITL-977; documented beside the existing nullability gap without extending saved syntax. Final malformed-identifier audit added missing-delimiter and empty-name cases: **4 failed, 13 passed** before the fix (`.untracked/malformed-envelope-red.log`); these now raise `LogicalTypeIdentifierError` through the existing node/column reporting path instead of silently becoming unavailable. + +Final corruption regressions: **17 passed** (`.untracked/malformed-envelope-green.log`). Coordinator withdrew the custom UV_CACHE_DIR/--no-sync instruction; subsequent commands use ordinary uv with the worktree .venv and escalated shared-cache access. Interrupted the superseded 3.12 run, then started a fresh final run with `uv run --python 3.12 pytest tests/ -q -ra --tb=short` and a fresh basedpyright check. Main remains b46549fd after fetch. No additional code changes planned during final validation. + +Final Python 3.12 full suite: **9448 passed, 101 skipped, 19 xfailed, 43 warnings in 303.10s**, exit 0 (`.untracked/full-py312-final.log`). Named skip/xfail inventory is `.untracked/py312-skips-and-xfails.txt`. Final basedpyright: **0 errors**, exit 0; ruff and format pass; changelog 19 fragments and design 117 entries validate. Python 3.11 full suite started with ordinary `uv run --python 3.11`, replacing the worktree environment only after 3.12 completed. Fixed the remaining stale operator-storage docstring without behavior changes. + +Final Python 3.11 full suite: **9448 passed, 101 skipped, 19 xfailed, 6 warnings in 305.15s**, exit 0 (`.untracked/full-py311-final.log`). Python 3.13 follows sequentially in the same worktree .venv. + +Final Python 3.13 full suite: **9448 passed, 101 skipped, 19 xfailed, 43 warnings in 312.46s**, exit 0 (`.untracked/full-py313-final.log`). All three full suites passed sequentially. Final main fetch still matches b46549fd. No golden changes. + +## Approved CodeRabbit follow-up round + +Review 5876630496 on f1d26fc2 produced five verified follow-ups. Coordinator approved all five, each failing-first, with no new saved fields, hashes, layouts or golden movement. Merge latest origin/main first (#302 touched operator_node.py), then: + +1. A loaded REPLAY operator whose cache table is absent raises an error naming the node and explicitly saying the cache table is absent; present-table behavior remains. +2. Keep readable persistent and ephemeral records until latest-index selection. Pin only-persistent, only-ephemeral, newer-ephemeral and newer-persistent cases, and document the selection rule in the PR body. +3. Restore source tag/data schemas with the already resolved saved context converter in both schema branches. +4. Loaded jobs report false from is_runnable. +5. Loaded function run rejects its missing pod before run setup, retaining live READ_ONLY/CACHE_ONLY behavior. + +Then targeted/full sequential suites, static checks, one push for the round, CodeRabbit on the new head, and the agreed review stop rule. + +Merged origin/main c8e9eb21 (#302) without conflicts. Review regressions: **5 failed, 50 passed** (`.untracked/review2-red.log`); saved-context routing regressions: **2 failed** after fixture setup was corrected (`.untracked/context-converter-red.log`). Implemented all five fixes. Persistent rows are concatenated after ephemeral rows, so the existing latest-index selector retains persistent precedence on ties while greater indices always win. Four public computation/readback cases cover each store alone and both orderings; no fabricated result records are needed. The pre-existing load/bind is_runnable test was updated to the approved read-only contract. + +Review-round targeted suite: **256 passed**. Cross-store tests after protocol narrowing: **4 passed**. Ruff and format pass; basedpyright **0 errors**, exit 0. Full Python 3.12: **9534 passed, 101 skipped, 23 xfailed, 43 warnings in 231.73s**, exit 0 (`.untracked/review2-py312.log`). The increased test/xfail counts include the newly merged #302 tests. Python 3.11 follows sequentially. + +Review-round Python 3.11: **9534 passed, 101 skipped, 23 xfailed, 6 warnings in 230.01s**, exit 0. Python 3.13 follows sequentially in the worktree environment. + +Review-round Python 3.13: **9534 passed, 101 skipped, 23 xfailed, 43 warnings in 229.97s**, exit 0. All three full suites passed sequentially; final static/fragment checks pass. Main is still c8e9eb21; golden files and hash-step injection remain unchanged. Ready for one push and final-head review. diff --git a/tests/test_core/nodes/test_function_node_iteration.py b/tests/test_core/nodes/test_function_node_iteration.py index 57908f717..2e1f10c83 100644 --- a/tests/test_core/nodes/test_function_node_iteration.py +++ b/tests/test_core/nodes/test_function_node_iteration.py @@ -138,7 +138,7 @@ def test_run_unavailable_raises(self): node = _make_node() node._load_status = LoadStatus.UNAVAILABLE - with pytest.raises(RuntimeError, match="unavailable"): + with pytest.raises(RuntimeError, match="was loaded without a store"): node.run() def test_execute_error_policy_continue_skips_failures(self): diff --git a/tests/test_pipeline/loaded_job/__init__.py b/tests/test_pipeline/loaded_job/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/test_pipeline/loaded_job/save_job_script.py b/tests/test_pipeline/loaded_job/save_job_script.py new file mode 100644 index 000000000..f31e8028b --- /dev/null +++ b/tests/test_pipeline/loaded_job/save_job_script.py @@ -0,0 +1,119 @@ +"""Build, run and save a PipelineJob covering every source kind (ITL-904). + +Run as a script in a separate process, ``python save_job_script.py ``, so +that the test loading the job starts from a fresh process: nothing the save created +(function pods, sources, the store object) is reachable from the loading process. + +Writes into ````: + +* ``c.csv`` and ``dt/`` -- the CSV and Delta inputs; +* ``store/`` -- the Delta store the job ran against; +* ``job.json`` -- the saved job; +* ``expected/