diff --git a/changelog.d/ITL-1019.md b/changelog.d/ITL-1019.md index f534c7756..6aceb7e5f 100644 --- a/changelog.d/ITL-1019.md +++ b/changelog.d/ITL-1019.md @@ -11,9 +11,39 @@ - A job or pipeline holding referential tags and fall-through pods can be saved and loaded; it keeps its hashes and bindings. `ReferenceFallThroughPod.to_config` and `from_config` work. +- An operator node stores a referential stream (`LOG`) and replays it (`REPLAY`): its + table marks the referential tags of the operator's own output with the same marker, + never one that came in from upstream. Every read of the table (a replay, the stored + view `get_all_records()`, a loaded job's read) checks the marker against the node (its + operator's prediction, or a loaded job's saved output schema) and refuses a + disagreement (`InputValidationError`, naming the table and both lists). Source info + stays part of each row's record hash; the marker is put on after the hash and never + enters it. +- `DerivedSource` over a referential stream: `node.as_source()` of a function or operator + node presents the bindings its stored table states (checked against the node's keys), + in the plain form and with `tags_as_data`; `DerivedSource(source, tags_as_data=...)` + over a referential source keeps the source's own. A `tags_as_data` copy of a + referential tag is a new plain data column. Nothing new enters the identity for the + binding: the plain form's content hash is the origin node's, and every pipeline hash + hashes the output schema, where a referential tag is in both halves. A saved derived + source writes no new key: its saved schema states the binding. + +- `FieldMetadataKeepingDatabaseProtocol` (`orcapod.protocols.database_protocols`): a + database states with `keeps_field_metadata` whether it keeps Arrow field metadata. + `InMemoryArrowDatabase`, `DeltaTableDatabase` and a `ConnectorArrowDatabase` over + `SpiralDBConnector` state `True`. The SQLite and PostgreSQL connector databases and + `NoOpArrowDatabase` state `False`. `ExtensionAwareDatabase` states what the database + it wraps states, and a connector database what its connector states. A database + without the member is treated as not keeping field metadata. ### Changed +- A node that would store a referential tag refuses every database that does not state + `keeps_field_metadata = True`, before anything is written, so an unknown database + never passes by default. A database that states `True` is not probed. Before, only a + database that implements `validate_records` was checked, with a probe of null-typed + columns. + - Saved files of format `0.3.0` state every binding: each source config that takes the option writes `tag_only_columns` (`True` when every tag is plain), and each function node writes `fall_through_references` (empty for a plain pod). The format version decides @@ -26,4 +56,22 @@ - The SQLite and PostgreSQL databases keep no Arrow field metadata, so a function node that would store a referential tag there is refused (`InputValidationError`, naming the database and the two ways out: a database that keeps field metadata, or - `tag_only_columns`) before anything is written. + `tag_only_columns`) before anything is written. The same holds for an operator node + attached to such a database in `LOG` or `REPLAY` mode. +- SpiralDB's `keeps_field_metadata` is verified against the pyspiral fake only (first + write, append, merge on overlap, reread). The live service is unverified. The + connector database refuses schema evolution, so no merge can drop the marker. The + SpiralDB connector writes a table's Arrow metadata before its rows, so a failed + metadata write stores no row unmarked; a write that stores no row publishes nothing; + a failed write puts the metadata entry back as it was. +- Concurrent writers to one SpiralDB table through separate connectors are not + coordinated: one writer's rollback of a failed write can remove the metadata entry + another writer just published, so rows can read back without their referential tag + marker. A node refuses such a table on read (the marker must match the node), so no + wrong binding is ever read (ITL-1190). +- Writes of differing field metadata to one table are not refused. The in-memory + database and the connector database combine buffered and committed rows with the + first table's field metadata, so a batch whose marker differs from the rows already + there takes theirs. Orcapod's own writers never do this: a table's path derives from + its schema, where a referential tag is in both halves, so one path has one binding. + Only an outside writer can mix them, and a lost marker is refused on read (ITL-1190). diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index 411c6efad..f20860801 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -190,10 +190,14 @@ column that carries Arrow field metadata: the key `orcapod:tag_kind` with the va marker. The function's result table never records referential status, so plain and referential runs of the same function share their results. The marker moves no hash, and the stores keep it on every write (first write, append, merge, schema evolution; in -memory and on Delta). The SQLite and PostgreSQL databases keep no Arrow -metadata at all: a function node that would store a referential tag there is refused when -its database is attached, before anything is written. Use a database that keeps field -metadata, or keep those tags plain with `tag_only_columns`. +memory and on Delta). A database states whether it keeps +field metadata with `keeps_field_metadata` +(`orcapod.protocols.database_protocols.FieldMetadataKeepingDatabaseProtocol`): the +in-memory, Delta and SpiralDB databases do; the SQLite and PostgreSQL databases keep no +Arrow metadata at all. A node that would store a referential tag in a database that does +not state it (a database without the member included) is refused when its database is +attached, before anything is written. Use a database that keeps field metadata, or keep +those tags plain with `tag_only_columns`. ```python from orcapod.core.referential_tags import stored_references @@ -218,6 +222,32 @@ A node reads the binding back from its table, and refuses a table whose marker d with what the node declares (its pod's fall-through tags, or for a loaded job the saved file's). +An operator node that stores its output (`LOG`) marks its table the same way: the +referential tags of the operator's own output carry the marker, written from the +operator's binding, never from a marker that came in from upstream. A `REPLAY` read, and +the stored view `get_all_records()`, check the marker against the node (its operator's +prediction, or for a loaded job the saved output schema, where a referential tag is in +both halves). A database that keeps no field metadata is refused when it is attached in +`LOG` or `REPLAY` mode. Source info stays part of each stored row's record hash; the +marker never enters it. + +`node.as_source()` gives the stored rows as a source with the bindings the stored table +states, in the plain form and with `tags_as_data` (a copy of a referential tag is a new, +plain data column). Nothing is added to the derived source's identity for the binding: its +plain form's content hash is the node's, which carries the binding already, and its +pipeline hash is its output schema, where a referential tag is in both halves. + +```python +derived = job.nodes["scale"].as_source() +print(derived.keys()) +# (('unit', 'time'), ('unit', 'time', 'scaled')) +print(derived.as_table().column_names) +# ['unit', 'time', 'scaled'] +``` + +In a stream the binding is the shared name, so the marker stays with the stored table: +the rows of a replay or of a derived source carry none. + A saved job or pipeline (format `0.3.0`) states every binding: each source config writes `tag_only_columns` (`True` when every tag is plain), and each function node writes `fall_through_references` (empty for a plain pod). The format version, never a missing @@ -228,9 +258,7 @@ the file is checked against the stored tables when a node reads them. ## Not supported yet These raise "not supported with referential tags yet" when a referential tag reaches them: -`MergeJoin`, `MapColumns` (`map_columns`; rename with `map_tags` and `map_data`), storing an operator node's results (`LOG` and `REPLAY` cache modes), -and `DerivedSource` over a stream with referential tags (so `node.as_source()` of a node -with referential tags). +`MergeJoin` and `MapColumns` (`map_columns`; rename with `map_tags` and `map_data`). `PollingSource` keeps its tags plain. Each error names its issue and the way out: build the source with `tag_only_columns=True` to keep every tag plain, as in v0.3. A plain tag is not data, so a function that reads a tag's value (as `scale` above reads `time`) can no longer diff --git a/docs/dev/design_issues/ITL-1023-RT1.md b/docs/dev/design_issues/ITL-1023-RT1.md index b8750dc62..371aa9b7d 100644 --- a/docs/dev/design_issues/ITL-1023-RT1.md +++ b/docs/dev/design_issues/ITL-1023-RT1.md @@ -5,4 +5,4 @@ **Severity:** medium **Issue:** ITL-1023 -The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, and `DerivedSource` over a referential stream (ITL-1019; saving a pipeline or job and the function node's stored bindings are supported since ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. +The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. diff --git a/docs/dev/design_issues/ITL-1163-LJ4.md b/docs/dev/design_issues/ITL-1163-LJ4.md new file mode 100644 index 000000000..540464978 --- /dev/null +++ b/docs/dev/design_issues/ITL-1163-LJ4.md @@ -0,0 +1,8 @@ +## `src/orcapod/pipeline/job.py` — reading a loaded `PipelineJob` + +### LJ4 — A saved job does not record an operator node's cache mode +**Status:** open +**Severity:** medium +**Issue:** ITL-1163 + +`OperatorJobNode.from_descriptor` reads `descriptor.get("cache_mode", "off")`, but `PipelineJob.save` never writes `cache_mode` into an operator node's descriptor. Every loaded operator node is therefore `OFF`, whatever mode it ran in. A load-only run of a loaded job runs the operator again over its rebuilt inputs instead of replaying the table a `LOG` or `REPLAY` node stored. The stored rows are still read by `node.get_all_records()` and `node.as_source()`, and by `REPLAY` when the caller attaches that mode again after loading. This is the case in plain mode and with referential tags (found while adding operator-node storage of referential streams, ITL-1019; `tests/test_pipeline/test_saved_job_with_referential_operator_nodes.py` attaches `REPLAY` again). A fix writes the mode into the saved descriptor, which is a saved-file format question. diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 4152fb4d6..c1d85af0f 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -25,7 +25,14 @@ from typing import TYPE_CHECKING, Any, Literal, cast from orcapod import contexts -from orcapod.core.referential_tags import not_supported_yet, stream_references +from orcapod.core.referential_tags import ( + mark_references, + reference_names, + require_database_keeps_marker, + stored_references, + stream_references, + without_reference_marks, +) from orcapod.channels import Channel, ChannelClosed, ReadableChannel, WritableChannel from orcapod.config import OrcapodConfig from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod @@ -48,6 +55,7 @@ from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.core.nodes.staleness import RunHoldingNode from orcapod.errors import ( + InputValidationError, PipelineJobRequiredError, OrcapodDatabaseReadError, OrcapodDatabaseWriteError, @@ -881,7 +889,18 @@ def attach_databases( Args: pipeline_database: Database for pipeline records. cache_mode: Caching behaviour (OFF, LOG, or REPLAY). + + Raises: + InputValidationError: If the node's output has referential tags, the cache + mode stores or replays them (``LOG`` or ``REPLAY``), and the database + cannot keep the marker that states them (ITL-1019). Nothing is + attached or written. """ + if cache_mode != CacheMode.OFF: + # A stored table states its referential tags; refuse a database that + # cannot, before anything runs or is written (ITL-1019). + require_database_keeps_marker(pipeline_database, stream_references(self)) + # A database that cannot store the cache refuses it now, before anything # runs or changes (ITL-1162). self.require_storable_tables(pipeline_database, cache_mode) @@ -1050,6 +1069,93 @@ def from_descriptor( return node + # ------------------------------------------------------------------ + # Referential tags in the stored table (ITL-1019) + # ------------------------------------------------------------------ + + def declared_references(self) -> tuple[str, ...]: + """Return the referential tags of this node's output, as the node declares them. + + A node built in this process declares its operator's predicted output: the + names in both halves of ``keys()``. A node loaded by ``PipelineJob.load`` + declares its saved descriptor's output schema, the saved job being an + independent point of verification of the stored table (developer, November 16, + 2026). These are the tags the node's stored table must mark. + + Returns: + The referential tags, in tag order; empty for a plain output. + """ + saved = self._descriptor.get("output_schema") if self._loaded else None + if saved is not None: + return reference_names(saved.get("tag", {}), saved.get("data", {})) + return stream_references(self) + + def _check_stored_references(self, schema: "pa.Schema") -> None: + """Refuse a stored table whose referential tag marker disagrees with the node. + + The stored table states its bindings (developer, November 16, 2026, ITL-1019): + a referential tag column carries the marker of ``referential_tags``. The node + reads the table only when it agrees with what the node declares + (``declared_references``). + + Args: + schema: The schema of the node's stored table. + + Raises: + InputValidationError: If the table's marker is invalid, or the table and + the node disagree on which tags are referential. + """ + assert self._pipeline_database is not None + table_path = self._pipeline_database.base_path + self._stored_table_path + try: + stored = stored_references(schema) + except ValueError as exc: + raise InputValidationError( + f"Operator node {self.label!r}, stored table {table_path}: {exc}" + ) from exc + expected = self.declared_references() + if set(stored) != set(expected): + declarer = "saved job" if self._loaded else "operator" + raise InputValidationError( + f"Operator node {self.label!r}: the stored table {table_path} marks " + f"the referential tags {sorted(stored)}, but the node's {declarer} " + f"declares {sorted(expected)}. The table was not written by this " + "node's pipeline; nothing was read." + ) + + def _stream_over_stored(self, records: "pa.Table") -> ArrowTableStream: + """Return a stream over rows read from the stored table, without the marker. + + In a stream the binding is the shared name of a tag and a data column, so the + marker stays with the stored table: carried on, it would be written by a + downstream store whose binding it is not. + + The binding is the node's (``declared_references``), which every read checked + against the table's marker (``_check_stored_references``). + + Args: + records: The stored rows, without the bookkeeping columns. + + Returns: + The stream, in this node's data context. + """ + # Stored rows are this node's: give them its data context, not the default. + # ``data_context`` types the Python datagrams only; the rows' Arrow context + # column comes from the table, and a stream fills a missing one with the + # default key. + rows = without_reference_marks(records) + if constants.CONTEXT_KEY not in rows.column_names: + rows = rows.append_column( + pa.field(constants.CONTEXT_KEY, pa.large_string(), nullable=False), + pa.array([self.data_context_key] * rows.num_rows, pa.large_string()), + ) + return ArrowTableStream( + rows, + tag_columns=self.keys()[0], + reference_tags=self.declared_references(), + data_context=self.data_context, + ) + # ------------------------------------------------------------------ # Internal helpers # ------------------------------------------------------------------ @@ -1165,11 +1271,12 @@ def _filter_by_content_hash(self, table: pa.Table) -> pa.Table: def _store_output_stream(self, stream: StreamProtocol) -> None: """Materialize stream and store in the pipeline database with per-row dedup. - Raises: - NotImplementedError: If the stream has a referential tag (ITL-1019). + The stored table states its bindings (developer, November 16, 2026, ITL-1019): + the columns of the node's referential tags carry the marker of + ``referential_tags``, written from the operator's own output binding and + never from metadata that came in from upstream. The marker is put on after the + per-row record hashes are computed, so it never enters them. """ - if stream_references(stream): - raise not_supported_yet("Operator node storage", "ITL-1019") pipeline_database = self._pipeline_database if pipeline_database is None: raise RuntimeError( @@ -1238,7 +1345,10 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: try: pipeline_database.add_records( self._stored_table_path, - with_data_context(output_table, self.data_context_key), + with_data_context( + mark_references(output_table, stream_references(self)), + self.data_context_key, + ), record_id_column=self.HASH_COLUMN_NAME, skip_duplicates=True, ) @@ -1278,9 +1388,18 @@ def _read_current_context_records(self) -> "pa.Table | None": The rows of every run that shares the table, before ``_filter_by_content_hash`` keeps this node's. Rows of another data context are left out and reported once. + Every read of the stored table goes through here, so every read checks the + table's referential tag marker against the node (``_check_stored_references``). + The rows keep the marker; a read into a stream drops it + (``_stream_over_stored``). + Returns: The rows, with ``HASH_COLUMN_NAME`` and ``NODE_CONTENT_HASH_COL`` and without the ``__data_context`` column, or ``None`` when the table does not exist. + + Raises: + InputValidationError: If the table and the node disagree on which tags are + referential (ITL-1019). """ assert self._pipeline_database is not None self._account_for_stale_tables() @@ -1289,6 +1408,7 @@ def _read_current_context_records(self) -> "pa.Table | None": ) if records is None: return None + self._check_stored_references(records.schema) stored_rows = records.num_rows records = keep_rows_of_data_context(records, self.data_context_key) if records.num_rows < stored_rows and not self._skipped_rows_reported: @@ -1480,13 +1600,7 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": ] if cols_to_drop: records_table = records_table.drop(cols_to_drop) - tag_keys = self.keys()[0] - if records is not None and stream_references(self): - raise not_supported_yet("Operator node storage", "ITL-1019") - # Stored rows are this node's: give them its data context, not the default. - return ArrowTableStream( - records_table, tag_columns=tag_keys, data_context=self.data_context - ) + return self._stream_over_stored(records_table) def get_cached_output(self) -> StreamProtocol | None: """Return cached output stream in REPLAY mode, else None. @@ -1626,12 +1740,7 @@ def _replay_from_cache(self) -> None: if cols_to_drop: records = records.drop(cols_to_drop) - tag_keys = self.keys()[0] - if stream_references(self): - raise not_supported_yet("Operator node storage", "ITL-1019") - self._cached_output_stream = ArrowTableStream( - records, tag_columns=tag_keys, data_context=self.data_context - ) + self._cached_output_stream = self._stream_over_stored(records) self._update_modified_time() def run(self, *, load_only: bool = False) -> StreamView: @@ -1825,7 +1934,10 @@ def as_table( ), ) return ArrowTableStream( - empty, tag_columns=tuple(tag_schema), data_context=self.data_context + empty, + tag_columns=tuple(tag_schema), + reference_tags=self.declared_references(), + data_context=self.data_context, ).as_table(columns=columns, all_info=all_info) empty_stream = ArrowTableStream( self._make_empty_table(), @@ -1883,6 +1995,14 @@ def get_all_records( per ``ColumnConfig`` conventions. Returns ``None`` when no database is attached or no matching records exist. Switching to OFF does not remove records stored by an earlier LOG run. + + This is the raw records view: a referential tag column keeps the marker the + stored table states it with (ITL-1019), checked against the node on read. + ``as_source()`` reads the binding from it. + + Raises: + InputValidationError: If the stored table and the node disagree on which + tags are referential. """ if self._pipeline_database is None: return None diff --git a/src/orcapod/core/referential_tags.py b/src/orcapod/core/referential_tags.py index 7c508d813..41a74daa1 100644 --- a/src/orcapod/core/referential_tags.py +++ b/src/orcapod/core/referential_tags.py @@ -261,40 +261,49 @@ def require_database_keeps_marker(database: object, references: Iterable[str]) - """Refuse, before anything is written, a database that cannot store the marker. A stored table must state what it stores (developer, November 16, 2026, ITL-1019), - so a referential tag is never stored unmarked. A database that checks rows before - writing them (``RecordValidatingDatabaseProtocol``, the connector-backed databases) - is asked whether it keeps the marker and refused if it does not. Any other database - is accepted without a check: the in-memory and Delta databases keep Arrow field - metadata. + so a referential tag is never stored unmarked. A database is accepted only when it + states that it keeps Arrow field metadata + (``FieldMetadataKeepingDatabaseProtocol.keeps_field_metadata``); one that does not + have the member is refused, so an unknown database never passes by default. The + statement decides: a database that states it is not probed. One that does not, + and can check rows before writing them (``RecordValidatingDatabaseProtocol``), is + asked why, and the reason it gives is part of the error. Args: database: The database a node will store its referential tags in. references: The referential tags it will store; nothing is checked when empty. Raises: - InputValidationError: If the database cannot keep the marker, naming the - database kind and the two ways out. + InputValidationError: If the database does not state that it keeps field + metadata, naming the database class and the two ways out. """ - from orcapod.protocols.database_protocols import RecordValidatingDatabaseProtocol + from orcapod.protocols.database_protocols import ( + RecordValidatingDatabaseProtocol, + keeps_field_metadata, + ) references = tuple(references) - if not references or not isinstance(database, RecordValidatingDatabaseProtocol): + if not references or keeps_field_metadata(database): return - probe = mark_references( - pa.table({name: pa.array([], type=pa.null()) for name in references}), - references, + reason = "it does not state that it keeps field metadata (keeps_field_metadata)" + if isinstance(database, RecordValidatingDatabaseProtocol): + # Only for the error: the reason the database gives for refusing marked rows. + probe = mark_references( + pa.table({name: pa.array([], type=pa.null()) for name in references}), + references, + ) + try: + database.validate_records(probe) + except ValueError as exc: + reason = str(exc) + raise InputValidationError( + f"{type(database).__name__} cannot store the referential tags " + f"{list(references)}: a stored referential tag carries Arrow field " + f"metadata, and this database keeps none ({reason}). Use a database that " + "keeps field metadata (DeltaTableDatabase, InMemoryArrowDatabase, the " + "SpiralDB database), or keep these tags plain with tag_only_columns on " + "the source. Nothing was written." ) - try: - database.validate_records(probe) - except ValueError as exc: - raise InputValidationError( - f"{type(database).__name__} cannot store the referential tags " - f"{list(references)}: a stored referential tag carries Arrow field " - f"metadata, and this database keeps none ({exc}). Use a database that " - "keeps field metadata (DeltaTableDatabase, InMemoryArrowDatabase, the " - "SpiralDB database), or keep these tags plain with tag_only_columns on " - "the source. Nothing was written." - ) from exc def without_reference_marks(table: "pa.Table") -> "pa.Table": diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index d934366ae..8e0d7acea 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -5,7 +5,11 @@ from datetime import datetime from typing import TYPE_CHECKING, Any, ClassVar -from orcapod.core.referential_tags import not_supported_yet, stream_references +from orcapod.core.referential_tags import ( + stored_references, + stream_references, + without_reference_marks, +) from typing_extensions import override from orcapod.core.sources.base import RootSource @@ -169,6 +173,16 @@ class DerivedSource(RootSource): rows with the correct columns. In the ``tags_as_data`` form the rows appear once the node has run. + Referential tags (ITL-1019): the source carries the bindings of what it reads, + in both forms. Over a job node they are the ones the node's stored table states + (the marker of ``referential_tags``), which must be the node's own (a name in + both halves of its keys); a table that disagrees is refused. Over a source they + are the source's own. A ``tags_as_data`` copy of a referential tag is a new, + plain data column. Nothing is added to the identity for the binding, because it + is already there: the plain form's ``content_hash`` is the origin's, which carries + it, and every ``pipeline_hash`` and the ``tags_as_data`` form's ``content_hash`` + hash the output schema, in which a referential tag is in both halves. + Args: origin: The source or DB-backed job node to derive from. source_id: Canonical source name. Defaults as described above. @@ -239,8 +253,6 @@ def __init__( record_id_columns: str | Collection[str] | None = None, **kwargs: Any, ) -> None: - if stream_references(origin): - raise not_supported_yet("DerivedSource", "ITL-1019") key_columns = normalize_record_id_columns(record_id_columns) if tags_as_data is None: if isinstance(origin, RootSource): @@ -399,11 +411,46 @@ def keys( return self._origin.keys(columns=columns, all_info=all_info) return self._get_stream().keys(columns=columns, all_info=all_info) + def _references(self) -> tuple[str, ...]: + """Return the referential tags this source presents: its origin's. + + A derived source carries the bindings of what it reads (gap audit Q7). Over a + job node they are the ones its stored table states, which ``_origin_records`` + checks against the node's keys on every read; over a source, the source's own. + """ + return stream_references(self._origin) + def _origin_records(self) -> "pa.Table | None": - """Return the origin's rows, or ``None`` when a job node has none stored.""" + """Return the origin's rows, or ``None`` when a job node has none stored. + + A job node's stored rows state their referential tags with a marker on the + column (ITL-1019). The derived source presents exactly those: the marker must + name the tags the node's keys share between tag and data, and is then + dropped, since in a stream the binding is the shared name. + + Raises: + InputValidationError: If a job node's stored rows mark other referential + tags than the node's keys declare, or carry an invalid marker. + """ if isinstance(self._origin, RootSource): return self._origin.as_table() - return self._origin.get_all_records() + records = self._origin.get_all_records() + if records is None: + return None + try: + stored = stored_references(records.schema) + except ValueError as exc: + raise InputValidationError( + f"DerivedSource over {self._origin.label!r}: {exc}" + ) from exc + declared = self._references() + if set(stored) != set(declared): + raise InputValidationError( + f"DerivedSource over {self._origin.label!r}: the stored records mark " + f"the referential tags {sorted(stored)}, but the node declares " + f"{sorted(declared)}; nothing was read." + ) + return without_reference_marks(records) def _empty_origin_table(self) -> "pa.Table": """Return a zero-row table with the origin's declared columns.""" @@ -452,6 +499,7 @@ def _derive_from( result = builder.build( table, tag_columns=self._origin.keys()[0], + reference_tags=self._references(), source_id=self.source_id, record_id_columns=self._record_id_columns or (), ) @@ -476,8 +524,7 @@ def _get_stream(self) -> ArrowTableStream: self._cached_table = self._empty_origin_table() else: self._cached_table = records - tag_keys = self._origin.keys()[0] - return ArrowTableStream(self._cached_table, tag_columns=tag_keys) + return self._stream_over(self._cached_table) def _held_rows(self) -> "pa.Table | None": """The rows this source holds from its origin, or ``None`` before a read.""" @@ -569,9 +616,7 @@ def head( self, origin=origin_kind ) if self._tags_as_data is None and self._cached_table is not None: - stream = ArrowTableStream( - self._cached_table, tag_columns=self._origin.keys()[0] - ) + stream = self._stream_over(self._cached_table) return stream.head(n, columns=config).for_stream(self, origin=origin_kind) if isinstance(self._origin, RootSource): origin_head = self._origin.head(n) @@ -637,7 +682,11 @@ def _stream_over(self, table: "pa.Table") -> ArrowTableStream: """Return this source's stream over the origin rows ``table``.""" if self._tags_as_data is not None: return self._derive_from(table, self._tags_as_data) - return ArrowTableStream(table, tag_columns=self._origin.keys()[0]) + return ArrowTableStream( + table, + tag_columns=self._origin.keys()[0], + reference_tags=self._references(), + ) def iter_data(self): return self._get_stream().iter_data() diff --git a/src/orcapod/databases/connector_arrow_database.py b/src/orcapod/databases/connector_arrow_database.py index 9f8c7e644..205a108ac 100644 --- a/src/orcapod/databases/connector_arrow_database.py +++ b/src/orcapod/databases/connector_arrow_database.py @@ -327,6 +327,17 @@ def add_record( flush=flush, ) + @property + def keeps_field_metadata(self) -> bool: + """Whether this database keeps Arrow field metadata: as its connector states. + + A connector without the ``keeps_field_metadata`` member does not keep it + (ITL-1019). + """ + from orcapod.protocols.database_protocols import keeps_field_metadata + + return keeps_field_metadata(self._connector) + def validate_records(self, records: pa.Table) -> None: """Raise ``ValueError`` if the connector could not store ``records`` as given. diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index 19dd15fea..c87d71c28 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -1621,6 +1621,12 @@ def from_config(cls, config: dict[str, Any]) -> "DeltaTableDatabase": _path_prefix=_path_prefix, ) + @property + def keeps_field_metadata(self) -> bool: + """Whether this database keeps Arrow field metadata: it does, on every write + path including schema evolution (ITL-1019).""" + return True + def flush(self) -> None: """Flush all pending batches. diff --git a/src/orcapod/databases/extension_aware_database.py b/src/orcapod/databases/extension_aware_database.py index e8fbb3ef5..63306b747 100644 --- a/src/orcapod/databases/extension_aware_database.py +++ b/src/orcapod/databases/extension_aware_database.py @@ -164,6 +164,17 @@ def add_records( flush=flush, ) + @property + def keeps_field_metadata(self) -> bool: + """Whether the wrapped database keeps Arrow field metadata (ITL-1019). + + A wrapped database without the member does not keep it, so the wrapper never + states more than the database it wraps. + """ + from orcapod.protocols.database_protocols import keeps_field_metadata + + return keeps_field_metadata(self._db) + def validate_records(self, records: pa.Table) -> None: """Raise ``ValueError`` if the wrapped database could not store ``records``. diff --git a/src/orcapod/databases/in_memory_databases.py b/src/orcapod/databases/in_memory_databases.py index a4686da94..7857c96fd 100644 --- a/src/orcapod/databases/in_memory_databases.py +++ b/src/orcapod/databases/in_memory_databases.py @@ -289,6 +289,11 @@ def add_records( # Flush # ------------------------------------------------------------------ + @property + def keeps_field_metadata(self) -> bool: + """Whether this database keeps Arrow field metadata: it does (ITL-1019).""" + return True + def flush(self) -> None: for record_key in list(self._pending_batches.keys()): pending = self._pending_batches.pop(record_key) diff --git a/src/orcapod/databases/noop_database.py b/src/orcapod/databases/noop_database.py index 375e38985..5703c5967 100644 --- a/src/orcapod/databases/noop_database.py +++ b/src/orcapod/databases/noop_database.py @@ -86,6 +86,11 @@ def get_records_with_column_value( ) -> "pa.Table | None": return None + @property + def keeps_field_metadata(self) -> bool: + """Whether this database keeps Arrow field metadata: it keeps nothing (ITL-1019).""" + return False + def flush(self) -> None: pass diff --git a/src/orcapod/databases/postgresql_connector.py b/src/orcapod/databases/postgresql_connector.py index e8befed1b..39be7f6d2 100644 --- a/src/orcapod/databases/postgresql_connector.py +++ b/src/orcapod/databases/postgresql_connector.py @@ -612,6 +612,11 @@ def upsert_records( conn.rollback() raise + @property + def keeps_field_metadata(self) -> bool: + """Whether this connector keeps Arrow field metadata: it does not (ITL-1019).""" + return False + def unstorable_column_reason( self, name: str, arrow_type: pa.DataType, *, primary_key: bool ) -> str | None: diff --git a/src/orcapod/databases/spiraldb_connector.py b/src/orcapod/databases/spiraldb_connector.py index f88e5c455..36bc4a995 100644 --- a/src/orcapod/databases/spiraldb_connector.py +++ b/src/orcapod/databases/spiraldb_connector.py @@ -920,14 +920,46 @@ def upsert_records( meta_kv = _serialize_arrow_metadata(records) def write_rows(rows: pa.Table) -> None: + # The Arrow metadata entry as it was before this write, put back if the + # write fails. Read before anything is written, so a failed read leaves + # nothing to clean up. + previous_meta = ( + tbl.get_metadata().get(_ARROW_METADATA_KEY) + if meta_kv is not None + else None + ) # Record first, so rows are never stored without their record; if the rows # then fail, remove the record again, since it names columns the table lacks. wrote_record = record_first and write_nullability_record( tbl, records.schema ) try: + # The Arrow metadata too goes before the rows, so rows never land + # without the metadata that states them (a referential tag's marker, + # ITL-1019): a failed metadata write stores no row, and its retry + # writes both. + if meta_kv is not None: + tbl.set_metadata(meta_kv) tbl.write(rows) except BaseException: + # The Arrow metadata went in before the rows: put the entry back as it + # was, so the rows already stored keep theirs and a later write without + # metadata does not read back with this one. + if meta_kv is not None: + try: + if previous_meta is None: + tbl.drop_metadata([_ARROW_METADATA_KEY]) + else: + tbl.set_metadata({_ARROW_METADATA_KEY: previous_meta}) + except Exception as cleanup_exc: + logger.warning( + "Could not restore the Arrow metadata of %s after a failed " + "write (%s: %s); its rows may read back with the metadata " + "of the failed write.", + qualified, + type(cleanup_exc).__name__, + cleanup_exc, + ) # Remove the record only while the table still has no value columns: # once another writer's rows have landed, they rely on it. if wrote_record: @@ -944,8 +976,6 @@ def write_rows(rows: pa.Table) -> None: cleanup_exc, ) raise - if meta_kv is not None: - tbl.set_metadata(meta_kv) if not skip_existing: # Always upsert-by-key: existing rows overwritten, novel rows inserted. @@ -966,6 +996,13 @@ def write_rows(rows: pa.Table) -> None: if len(novel) > 0: write_rows(novel) + @property + def keeps_field_metadata(self) -> bool: + """Whether this connector keeps Arrow field metadata: it does, in the native KV + store (ITL-1019). Verified against the pyspiral fake only; schema evolution + is refused by ``ConnectorArrowDatabase``.""" + return True + def unstorable_column_reason( self, name: str, arrow_type: pa.DataType, *, primary_key: bool ) -> str | None: diff --git a/src/orcapod/databases/sqlite_connector.py b/src/orcapod/databases/sqlite_connector.py index 72c89236b..d3d20d491 100644 --- a/src/orcapod/databases/sqlite_connector.py +++ b/src/orcapod/databases/sqlite_connector.py @@ -441,6 +441,11 @@ def upsert_records( rows = (tuple(row[c] for c in cols) for row in records.to_pylist()) conn.executemany(sql, rows) + @property + def keeps_field_metadata(self) -> bool: + """Whether this connector keeps Arrow field metadata: it does not (ITL-1019).""" + return False + def unstorable_column_reason( self, name: str, arrow_type: pa.DataType, *, primary_key: bool ) -> str | None: diff --git a/src/orcapod/protocols/database_protocols.py b/src/orcapod/protocols/database_protocols.py index bc0ad3486..7c373b13f 100644 --- a/src/orcapod/protocols/database_protocols.py +++ b/src/orcapod/protocols/database_protocols.py @@ -257,6 +257,43 @@ def validate_records(self, records: pa.Table) -> None: ... +@runtime_checkable +class FieldMetadataKeepingDatabaseProtocol(Protocol): + """A database that states whether it keeps Arrow field metadata. + + A stored referential tag is marked with Arrow field metadata (ITL-1019), so a node + stores one only in a database that keeps it on every write path (first write, + append, merge, schema evolution where the database supports it) and reads it back. + A database states this with ``keeps_field_metadata``. One that does not have the + member is treated as not keeping field metadata: a node refuses it before + anything is written (``referential_tags.require_database_keeps_marker``). + + ``InMemoryArrowDatabase`` and ``DeltaTableDatabase`` keep it. A + ``ConnectorArrowDatabase`` keeps it when its connector does: a connector states it + with the same member (``SpiralDBConnector`` does; ``SQLiteConnector`` and + ``PostgreSQLConnector`` do not). ``ExtensionAwareDatabase`` states what the + database it wraps states. ``NoOpArrowDatabase`` keeps nothing. + """ + + @property + def keeps_field_metadata(self) -> bool: + """Whether every write path keeps Arrow field metadata and reads give it back.""" + ... + + +def keeps_field_metadata(database: object) -> bool: + """Return whether ``database`` states that it keeps Arrow field metadata. + + Args: + database: A database, or a connector. + + Returns: + ``True`` only when it has ``keeps_field_metadata`` and that is ``True``; a + database without the member does not keep field metadata. + """ + return getattr(database, "keeps_field_metadata", False) is True + + @runtime_checkable class DatabaseRegistryProtocol(Protocol): """Protocol for a database config registry used during pipeline save/load. diff --git a/tests/test_core/referential_tags/test_database_capability.py b/tests/test_core/referential_tags/test_database_capability.py new file mode 100644 index 000000000..2f14aabf2 --- /dev/null +++ b/tests/test_core/referential_tags/test_database_capability.py @@ -0,0 +1,459 @@ +"""A database states whether it keeps Arrow field metadata (ITL-1019). + +The developer's rule: a database that cannot keep field metadata is refused before +anything is written, and when in doubt, refuse. A database states the capability with +``keeps_field_metadata`` (``FieldMetadataKeepingDatabaseProtocol``); one without the member +is refused when a node would store a referential tag in it. Every database that states +``True`` keeps the marker on its write paths: in memory and Delta are covered by +``test_persistence.py``; SpiralDB is covered here against the pyspiral fake only (the live +service is unverified). +""" + +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +import orcapod as op +from orcapod.contexts import resolve_context +from orcapod.core.operators import Join +from orcapod.core.referential_tags import mark_references, stored_references +from orcapod.databases import ( + ConnectorArrowDatabase, + DeltaTableDatabase, + InMemoryArrowDatabase, + NoOpArrowDatabase, + SpiralDBConnector, +) +from orcapod.databases.extension_aware_database import ExtensionAwareDatabase +from orcapod.databases.sqlite_connector import SQLiteConnector +from orcapod.errors import InputValidationError, UnstorableColumnError +from orcapod.protocols.db_connector_protocol import ColumnInfo +from orcapod.pipeline import PipelineJob +from orcapod.protocols.database_protocols import ( + FieldMetadataKeepingDatabaseProtocol, + keeps_field_metadata, +) +from orcapod.types import CacheMode +from tests.test_databases import fake_pyspiral + +CALLS: list[float] = [] + + +@op.function_pod(output_keys=["y"]) +def double(amp: float) -> float: + """Double the amplitude; reads no tag, so ``unit`` falls through.""" + CALLS.append(amp) + return 2 * amp + + +class ThirdPartyDatabase: + """A database adapter that keeps field metadata but does not say so. + + It delegates every call to an in-memory database, except the capability, which it + does not have: an unknown database. + """ + + def __init__(self, inner: Any) -> None: + self._inner = inner + + def at(self, *path_components: str) -> ThirdPartyDatabase: + return ThirdPartyDatabase(self._inner.at(*path_components)) + + def __getattr__(self, name: str) -> Any: + if name == "keeps_field_metadata": + raise AttributeError(name) + return getattr(self._inner, name) + + +class DeclaringThirdPartyDatabase(ThirdPartyDatabase): + """The same adapter, stating that it keeps field metadata.""" + + keeps_field_metadata = True + + def at(self, *path_components: str) -> DeclaringThirdPartyDatabase: + return DeclaringThirdPartyDatabase(self._inner.at(*path_components)) + + +def _extension_aware(database: Any) -> Any: + """An ``ExtensionAwareDatabase`` over ``database``, typed loosely for the tests.""" + converter: Any = resolve_context(None).type_converter + return ExtensionAwareDatabase(database, converter=converter) + + +def _adapter(cls: type[ThirdPartyDatabase]) -> Any: + """An adapter over a fresh in-memory database; duck-typed, as a third party's is.""" + return cls(InMemoryArrowDatabase()) + + +@pytest.fixture() +def spiral(tmp_path: Path) -> Iterator[None]: + with fake_pyspiral.install(tmp_path / "spiral"): + yield + + +def _source() -> op.sources.DictSource: + return op.sources.DictSource( + [{"unit": 1, "time": 10, "amp": 0.5}, {"unit": 2, "time": 20, "amp": 1.5}], + tag_columns=["unit", "time"], + tag_only_columns=["time"], + source_id="spikes", + ) + + +# --------------------------------------------------------------------------- +# What each database states +# --------------------------------------------------------------------------- + + +def test_each_database_states_the_capability(tmp_path: Path, spiral: None) -> None: + sqlite = ConnectorArrowDatabase(SQLiteConnector(":memory:")) + spiraldb = ConnectorArrowDatabase(SpiralDBConnector("p")) + stated = { + "in memory": InMemoryArrowDatabase(), + "delta": DeltaTableDatabase(tmp_path / "delta"), + "spiraldb": spiraldb, + "sqlite": sqlite, + "noop": NoOpArrowDatabase(), + "extension-aware over in memory": _extension_aware(InMemoryArrowDatabase()), + "extension-aware over sqlite": _extension_aware(sqlite), + "in memory, scoped": InMemoryArrowDatabase().at("a", "b"), + "spiraldb, scoped": spiraldb.at("a"), + } + assert {name: keeps_field_metadata(db) for name, db in stated.items()} == { + "in memory": True, + "delta": True, + "spiraldb": True, + "sqlite": False, + "noop": False, + "extension-aware over in memory": True, + "extension-aware over sqlite": False, + "in memory, scoped": True, + "spiraldb, scoped": True, + } + for db in stated.values(): + assert isinstance(db, FieldMetadataKeepingDatabaseProtocol) + + +def test_a_database_without_the_member_does_not_keep_field_metadata() -> None: + adapter = ThirdPartyDatabase(InMemoryArrowDatabase()) + assert not keeps_field_metadata(adapter) + assert not isinstance(adapter, FieldMetadataKeepingDatabaseProtocol) + # A wrapper never states more than the database it wraps. + assert not keeps_field_metadata(_extension_aware(adapter)) + + +# --------------------------------------------------------------------------- +# An unknown database is refused before anything is written +# --------------------------------------------------------------------------- + + +def test_a_function_node_refuses_an_undeclared_database() -> None: + inner = InMemoryArrowDatabase() + adapter: Any = ThirdPartyDatabase(inner) + job = PipelineJob(database=adapter) + CALLS.clear() + with pytest.raises(InputValidationError) as raised: + with job: + double.pod(_source(), label="double") + job.run() + message = str(raised.value) + assert "ThirdPartyDatabase cannot store the referential tags ['unit']" in message + assert "keeps_field_metadata" in message + assert "keeps field metadata" in message and "tag_only_columns" in message + assert CALLS == [] + assert inner.list_table_paths() == [] + + +@pytest.mark.parametrize("mode", [CacheMode.LOG, CacheMode.REPLAY]) +def test_an_operator_node_refuses_an_undeclared_database(mode: CacheMode) -> None: + def source(name: str, column: str) -> op.sources.ArrowTableSource: + table = pa.table({"id": [1, 2], column: [1.0, 2.0]}) + return op.sources.ArrowTableSource(table, tag_columns=["id"], source_id=name) + + job = PipelineJob(name="ops", database=InMemoryArrowDatabase()) + with job: + _ = Join()(source("a", "x"), source("b", "z"), label="joined") + inner = InMemoryArrowDatabase() + with pytest.raises(InputValidationError, match="ThirdPartyDatabase cannot store"): + job.nodes["joined"].attach_databases( + pipeline_database=ThirdPartyDatabase(inner), cache_mode=mode + ) + assert inner.list_table_paths() == [] + + +def test_an_extension_aware_wrapper_cannot_pass_an_undeclared_database() -> None: + job = PipelineJob( + database=_extension_aware(ThirdPartyDatabase(InMemoryArrowDatabase())) + ) + with pytest.raises( + InputValidationError, match="ExtensionAwareDatabase cannot store" + ): + with job: + double.pod(_source(), label="double") + job.run() + + +def test_a_declaring_adapter_is_accepted() -> None: + job = PipelineJob(database=_adapter(DeclaringThirdPartyDatabase)) + with job: + double.pod(_source(), label="double") + job.run() + assert job.nodes["double"].keys() == (("unit", "time"), ("unit", "y")) + + +def test_plain_tags_ask_nothing_of_an_undeclared_database() -> None: + job = PipelineJob(database=_adapter(ThirdPartyDatabase)) + plain = op.sources.DictSource( + [{"unit": 1, "amp": 0.5}], + tag_columns=["unit"], + tag_only_columns=True, + source_id="plain", + ) + with job: + double.pod(plain, label="double") + job.run() + assert job.nodes["double"].keys() == (("unit",), ("y",)) + + +# --------------------------------------------------------------------------- +# SpiralDB (pyspiral fake): the marker survives every write path it has +# --------------------------------------------------------------------------- + + +def _rows(ids: list[int], extra: bool = False) -> pa.Table: + columns: dict[str, pa.Array[Any]] = { + "rid": pa.array([str(i).encode() for i in ids], pa.large_binary()), + "unit": pa.array(ids, pa.int64()), + "y": pa.array([float(i) for i in ids], pa.float64()), + } + if extra: + columns["z"] = pa.array(ids, pa.int64()) + return mark_references(pa.Table.from_pydict(columns), ["unit"]) + + +def test_spiraldb_keeps_the_marker_on_every_write_path(spiral: None) -> None: + db = ConnectorArrowDatabase(SpiralDBConnector("p")) + path = ("node", "table") + + def references(database: ConnectorArrowDatabase = db) -> tuple[str, ...]: + table = database.get_all_records(path) + assert table is not None + return stored_references(table.schema) + + db.add_records(path, _rows([1, 2]), record_id_column="rid", flush=True) + assert references() == ("unit",) # first write + db.add_records(path, _rows([3]), record_id_column="rid", flush=True) + assert references() == ("unit",) # append + db.add_records( + path, _rows([3, 4]), record_id_column="rid", skip_duplicates=True, flush=True + ) + assert references() == ("unit",) # some ids stored already: a merge + # Read through a new connector: the marker is stored, not held in memory. + assert references(ConnectorArrowDatabase(SpiralDBConnector("p"))) == ("unit",) + # The connector database does not evolve a stored schema: it refuses, so no merge + # can drop the marker. + with pytest.raises(ValueError, match="Schema evolution is not supported"): + db.add_records(path, _rows([5], extra=True), record_id_column="rid", flush=True) + assert references() == ("unit",) + + +def test_a_referential_job_on_spiraldb_is_refused_as_a_plain_one_is( + spiral: None, +) -> None: + """SpiralDB refuses a function node's reserved columns (ITL-1162), plain or not. + + The marker is not what stops it: the referential job is refused for the same + column as the plain one, before anything is written. + """ + plain = op.sources.DictSource( + [{"unit": 1, "time": 10, "amp": 0.5}], + tag_columns=["unit", "time"], + tag_only_columns=True, + source_id="spikes", + ) + messages = [] + for source in (_source(), plain): + job = PipelineJob(database=ConnectorArrowDatabase(SpiralDBConnector("p"))) + with pytest.raises(UnstorableColumnError) as info: + with job: + double.pod(source, label="double") + messages.append(str(info.value)) + assert all("'__input_data_hash'" in message for message in messages) + + +def test_spiraldb_failed_marker_write_stores_no_row( + spiral: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """The marker is written before the rows: if it fails, no row is stored unmarked.""" + db = ConnectorArrowDatabase(SpiralDBConnector("p")) + path = ("node", "table") + db.add_records(path, _rows([1]), record_id_column="rid", flush=True) + real_set_metadata = fake_pyspiral.FakeTable.set_metadata + + def failing_set_metadata(self: Any, metadata: dict[str, bytes]) -> None: + # Only the Arrow metadata entry (the marker) fails. + if "__arrow_metadata__" in metadata: + raise RuntimeError("metadata service unavailable") + real_set_metadata(self, metadata) + + monkeypatch.setattr(fake_pyspiral.FakeTable, "set_metadata", failing_set_metadata) + with pytest.raises(RuntimeError, match="metadata service unavailable"): + db.add_records( + path, + _rows([2, 3]), + record_id_column="rid", + skip_duplicates=True, + flush=True, + ) + monkeypatch.setattr(fake_pyspiral.FakeTable, "set_metadata", real_set_metadata) + stored = ConnectorArrowDatabase(SpiralDBConnector("p")).get_all_records(path) + assert stored is not None and stored.column("unit").to_pylist() == [1] + + # The retry writes the rows and their marker. + db.add_records( + path, _rows([2, 3]), record_id_column="rid", skip_duplicates=True, flush=True + ) + table = ConnectorArrowDatabase(SpiralDBConnector("p")).get_all_records(path) + assert table is not None and set(table.column("unit").to_pylist()) == {1, 2, 3} + assert stored_references(table.schema) == ("unit",) + + +def test_spiraldb_write_of_stored_rows_keeps_the_stored_metadata(spiral: None) -> None: + """A write that stores no row never replaces a metadata entry already stored.""" + db = ConnectorArrowDatabase(SpiralDBConnector("p")) + path = ("node", "table") + db.add_records(path, _rows([1, 2]), record_id_column="rid", flush=True) + # The same rows, with other field metadata: ``unit`` unmarked, ``y`` annotated. + rows = _rows([1, 2]) + other = rows.cast( + pa.schema( + [ + field.with_metadata({"note": "other"}) + if field.name == "y" + else field.remove_metadata() + for field in rows.schema + ] + ) + ) + db.add_records( + path, other, record_id_column="rid", skip_duplicates=True, flush=True + ) + table = ConnectorArrowDatabase(SpiralDBConnector("p")).get_all_records(path) + assert table is not None and stored_references(table.schema) == ("unit",) + + +def test_spiraldb_marks_no_stored_row_it_skips(spiral: None) -> None: + """Rows stored unmarked stay unmarked when the same ids are written marked.""" + db = ConnectorArrowDatabase(SpiralDBConnector("p")) + path = ("node", "table") + rows = _rows([1, 2]) + unmarked = rows.cast(pa.schema([field.remove_metadata() for field in rows.schema])) + db.add_records(path, unmarked, record_id_column="rid", flush=True) + db.add_records(path, rows, record_id_column="rid", skip_duplicates=True, flush=True) + table = ConnectorArrowDatabase(SpiralDBConnector("p")).get_all_records(path) + assert table is not None and stored_references(table.schema) == () + + +def test_spiraldb_failed_first_write_leaves_no_marker( + spiral: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """A first write whose rows fail removes the metadata it published before them.""" + connector = SpiralDBConnector("p") + connector.create_table_if_not_exists( + "t", [ColumnInfo("rid", pa.large_binary())], pk_column="rid" + ) + real_write = fake_pyspiral.FakeTable.write + + def failing_write(self: Any, *args: Any, **kwargs: Any) -> None: + raise RuntimeError("write failed") + + monkeypatch.setattr(fake_pyspiral.FakeTable, "write", failing_write) + with pytest.raises(RuntimeError, match="write failed"): + connector.upsert_records("t", _rows([1]), id_column="rid") + monkeypatch.setattr(fake_pyspiral.FakeTable, "write", real_write) + + rows = _rows([1]) + unmarked = rows.cast(pa.schema([field.remove_metadata() for field in rows.schema])) + connector.upsert_records("t", unmarked, id_column="rid") + table = pa.Table.from_batches(list(connector.iter_batches('SELECT * FROM "t"'))) + assert stored_references(table.schema) == () + + +def test_spiraldb_failed_write_keeps_the_metadata_of_stored_rows( + spiral: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """A failed write restores the metadata entry the stored rows had, even key-only.""" + connector = SpiralDBConnector("p") + connector.create_table_if_not_exists( + "k", [ColumnInfo("rid", pa.large_binary())], pk_column="rid" + ) + + def key_only(ids: list[int]) -> pa.Table: + return mark_references( + pa.table( + {"rid": pa.array([str(i).encode() for i in ids], pa.large_binary())} + ), + ["rid"], + ) + + connector.upsert_records("k", key_only([1]), id_column="rid") + real_write = fake_pyspiral.FakeTable.write + + def failing_write(self: Any, *args: Any, **kwargs: Any) -> None: + raise RuntimeError("write failed") + + monkeypatch.setattr(fake_pyspiral.FakeTable, "write", failing_write) + with pytest.raises(RuntimeError, match="write failed"): + connector.upsert_records("k", key_only([2]), id_column="rid") + monkeypatch.setattr(fake_pyspiral.FakeTable, "write", real_write) + + table = pa.Table.from_batches(list(connector.iter_batches('SELECT * FROM "k"'))) + assert table.column("rid").to_pylist() == [b"1"] + assert stored_references(table.schema) == ("rid",) + + +def test_spiraldb_failed_metadata_read_leaves_no_nullability_record( + spiral: None, monkeypatch: pytest.MonkeyPatch +) -> None: + """A metadata read that fails after the record was written leaves no stale record.""" + from orcapod.databases.spiraldb_nullability import RECORD_KEY + + connector = SpiralDBConnector("p") + connector.create_table_if_not_exists( + "t", [ColumnInfo("rid", pa.large_binary())], pk_column="rid" + ) + real_get_metadata = fake_pyspiral.FakeTable.get_metadata + real_set_metadata = fake_pyspiral.FakeTable.set_metadata + record_written: list[bool] = [] + + def tracking_set_metadata(self: Any, metadata: dict[str, bytes]) -> None: + real_set_metadata(self, metadata) + if any(key.startswith(RECORD_KEY) for key in metadata): + record_written.append(True) + + def failing_get_metadata(self: Any) -> dict[str, bytes]: + # Every read after the nullability record is written fails. + if record_written: + raise RuntimeError("metadata read failed") + return real_get_metadata(self) + + monkeypatch.setattr(fake_pyspiral.FakeTable, "set_metadata", tracking_set_metadata) + monkeypatch.setattr(fake_pyspiral.FakeTable, "get_metadata", failing_get_metadata) + try: + connector.upsert_records("t", _rows([1]), id_column="rid") + wrote = True + except RuntimeError: + wrote = False + monkeypatch.undo() + + table = connector._project.table(connector._table_id("t")) # pyright: ignore[reportPrivateUsage] + if not wrote: + assert not any(key.startswith(RECORD_KEY) for key in table.get_metadata()) + else: + rows = pa.Table.from_batches(list(connector.iter_batches('SELECT * FROM "t"'))) + assert rows.num_rows == 1 diff --git a/tests/test_core/referential_tags/test_derived_source.py b/tests/test_core/referential_tags/test_derived_source.py new file mode 100644 index 000000000..1a7176ba8 --- /dev/null +++ b/tests/test_core/referential_tags/test_derived_source.py @@ -0,0 +1,294 @@ +"""A derived source carries the bindings of what it reads (ITL-1019, gap audit S1 and Q7). + +``node.as_source()`` over a function or operator node whose stored records hold referential +tags presents the bindings the stored table states (the marker of ``referential_tags``), +checked against the node's keys, in the plain form and the ``tags_as_data`` form. Nothing is +added to the identity for the binding: it is in the origin's content hash and in the output +schema, where a referential tag is in both halves. +""" + +from __future__ import annotations + +import warnings +from collections.abc import Callable +from pathlib import Path + +import pyarrow as pa +import pytest + +import orcapod as op +from orcapod.core.operators import Join +from orcapod.core.referential_tags import ( + REFERENTIAL_TAG_METADATA_KEY, + ReferenceMaterializationWarning, + mark_references, + without_reference_marks, +) +from orcapod.core.sources import DerivedSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.errors import InputValidationError +from orcapod.pipeline import ( + AsyncPipelineOrchestrator, + PipelineJob, + SyncPipelineOrchestrator, +) +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.types import CacheMode + +JOB = "derived" +REFERENTIAL_KEYS = (("unit", "time"), ("unit", "y")) + + +def _in_memory(_: Path) -> ArrowDatabaseProtocol: + return InMemoryArrowDatabase() + + +def _delta(tmp_path: Path) -> ArrowDatabaseProtocol: + return DeltaTableDatabase(tmp_path / "store") + + +STORES = pytest.mark.parametrize( + "make_store", [_in_memory, _delta], ids=["memory", "delta"] +) +ORCHESTRATORS = pytest.mark.parametrize( + "orchestrator", + [SyncPipelineOrchestrator, AsyncPipelineOrchestrator], + ids=["sync", "async"], +) +Orchestrator = type[SyncPipelineOrchestrator | AsyncPipelineOrchestrator] + + +@op.function_pod(output_keys=["y"]) +def double(amp: float) -> float: + """Double the amplitude; reads no tag, so ``unit`` falls through.""" + return 2 * amp + + +@op.function_pod(output_keys=["w"]) +def scale(unit: int, y: float) -> float: + """Reads the referential tag ``unit`` as data.""" + return unit * y + + +def _source(tag_only: object = ("time",), rows: int = 2) -> op.sources.DictSource: + """``unit`` referential and ``time`` plain unless ``tag_only`` says otherwise.""" + return op.sources.DictSource( + [{"unit": i, "time": 10 * i, "amp": i - 0.5} for i in range(1, rows + 1)], + tag_columns=["unit", "time"], + tag_only_columns=tag_only, # pyright: ignore[reportArgumentType] # the test varies it + source_id="spikes", + ) + + +def _function_job( + store: ArrowDatabaseProtocol, + orchestrator: Orchestrator = SyncPipelineOrchestrator, + tag_only: object = ("time",), + run: bool = True, +) -> PipelineJob: + job = PipelineJob(name=JOB, database=store) + with job: + double.pod(_source(tag_only), label="double") + if run: + job.run(orchestrator=orchestrator()) + return job + + +def _operator_job( + store: ArrowDatabaseProtocol, orchestrator: Orchestrator = SyncPipelineOrchestrator +) -> PipelineJob: + """A ``LOG`` join of the function node's output with itself filtered by ``unit``.""" + job = PipelineJob(name=JOB, database=store) + with job: + doubled = double.pod(_source(), label="double") + _ = Join()(doubled, _source().select_data_columns(["unit"]), label="joined") + job.nodes["joined"].attach_databases( + pipeline_database=store.at(JOB), cache_mode=CacheMode.LOG + ) + job.run(orchestrator=orchestrator()) + return job + + +def _marked(schema: pa.Schema) -> list[str]: + return [ + f.name for f in schema if REFERENTIAL_TAG_METADATA_KEY in (f.metadata or {}) + ] + + +def _by_unit(table: pa.Table) -> list[dict[str, object]]: + return sorted(table.to_pylist(), key=lambda row: str(row["unit"])) + + +# --------------------------------------------------------------------------- +# The plain form presents the bindings the stored table states +# --------------------------------------------------------------------------- + + +@STORES +@ORCHESTRATORS +def test_as_source_of_a_function_node_keeps_its_bindings( + tmp_path: Path, + make_store: Callable[[Path], ArrowDatabaseProtocol], + orchestrator: Orchestrator, +) -> None: + job = _function_job(make_store(tmp_path), orchestrator) + node = job.nodes["double"] + source = node.as_source() + assert source.keys() == node.keys() == REFERENTIAL_KEYS + assert source.output_schema() == node.output_schema() + table = source.as_table(all_info=True) + assert table.column_names.count("unit") == 1 + assert _marked(table.schema) == [] + assert _by_unit(source.as_table()) == _by_unit(node.as_table()) + for tag, data in source.iter_data(): + assert tag["unit"] == data["unit"] + head = source.head() + assert head.table.column_names.count("unit") == 1 + + +@STORES +@ORCHESTRATORS +def test_as_source_of_an_operator_node_keeps_its_bindings( + tmp_path: Path, + make_store: Callable[[Path], ArrowDatabaseProtocol], + orchestrator: Orchestrator, +) -> None: + with warnings.catch_warnings(): + warnings.simplefilter("ignore", ReferenceMaterializationWarning) + job = _operator_job(make_store(tmp_path), orchestrator) + node = job.nodes["joined"] + source = node.as_source() + assert source.keys() == node.keys() + assert "unit" in source.keys()[0] and "unit" in source.keys()[1] + assert _by_unit(source.as_table()) == _by_unit(node.as_table()) + assert _marked(source.as_table(all_info=True).schema) == [] + + +def test_a_derived_source_before_its_node_ran_has_the_binding_and_no_rows() -> None: + job = _function_job(InMemoryArrowDatabase(), run=False) + source = job.nodes["double"].as_source() + assert source.keys() == REFERENTIAL_KEYS + assert source.as_table().num_rows == 0 + assert source.as_table().column_names.count("unit") == 1 + + +def test_a_derived_source_refuses_records_that_disagree_with_the_node() -> None: + written = InMemoryArrowDatabase() + _function_job(written) + tampered = InMemoryArrowDatabase() + for path in written.list_table_paths(): + table = written.get_all_records(path, record_id_column="__copy_id") + assert table is not None + tampered.add_records( + path, + without_reference_marks(table), + record_id_column="__copy_id", + flush=True, + ) + source = _function_job(tampered, run=False).nodes["double"].as_source() + # The node checks its own table first and names it. + with pytest.raises( + InputValidationError, + match=r"pipeline table .* marks the referential tags \[\], but the node's pod " + r"declares \['unit'\]", + ): + _ = source.as_table() + + +def test_a_derived_source_checks_the_records_it_reads_itself( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The derived source reads the binding from the records, not from the node alone.""" + node = _function_job(InMemoryArrowDatabase()).nodes["double"] + records = node.get_all_records() + assert records is not None + # Records that mark one tag more than the node shares between tag and data. + wrong = mark_references(records, ["unit", "time"]) + monkeypatch.setattr(node, "get_all_records", lambda *_, **__: wrong) + with pytest.raises( + InputValidationError, + match=r"the stored records mark the referential tags \['time', 'unit'\], but " + r"the node declares \['unit'\]", + ): + _ = node.as_source().as_table() + + +# --------------------------------------------------------------------------- +# The tags_as_data form +# --------------------------------------------------------------------------- + + +@ORCHESTRATORS +def test_tags_as_data_over_a_referential_node(orchestrator: Orchestrator) -> None: + job = _function_job(InMemoryArrowDatabase(), orchestrator) + node = job.nodes["double"] + source = node.as_source(tags_as_data={"time": "start", "unit": "unit_copy"}) + # The referential tag keeps its binding; each copy is a new plain data column. + assert source.keys() == (("unit", "time"), ("unit", "y", "start", "unit_copy")) + table = source.as_table(all_info=True) + assert table.column_names.count("unit") == 1 + assert _marked(table.schema) == [] + # Fresh provenance: every data column, the referential one included, gets a token. + assert {"_source_unit", "_source_y", "_source_start", "_source_unit_copy"} <= set( + table.column_names + ) + for row in source.as_table().to_pylist(): + assert row["unit_copy"] == row["unit"] and row["start"] == 10 * row["unit"] + + +def test_tags_as_data_over_a_referential_source() -> None: + source = DerivedSource(_source(), tags_as_data={"time": "start"}) + assert source.keys() == (("unit", "time"), ("unit", "amp", "start")) + assert source.as_table().column_names.count("unit") == 1 + + +# --------------------------------------------------------------------------- +# Identity: the binding enters through what is already hashed +# --------------------------------------------------------------------------- + + +def test_the_binding_enters_the_identity_through_the_origin_and_the_schema() -> None: + referential = _function_job(InMemoryArrowDatabase()).nodes["double"].as_source() + plain = ( + _function_job(InMemoryArrowDatabase(), tag_only=True) + .nodes["double"] + .as_source() + ) + assert plain.keys() == (("unit", "time"), ("y",)) + # Plain form: the content hash is the origin node's, which carries the binding. + assert referential.identity_structure() == ( + referential._origin.content_hash(), # pyright: ignore[reportPrivateUsage] # the identity rule under test + ) + assert referential.content_hash() != plain.content_hash() + # Pipeline hash: schema only; a referential tag is in both halves of it. + assert referential.pipeline_identity_structure() == referential.output_schema() + assert referential.pipeline_hash() != plain.pipeline_hash() + # The default source id keeps its construction: derived: