diff --git a/changelog.d/ITL-1162.md b/changelog.d/ITL-1162.md new file mode 100644 index 000000000..cfcd589f6 --- /dev/null +++ b/changelog.d/ITL-1162.md @@ -0,0 +1,11 @@ +### Fixed + +- A `ConnectorArrowDatabase` over `SQLiteConnector` or `PostgreSQLConnector` now refuses a table it cannot store before anything of it is written, with `UnstorableColumnError` naming the connector, the column, its Arrow type and what to use instead (ITL-1162). A connector stores a column only when it reads it back as written: on SQLite `bool`, `int64`, `double`, `large_string` and `large_binary`; on PostgreSQL also `int32`, `float`, `date32`, `timestamp[us]` and `timestamp[us, tz=UTC]`. Before, a function node on such a database ran its function and then failed at commit (`sqlite3.ProgrammingError: ... type 'list' is not supported`), timestamps were stored as text and read back as strings, and narrower integers and floats read back widened, failing the next write. A job now refuses a node whose stored table such a database cannot hold (every function node; a side-effect node on SQLite) when the job is built, before anything runs, naming the node. The connectors no longer fall back to a `TEXT` column for a type they have no column type for. +- `PostgreSQLConnector` now refuses a column name longer than 63 bytes (UTF-8), which PostgreSQL truncates, so it would read back as another name (ITL-1162). Before, such a column was stored under the truncated name. A table name longer than 63 bytes is refused too, with `UnstorableTableNameError`, at every entry that creates, writes, reads or checks a table, so a truncated name is never sent: two tables whose names share their first 63 bytes would otherwise silently be one. `SQLiteConnector` refuses a table name starting with `sqlite_`, which SQLite reserves. +- A `ConnectorArrowDatabase` over `SpiralDBConnector` now refuses, by the same rule, a column SpiralDB cannot store: a value column whose name starts with `__` (the `__record_id` key is allowed) or any column whose name contains `.`, and a timestamp column with a time zone, which SpiralDB reads back without it (ITL-1164). Before, a function node on such a database ran and then failed at commit (`Column name '__input_data_hash' has reserved prefix`). A job refuses every function node, side-effect node, operator cache and cached source on such a database when it is built. + +### Known limitations + +- A connector job database cannot hold function nodes in v0.3.0: neither SQL connector (SQLite, PostgreSQL) nor SpiralDB. A function node's result table holds list-of-struct executor columns, which neither SQL connector stores, and `__`-prefixed value columns, which SpiralDB reserves (ITL-1164). A side-effect node is refused on SQLite (its log's UTC timestamp) and on SpiralDB, and an operator cache or cached source on SpiralDB (`__data_context`). Use a Delta database (`DeltaTableDatabase`) or `InMemoryArrowDatabase`. Storing such columns on connector databases is planned for a later release (ITL-1162, ITL-1164). +- On PostgreSQL, an operator cache with caching on is refused in v0.3.0 when it is written (before anything is stored): its system-tag column names (`_tag_source_id::` and a 64-character hash) are longer than PostgreSQL's 63-byte limit. Storing them under a mapped name is planned for a later release (ITL-1166). +- On PostgreSQL in v0.3.0, a table whose Orcapod path gives a name longer than 63 bytes is refused. Every node table and cached source path holds a 64-character hash, so only tables written directly through `ConnectorArrowDatabase` under short paths fit; reading an existing table with `PostgreSQLTableSource` is unaffected. Mapping long names is planned for a later release (ITL-1166). diff --git a/design_issues/ITL-1162-CA3.md b/design_issues/ITL-1162-CA3.md new file mode 100644 index 000000000..73e1c1ee6 --- /dev/null +++ b/design_issues/ITL-1162-CA3.md @@ -0,0 +1,42 @@ +## `src/orcapod/databases/connector_arrow_database.py` + +### CA3 — Connector databases cannot store Orcapod's own stored tables (list, struct and timestamp columns; SpiralDB reserved names) +**Status:** open +**Severity:** high +**Issue:** ITL-1162 + +A `ConnectorArrowDatabase` over `SQLiteConnector` or `PostgreSQLConnector` cannot store a +function node's result table. Its `__pf_exec_executor_info` and `__pf_exec_extra_info` +columns are `large_list>`. The SQLite connector logged "Unsupported Arrow +type ...; mapping to TEXT" and then failed to bind a Python list (`sqlite3.ProgrammingError`) +at commit, after the function had run. The same table's `__pod_ts` (`timestamp[us, tz=UTC]`) +and a side-effect log's `executed_at` were stored as TEXT and read back as `large_string`. +`int8`, `int32` and `float32` columns read back widened, so the second write to the table +failed its schema comparison. PostgreSQL raises for list columns at `CREATE TABLE` and has no +column type for structs, and truncates a column name to 63 bytes, so an operator cache's +system-tag columns (`_tag_source_id::<64-hex>`) read back under other names. Its table names +(the node path joined by `__`, 109-173 characters) are truncated the same way, so two tables +sharing their first 63 bytes would silently be one. SpiralDB refuses value columns named with a `__` prefix and any column +name containing `.`, and drops a timestamp's time zone (ITL-1164), so it cannot hold a function +result table (`__input_data_hash`, ...) or any table's `__data_context`. + +**Fix (refusal; developer ruling, November 16, 2026):** each connector states which columns +it stores, in one place (`DBConnectorProtocol.unstorable_column_reason(name, arrow_type, +primary_key=...)`): a column it reads back as written, with the same Arrow type in its large +variant. `ConnectorArrowDatabase.require_storable` raises `UnstorableColumnError` (connector, +column, Arrow type, reason, what to use instead) in `add_records`, before anything is +buffered. A function job node (its result table) and a side-effect job node (its invocation +log) ask the database when it is attached, so a job refuses them before anything runs, naming +the node. An operator job node with a cache and a cached source ask about their bookkeeping +columns (`__node_content_hash`, `__data_context`) when attached or built; their user columns +are checked at the write. `SpiralDBConnector` states its name rule the same way (developer +ruling, November 16, 2026). `PostgreSQLConnector` also states which table names it holds +(`unstorable_table_name_reason`, 63 bytes), checked at every table entry +(`UnstorableTableNameError`). The connectors' silent TEXT fallback in `CREATE TABLE` now +raises. + +**Open (the real fix, a later release):** store list and struct columns (for example as JSON +text read back as the same Arrow type) and timestamps on SQLite, and rename SpiralDB's +reserved column names on write and recover them on read, and restore a timestamp's time +zone (ITL-1164); map PostgreSQL column and table names over 63 bytes the same way (ITL-1166), so a function node can +run on a connector database. diff --git a/docs/concepts/pipeline-job.md b/docs/concepts/pipeline-job.md index f5bca8daa..745e875ce 100644 --- a/docs/concepts/pipeline-job.md +++ b/docs/concepts/pipeline-job.md @@ -120,6 +120,60 @@ run and computes nothing on a second `run()`. A job created without a database can be recorded but not run. Give it one later with `job.bind(database=...)`; `run()` without a database raises `ValueError`. +### Connector databases + +A `ConnectorArrowDatabase` stores a column only when its connector can store it as written. +Each connector states which columns those are, and anything else is refused with +`UnstorableColumnError` before anything of the table is written. + +The SQL connectors store a column only when they read it back with the same Arrow type (text +and binary columns come back as `large_string` and `large_binary`, the types Orcapod uses): + +| Connector | Column types it stores | +|---|---| +| `SQLiteConnector` | `bool`, `int64`, `double`, `large_string`, `large_binary` | +| `PostgreSQLConnector` | `bool`, `int32`, `int64`, `float`, `double`, `large_string`, `large_binary`, `date32`, `timestamp[us]`, `timestamp[us, tz=UTC]` | + +Any other type (lists, structs, dates or timestamps on SQLite, narrower integers and floats, +logical types) is refused. PostgreSQL also refuses a column name or a table name longer than +63 bytes, which it would truncate (`UnstorableTableNameError` for a table). `SpiralDBConnector` stores other types as written, except a +timestamp with a time zone (SpiralDB drops the zone), and SpiralDB reserves the `__` prefix on +value columns (the `__record_id` key is allowed) and `.` in any column name. + +Orcapod's own stored tables hold such columns. A function node's result table holds +list-valued executor columns and `__`-prefixed columns. A side-effect node's log holds a UTC +timestamp and `__data_context`, and an operator cache and a cached source hold +`__data_context` too. So a job refuses these nodes when it is built, before anything runs: + +- every function node, on all three connectors; +- every side-effect node, on SQLite and SpiralDB; +- every operator cache (an operator node with caching on) and cached source, on SpiralDB. + +An operator cache or cached source on a SQL connector is refused when it is written, before +anything is stored, if one of its columns has a type or name the connector does not store. On +PostgreSQL that is every operator cache with caching on: its system-tag column names +(`_tag_source_id::` and a 64-character hash) are longer than 63 bytes. +A table's name is its Orcapod path joined by `__`, and every node table and cached source path +holds a 64-character hash, so on PostgreSQL only tables you write through `ConnectorArrowDatabase` +yourself under short paths fit; reading an existing table with `PostgreSQLTableSource` is +unaffected. Use +`InMemoryArrowDatabase` or `DeltaTableDatabase` for such jobs. Storing these columns on +connector databases is planned for a later release. + +```python +from orcapod.databases import ConnectorArrowDatabase, SQLiteConnector +from orcapod.errors import UnstorableColumnError + +sqlite_database = ConnectorArrowDatabase(SQLiteConnector(":memory:")) +sql_job = PipelineJob(name="bmi", database=sqlite_database) +try: + with sql_job: + compute_bmi.pod(mice(cohort_a), label="bmi") +except UnstorableColumnError as exc: + print(exc) +# node 'bmi': a SQLiteConnector database cannot store column '__pf_exec_executor_info' of Arrow type large_list<...>: SQLite has no column type for it. Use an InMemoryArrowDatabase or a DeltaTableDatabase instead. +``` + ## What a re-run computes A stored result is reused when the function's identity (its name, its output schema and its diff --git a/src/orcapod/contexts/stored.py b/src/orcapod/contexts/stored.py index e0ae1a58f..ecb5b4f4f 100644 --- a/src/orcapod/contexts/stored.py +++ b/src/orcapod/contexts/stored.py @@ -108,11 +108,16 @@ def with_data_context(table: "pa.Table", context_key: str) -> "pa.Table": ``large_string`` holding ``context_key`` on every row. """ return table.append_column( - pa.field(constants.DATA_CONTEXT_COL, pa.large_string(), nullable=False), + data_context_field(), pa.array([context_key] * table.num_rows, type=pa.large_string()), ) +def data_context_field() -> "pa.Field[pa.DataType]": + """The ``__data_context`` field every stored table ends with (``with_data_context``).""" + return pa.field(constants.DATA_CONTEXT_COL, pa.large_string(), nullable=False) + + def require_same_data_context(input_context_key: str, own_context_key: str) -> None: """Refuse an input that was hashed under another data context than the writer's. @@ -681,3 +686,23 @@ def stored_data_contexts( keys = cast("list[str]", per_value.field(0).to_pylist()) counts = cast("list[int]", per_value.field(1).to_pylist()) return dict(zip(keys, counts)) + + +def require_storable(database: object, schema: pa.Schema) -> None: + """Refuse ``schema`` if ``database`` states it cannot store one of its columns. + + The one check for ``ColumnCheckingDatabaseProtocol`` (ITL-1162): a job node asks + it about each table it stores when a database is attached, before anything runs. + A database that states no rule (in-memory, Delta) stores every column, and + nothing is checked. + + Args: + database: Any database object. + schema: The Arrow schema of a table the database is to hold. + + Raises: + UnstorableColumnError: If the database cannot store a column of ``schema``. + """ + check = getattr(database, "require_storable", None) + if callable(check): + _ = check(schema) diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index ea1fea3bf..0ca3094e7 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -8,6 +8,7 @@ from collections.abc import Iterator from typing import TYPE_CHECKING, Any +from orcapod.core.data_function import CachedDataFunction from orcapod.core.datagrams import Datagram from orcapod.core.function_pod import ( FunctionPodStream, @@ -17,8 +18,12 @@ from orcapod.core.streams.view import StreamView from orcapod.system_constants import constants from orcapod.types import ColumnConfig, Schema -from orcapod.contexts.stored import StaleTables -from orcapod.core.result_cache import ResultCache +from orcapod.contexts.stored import ( + StaleTables, + data_context_field, + require_storable, +) +from orcapod.core.result_cache import ResultCache, pod_timestamp_type from orcapod.hooks import InvocationStatus from orcapod.protocols.core_protocols import ( FunctionPodProtocol, @@ -71,6 +76,53 @@ def __init__( name_path=self.uri[: 2 if function_pod.ctx_arg_name is not None else 1], ) + def require_storable_results(self) -> None: + """Refuse a result database that cannot store this pod's result table (ITL-1162). + + The result table holds the input data hash, the function's variation and + execution columns (``__pf_var_*``, ``__pf_exec_*``), its output columns, + the ``__pod_ts`` timestamp and ``__data_context``, typed from their declared + schemas; nothing is computed. (The source-info columns are text.) A + database that states which columns it stores (``ConnectorArrowDatabase``) + refuses here, before the function runs; any other database accepts. + + Raises: + UnstorableColumnError: If the result database cannot store a column. + """ + import pyarrow as pa + + pf = self._function_pod.data_function + python_schema = { + **{ + f"{constants.PF_VARIATION_PREFIX}{name}": hint + for name, hint in pf.get_function_variation_data_schema().items() + }, + **{ + f"{constants.PF_EXECUTION_PREFIX}{name}": hint + for name, hint in pf.get_execution_data_schema().items() + }, + **pf.output_data_schema, + } + declared = self.data_context.type_converter.python_schema_to_arrow_schema( + python_schema + ) + schema = pa.schema( + [ + pa.field(constants.INPUT_DATA_HASH_COL, pa.large_binary()), + *declared, + pa.field(constants.POD_TIMESTAMP, pod_timestamp_type()), + data_context_field(), + ] + ) + require_storable(self._cache.result_database, schema) + # A function with its own result database (``function_pod(result_database=)``) + # stores the same table there. + if isinstance(pf, CachedDataFunction): + require_storable(pf.result_database, schema) + # A pod with its own cache database (``pod_cache_database``) stores there too. + if isinstance(self._function_pod, CachedFunctionPod): + self._function_pod.require_storable_results() + @property def result_database(self) -> ArrowDatabaseProtocol: """The underlying result database.""" diff --git a/src/orcapod/core/data_function.py b/src/orcapod/core/data_function.py index 83c4a7fd1..4c6213d74 100644 --- a/src/orcapod/core/data_function.py +++ b/src/orcapod/core/data_function.py @@ -1115,6 +1115,11 @@ def __init__( name_path=record_path_prefix + self.uri[:1], ) + @property + def result_database(self) -> ArrowDatabaseProtocol: + """The database this function's results are cached in.""" + return self._cache.result_database + def _build_metadata_datagrams(self) -> tuple[Datagram, Datagram]: """Build variation and execution Datagrams for cache storage.""" variation_datagram = Datagram( diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 160c97af5..033157f65 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -56,6 +56,7 @@ TypeIdentityError, OrcapodDatabaseReadError, OrcapodDatabaseWriteError, + UnstorableColumnError, ) from orcapod.contexts.stored import ( StaleTables, @@ -66,6 +67,7 @@ find_stale_tables, report_rows_of_other_data_contexts, report_unexpected_row_count, + require_storable, ) from orcapod.utils.database_errors import ( commit_best_effort, @@ -125,6 +127,28 @@ _PIPELINE_RECOMPUTATION_INDEX_COL = "__pipeline_recomputation_index" +def _pipeline_meta_schema() -> pa.Schema: + """The bookkeeping columns of a function node's pipeline record (``pdb_v2``). + + The record holds the tags, system tags and input source-info columns, then + these. One definition for the writer (``add_pipeline_record``) and for the + check of the pipeline database when it is attached (ITL-1162). + """ + return pa.schema( + [ + pa.field(constants.DATA_RECORD_ID, pa.large_binary()), + pa.field(constants.NODE_CONTENT_HASH_COL, pa.large_binary()), + pa.field(constants.INPUT_DATA_HASH_COL, pa.large_binary()), + pa.field(constants.OUTPUT_DATA_HASH_COL, pa.large_binary()), + pa.field(constants.DATA_CONTEXT_COL, pa.large_string()), + pa.field(f"{constants.META_PREFIX}computed", pa.bool_()), + pa.field(constants.IS_EPHEMERAL_COL, pa.bool_()), + pa.field(_PIPELINE_BASE_ENTRY_ID_COL, pa.large_binary()), + pa.field(_PIPELINE_RECOMPUTATION_INDEX_COL, pa.int32()), + ] + ) + + def _build_record_id_preimage( tag: "TagProtocol", input_data: "DataProtocol", @@ -830,6 +854,30 @@ def __repr__(self) -> str: f"input_stream={short_stream_repr(self._input_stream)})" ) + def require_storable_results(self, result_database: ArrowDatabaseProtocol) -> None: + """Refuse a result database that cannot store this node's result table (ITL-1162). + + Nothing is computed and nothing changes. A node without its function stores + no results and is not checked. + + Args: + result_database: The database results would be stored in (the + persistent result database or an ephemeral one). + + Raises: + UnstorableColumnError: If the database cannot store a column, naming + this node. + """ + if self._function_pod is None: + return + try: + CachedFunctionPod( + self._function_pod, result_database=result_database + ).require_storable_results() + except UnstorableColumnError as exc: + exc.add_context(node_label=self.label) + raise + @abstractmethod def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: """Assign or remove the ephemeral result database for this node. @@ -1364,6 +1412,38 @@ def _account_for_stale_tables(self) -> None: # attach_databases # ------------------------------------------------------------------ + def require_storable_tables( + self, + pipeline_database: ArrowDatabaseProtocol, + result_database: ArrowDatabaseProtocol | None = None, + ) -> None: + """Refuse databases that cannot store this node's tables (ITL-1162). + + Checks the result table (``CachedFunctionPod.require_storable_results``) + against ``result_database`` and the pipeline records' own columns against + ``pipeline_database``. Nothing is computed and nothing changes. A node + loaded without its function stores nothing and is not checked. + + Args: + pipeline_database: The database for pipeline records. + result_database: The database for results; defaults to + ``pipeline_database.at("_result")``, as in ``attach_databases``. + + Raises: + UnstorableColumnError: If a database cannot store a column, naming + this node. + """ + if self._function_pod is None: + return + if result_database is None: + result_database = pipeline_database.at("_result") + self.require_storable_results(result_database) + try: + require_storable(pipeline_database, _pipeline_meta_schema()) + except UnstorableColumnError as exc: + exc.add_context(node_label=self.label) + raise + def attach_databases( self, pipeline_database: ArrowDatabaseProtocol | None = None, @@ -1400,6 +1480,12 @@ def attach_databases( ``set_ephemeral_database(ephemeral_database)`` after ``attach_databases()``. """ + if pipeline_database is not None: + # A database that cannot store this node's tables refuses them now, + # before anything runs or changes (ITL-1162). + self.require_storable_tables(pipeline_database, result_database) + if ephemeral_database is not None: + self.require_storable_results(ephemeral_database) # Another database has its own tables of other generations to account for. self._stale_tables_checked = False self._current_tables_checked = False @@ -1871,6 +1957,10 @@ def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> No database: The ``ArrowDatabaseProtocol`` to use for ephemeral result storage, or ``None`` to detach and revert to persistent-only writes. """ + if database is not None: + # A database that cannot store the result table refuses it now, before + # anything runs or changes (ITL-1162). + self.require_storable_results(database) # Another database has its own tables of other generations to account for, # as in ``attach_databases``. self._stale_tables_checked = False @@ -2844,40 +2934,27 @@ def _add_pipeline_record( input_source_table = input_table_with_source.select(source_col_names) # Build the meta columns table. - meta_table = pa.table( + meta_table = pa.Table.from_pydict( { - constants.DATA_RECORD_ID: pa.array( - [data_record_id.bytes], type=pa.large_binary() - ), - constants.NODE_CONTENT_HASH_COL: pa.array( - [self.content_hash().to_prefixed_digest()], type=pa.large_binary() - ), - constants.INPUT_DATA_HASH_COL: pa.array( - [input_data.content_hash().to_prefixed_digest()], - type=pa.large_binary(), - ), - constants.OUTPUT_DATA_HASH_COL: pa.array( - [ - output_data_hash.to_prefixed_digest() - if output_data_hash is not None - else None - ], - type=pa.large_binary(), - ), - constants.DATA_CONTEXT_COL: pa.array( - [self._stored_data_context_key], type=pa.large_string() - ), - f"{constants.META_PREFIX}computed": pa.array( - [computed], type=pa.bool_() - ), - constants.IS_EPHEMERAL_COL: pa.array([is_ephemeral], type=pa.bool_()), - _PIPELINE_BASE_ENTRY_ID_COL: pa.array( - [base_entry_id], type=pa.large_binary() - ), - _PIPELINE_RECOMPUTATION_INDEX_COL: pa.array( - [new_index], type=pa.int32() - ), - } + constants.DATA_RECORD_ID: [data_record_id.bytes], + constants.NODE_CONTENT_HASH_COL: [ + self.content_hash().to_prefixed_digest() + ], + constants.INPUT_DATA_HASH_COL: [ + input_data.content_hash().to_prefixed_digest() + ], + constants.OUTPUT_DATA_HASH_COL: [ + output_data_hash.to_prefixed_digest() + if output_data_hash is not None + else None + ], + constants.DATA_CONTEXT_COL: [self._stored_data_context_key], + f"{constants.META_PREFIX}computed": [computed], + constants.IS_EPHEMERAL_COL: [is_ephemeral], + _PIPELINE_BASE_ENTRY_ID_COL: [base_entry_id], + _PIPELINE_RECOMPUTATION_INDEX_COL: [new_index], + }, + schema=_pipeline_meta_schema(), ) # Combine: tag (with system tags) + input source columns + meta columns diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 15ca8ce54..5139d84a4 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -50,8 +50,11 @@ PipelineJobRequiredError, OrcapodDatabaseReadError, OrcapodDatabaseWriteError, + UnstorableColumnError, ) from orcapod.contexts.stored import ( + data_context_field, + require_storable, keep_rows_of_data_context, require_same_data_context, account_for_stale_tables, @@ -832,6 +835,41 @@ def clear_cache(self) -> None: # attach_databases # ------------------------------------------------------------------ + def require_storable_tables( + self, + pipeline_database: ArrowDatabaseProtocol, + cache_mode: CacheMode = CacheMode.OFF, + ) -> None: + """Refuse a database that cannot store this node's cache (ITL-1162). + + With caching on, checks the cache's own columns (``__node_content_hash``, + ``__data_context``); the operator's output columns are checked when they + are written. Nothing is computed and nothing changes. + + Args: + pipeline_database: The database for the cache. + cache_mode: The caching behaviour; ``OFF`` stores nothing. + + Raises: + UnstorableColumnError: If the database cannot store a column, naming + this node. + """ + if cache_mode == CacheMode.OFF: + return + try: + require_storable( + pipeline_database, + pa.schema( + [ + pa.field(constants.NODE_CONTENT_HASH_COL, pa.large_binary()), + data_context_field(), + ] + ), + ) + except UnstorableColumnError as exc: + exc.add_context(node_label=self.label) + raise + def attach_databases( self, pipeline_database: ArrowDatabaseProtocol, @@ -843,6 +881,9 @@ def attach_databases( pipeline_database: Database for pipeline records. cache_mode: Caching behaviour (OFF, LOG, or REPLAY). """ + # A database that cannot store the cache refuses it now, before anything + # runs or changes (ITL-1162). + self.require_storable_tables(pipeline_database, cache_mode) # Another database has its own tables of other generations to account for. self._stale_tables_checked = False self._skipped_rows_reported = False diff --git a/src/orcapod/core/result_cache.py b/src/orcapod/core/result_cache.py index fa3ee1de3..f89caaa83 100644 --- a/src/orcapod/core/result_cache.py +++ b/src/orcapod/core/result_cache.py @@ -41,6 +41,11 @@ logger = logging.getLogger(__name__) +def pod_timestamp_type() -> pa.DataType: + """The Arrow type of a result row's ``__pod_ts`` column: UTC microseconds.""" + return pa.timestamp("us", tz="UTC") + + class ResultCache: """Shared result caching backed by an ``ArrowDatabaseProtocol``. @@ -329,7 +334,7 @@ def store( data_table = data_table.append_column( constants.POD_TIMESTAMP, - pa.array([datetime.now(timezone.utc)], type=pa.timestamp("us", tz="UTC")), + pa.array([datetime.now(timezone.utc)], type=pod_timestamp_type()), ) data_table = with_data_context(data_table, data_context_key) diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index edf4abe6e..5e1358330 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -11,6 +11,7 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.sources.base import RootSource +from orcapod.errors import UnstorableColumnError from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import ( check_head_size, @@ -23,6 +24,8 @@ head_of, ) from orcapod.contexts.stored import ( + data_context_field, + require_storable, keep_rows_of_data_context, require_same_data_context, account_for_stale_tables, @@ -102,6 +105,14 @@ def __init__( data_context=data_context, config=config, ) + # A database that cannot store the cache's own columns refuses it now, + # before anything is read (ITL-1162). The source's columns are checked + # when they are written. + try: + require_storable(cache_database, pa.schema([data_context_field()])) + except UnstorableColumnError as exc: + exc.add_context(node_label=self.label) + raise self._source: SourceProtocol = source self._cache_database = cache_database self._cache_path_prefix = cache_path_prefix diff --git a/src/orcapod/databases/connector_arrow_database.py b/src/orcapod/databases/connector_arrow_database.py index af48cecd8..de5df5ca2 100644 --- a/src/orcapod/databases/connector_arrow_database.py +++ b/src/orcapod/databases/connector_arrow_database.py @@ -24,6 +24,12 @@ from typing import TYPE_CHECKING, Any, cast from orcapod.databases.utils import coerce_record_id +from orcapod.errors import UnstorableColumnError, UnstorableTableNameError +from orcapod.utils.arrow_utils import ( + normalize_schema_to_large_types, + normalize_table_to_large_types, + normalize_to_large_types, +) from orcapod.protocols.db_connector_protocol import ColumnInfo, DBConnectorProtocol from orcapod.utils.lazy_module import LazyModule @@ -185,6 +191,7 @@ def _deduplicate_within_table(self, table: pa.Table) -> pa.Table: def _get_committed_table(self, record_path: tuple[str, ...]) -> pa.Table | None: """Fetch all committed records for a path from the connector.""" table_name = self._path_to_table_name(self._path_prefix + record_path) + self._require_table_name(table_name) if table_name not in self._connector.get_table_names(): return None batches = list(self._connector.iter_batches(f'SELECT * FROM "{table_name}"')) @@ -244,11 +251,62 @@ def table_exists(self, record_path: tuple[str, ...]) -> bool: ``True`` if a table exists at the given path, ``False`` otherwise. """ table_name = self._path_to_table_name(self._path_prefix + record_path) + self._require_table_name(table_name) record_key = self._get_record_key(record_path) if record_key in self._pending_batches: return True return table_name in self._connector.get_table_names() + # ── Storable columns ────────────────────────────────────────────────────── + + def require_storable(self, schema: pa.Schema) -> None: + """Refuse a table schema with a column the connector cannot store (ITL-1162). + + The connector states which columns it stores + (``DBConnectorProtocol.unstorable_column_reason``): a column it would read + back as another type, or could not write at all, is refused. Called by + ``add_records`` before anything of the table is buffered, and by a job node + when the database is attached to it, before anything runs. + + Args: + schema: The Arrow schema of a table this database is to hold. The + ``__record_id`` column, when present, is checked as the key. + + Raises: + UnstorableColumnError: For the first column the connector cannot + store, naming the connector, the column, its Arrow type and why. + """ + for field in schema: + reason = self._connector.unstorable_column_reason( + field.name, + field.type, + primary_key=field.name == self.RECORD_ID_COLUMN, + ) + if reason is not None: + raise UnstorableColumnError( + database_kind=type(self._connector).__name__, + column=field.name, + arrow_type=field.type, + reason=reason, + ) + + def _require_table_name(self, table_name: str) -> None: + """Refuse a table name the connector would not keep (ITL-1162). + + Called at every entry that creates, writes, reads or checks a table, so a + name the backend would change (truncate) is never sent to it. + + Raises: + UnstorableTableNameError: Naming the connector, the name and why. + """ + reason = self._connector.unstorable_table_name_reason(table_name) + if reason is not None: + raise UnstorableTableNameError( + database_kind=type(self._connector).__name__, + table_name=table_name, + reason=reason, + ) + # ── Write methods ───────────────────────────────────────────────────────── def add_record( @@ -306,6 +364,16 @@ def add_records( f"got {rid_type}. Encode the column to bytes before calling add_records()." ) + # Checked and buffered in the large variants every connector reads back + # (view types included), so a read that joins buffered and committed rows + # sees one layout (ITL-1122, ITL-1162). + self.require_storable( + normalize_schema_to_large_types(records.schema, keep_null=True) + ) + self._require_table_name( + self._path_to_table_name(self._path_prefix + record_path) + ) + records = normalize_table_to_large_types(records, keep_null=True) self._connector.validate_records(records) records = self._deduplicate_within_table(records) @@ -402,6 +470,7 @@ def flush(self) -> None: for record_key in list(self._pending_batches.keys()): record_path = tuple(record_key.split("/")) table_name = self._path_to_table_name(record_path) + self._require_table_name(table_name) pending = self._pending_batches.pop(record_key) self._pending_record_ids.pop(record_key, None) skip_existing = self._pending_skip_existing.pop(record_key, False) @@ -414,11 +483,16 @@ def flush(self) -> None: # cryptic DB-level error or a silent partial write. existing_table_names = self._connector.get_table_names() if table_name in existing_table_names: + # A connector reads text and binary back in their large variants, + # which are the same types (ITL-1162). existing_cols = { - c.name: c.arrow_type + c.name: normalize_to_large_types(c.arrow_type, keep_null=True) for c in self._connector.get_column_info(table_name) } - pending_cols = {c.name: c.arrow_type for c in columns} + pending_cols = { + c.name: normalize_to_large_types(c.arrow_type, keep_null=True) + for c in columns + } if existing_cols != pending_cols: def _fmt(cols: dict) -> str: diff --git a/src/orcapod/databases/extension_aware_database.py b/src/orcapod/databases/extension_aware_database.py index 01e4fa8af..23cf95166 100644 --- a/src/orcapod/databases/extension_aware_database.py +++ b/src/orcapod/databases/extension_aware_database.py @@ -187,6 +187,21 @@ def flush_batch(self) -> Callable[[tuple[str, ...]], None]: """ return getattr(self._db, "flush_batch") # noqa: B009 # AttributeError when absent, by design + @property + def require_storable(self) -> Callable[[pa.Schema], None]: + """The wrapped database's ``require_storable`` (ITL-1162). + + A job node asks it about its stored tables when the database is attached, + so a wrapped connector database refuses a table it cannot store before + anything runs. Extension-typed columns are checked as given: a connector + that cannot store their storage type with its metadata cannot store them. + + Raises: + AttributeError: If the wrapped database has none, so this wrapper + checks columns only when the wrapped database does. + """ + return getattr(self._db, "require_storable") # noqa: B009 # AttributeError when absent, by design + # ── Structural delegation ───────────────────────────────────────────────── @property diff --git a/src/orcapod/databases/postgresql_connector.py b/src/orcapod/databases/postgresql_connector.py index 5bf1f6dd3..e8befed1b 100644 --- a/src/orcapod/databases/postgresql_connector.py +++ b/src/orcapod/databases/postgresql_connector.py @@ -21,6 +21,7 @@ from collections.abc import Iterator from typing import TYPE_CHECKING, Any +from orcapod.errors import UnstorableTableNameError from orcapod.types import ColumnInfo from orcapod.utils.lazy_module import LazyModule @@ -64,7 +65,7 @@ def _pg_type_to_arrow(pg_type_name: str, udt_name: str) -> pa.DataType: t = udt_name.lower() - if t == "bool": + if t in ("bool", "boolean"): return _pa.bool_() if t in ("int2", "smallint"): return _pa.int32() @@ -125,21 +126,32 @@ def _pg_type_to_arrow(pg_type_name: str, udt_name: str) -> pa.DataType: return _pa.large_string() -def _arrow_type_to_pg_sql(arrow_type: pa.DataType) -> str: - """Map an Arrow DataType to a PostgreSQL SQL type string for CREATE TABLE. +# PostgreSQL truncates an identifier (a quoted one too) to NAMEDATALEN - 1 bytes, 63 on a +# default build, so a longer column name reads back as another name (ITL-1162). +_PG_MAX_IDENTIFIER_BYTES = 63 + +# Every PostgreSQL table has these system columns; a user column of the same name is +# refused by CREATE TABLE, quoted or not (ITL-1162). +_PG_SYSTEM_COLUMNS = frozenset({"tableoid", "xmin", "cmin", "xmax", "cmax", "ctid"}) + + +def _pg_sql_type(arrow_type: pa.DataType) -> str | None: + """Return the PostgreSQL column type for an Arrow type, or ``None`` when there is none. Args: arrow_type: The Arrow DataType to convert. Returns: - A PostgreSQL SQL type string (e.g. ``"TEXT"``, ``"BIGINT"``). - - Raises: - ValueError: If ``arrow_type`` is ``pa.large_list(...)`` — array columns - are not supported in CREATE TABLE (they are read-only from existing tables). + A PostgreSQL SQL type string (e.g. ``"TEXT"``, ``"BIGINT"``), or ``None`` + for a type with no column type here (lists, structs, extension types and + the rest). """ import pyarrow as _pa + from orcapod.utils.arrow_utils import normalize_to_large_types + + # View layouts map as their large variants, as the storability rule reads them. + arrow_type = normalize_to_large_types(arrow_type, keep_null=True) if arrow_type == _pa.bool_(): return "BOOLEAN" if _pa.types.is_integer(arrow_type): @@ -164,13 +176,64 @@ def _arrow_type_to_pg_sql(arrow_type: pa.DataType) -> str: return "DATE" if _pa.types.is_timestamp(arrow_type): return "TIMESTAMPTZ" if arrow_type.tz is not None else "TIMESTAMP" + return None + + +def _arrow_type_to_pg_sql(arrow_type: pa.DataType) -> str: + """Map an Arrow DataType to a PostgreSQL SQL type string for CREATE TABLE. + + Args: + arrow_type: The Arrow DataType to convert. + + Returns: + A PostgreSQL SQL type string (e.g. ``"TEXT"``, ``"BIGINT"``). + + Raises: + ValueError: If PostgreSQL has no column type here for ``arrow_type``: + array columns (``pa.large_list(...)``) are read-only from existing + tables, and there is no fallback to TEXT for any other type + (ITL-1162). + """ + import pyarrow as _pa + + sql_type = _pg_sql_type(arrow_type) + if sql_type is not None: + return sql_type if _pa.types.is_large_list(arrow_type) or _pa.types.is_list(arrow_type): raise ValueError( f"Arrow type {arrow_type!r} (list/array) is not supported for " "CREATE TABLE. Array columns can only be read from existing tables." ) - logger.warning("Unsupported Arrow type %r; mapping to TEXT", arrow_type) - return "TEXT" + raise ValueError( + f"Arrow type {arrow_type!r} is not supported for CREATE TABLE: " + "PostgreSQL has no column type for it here." + ) + + +def _pg_unstorable_reason(arrow_type: pa.DataType) -> str | None: + """Say why PostgreSQL cannot store a column of ``arrow_type``, or ``None`` when it can. + + A type is stored when the column type it is written as reads back as the same + type in its large variant (``_pg_type_to_arrow``): ``bool``, ``int32``, + ``int64``, ``float``, ``double``, ``large_string``, ``large_binary``, + ``date32``, ``timestamp[us]`` and ``timestamp[us, tz=UTC]``. + + Args: + arrow_type: The column's Arrow type. + + Returns: + A short reason, or ``None`` when PostgreSQL stores the type. + """ + from orcapod.utils.arrow_utils import normalize_to_large_types + + arrow_type = normalize_to_large_types(arrow_type, keep_null=True) + sql_type = _pg_sql_type(arrow_type) + if sql_type is None: + return "PostgreSQL has no column type for it here" + read_back = _pg_type_to_arrow(sql_type.lower(), sql_type.lower()) + if read_back != arrow_type: + return f"PostgreSQL stores it as {sql_type}, which reads back as {read_back}" + return None def _coerce_pg_value(value: Any, arrow_type: "pa.DataType") -> Any: @@ -360,6 +423,7 @@ def current_location(self) -> tuple[str, str]: def get_pk_columns(self, table_name: str) -> list[str]: """Return primary-key column names in key-sequence order.""" + self._require_table_name(table_name) with self._lock: conn = self._require_open() self._validate_table_name(table_name) @@ -383,6 +447,7 @@ def get_pk_columns(self, table_name: str) -> list[str]: def get_column_info(self, table_name: str) -> list[ColumnInfo]: """Return column metadata with Arrow-mapped types.""" + self._require_table_name(table_name) with self._lock: conn = self._require_open() self._validate_table_name(table_name) @@ -474,6 +539,7 @@ def create_table_if_not_exists( pk_column: str, ) -> None: """Create a table with the given columns if it does not already exist.""" + self._require_table_name(table_name) col_names = [col.name for col in columns] if pk_column not in col_names: raise ValueError( @@ -508,6 +574,7 @@ def upsert_records( skip_existing: bool = False, ) -> None: """Write records to a table using upsert semantics.""" + self._require_table_name(table_name) with self._lock: conn = self._require_open() self._validate_table_name(table_name) @@ -545,6 +612,67 @@ def upsert_records( conn.rollback() raise + def unstorable_column_reason( + self, name: str, arrow_type: pa.DataType, *, primary_key: bool + ) -> str | None: + """Say why PostgreSQL cannot store a column, or ``None`` when it can (ITL-1162). + + PostgreSQL stores a column only when it reads it back as written: + ``bool``, ``int32``, ``int64``, ``float``, ``double``, ``large_string``, + ``large_binary``, ``date32``, ``timestamp[us]`` and + ``timestamp[us, tz=UTC]`` (``string`` and ``binary`` read back in their + large variants). Other integer widths, and timestamps of another unit or + time zone, read back as another type, and lists, structs and extension types have no column type here. + A name longer than 63 bytes (UTF-8) is refused: PostgreSQL truncates it, + so it would read back as another name. So is a system column name + (``xmin``, ``ctid``, ...). The key role does not matter. + + Args: + name: The column's name. + arrow_type: The column's Arrow type. + primary_key: Whether the column is the table's primary key. + + Returns: + A short reason, or ``None`` when PostgreSQL stores the column. + """ + if name in _PG_SYSTEM_COLUMNS: + return f"PostgreSQL reserves {name!r} as a system column name" + size = len(name.encode("utf-8")) + if size > _PG_MAX_IDENTIFIER_BYTES: + return ( + f"PostgreSQL truncates a column name to {_PG_MAX_IDENTIFIER_BYTES} " + f"bytes, and this name has {size}" + ) + return _pg_unstorable_reason(arrow_type) + + def unstorable_table_name_reason(self, table_name: str) -> str | None: + """Say why PostgreSQL cannot hold a table of this name, or ``None`` (ITL-1162). + + PostgreSQL truncates a name longer than 63 bytes (UTF-8), so two such + names with the same first 63 bytes would be one table. + + Args: + table_name: The table name. + + Returns: + A short reason, or ``None`` when PostgreSQL holds the name. + """ + size = len(table_name.encode("utf-8")) + if size > _PG_MAX_IDENTIFIER_BYTES: + return ( + f"PostgreSQL truncates a table name to {_PG_MAX_IDENTIFIER_BYTES} " + f"bytes, and this name has {size}" + ) + return None + + def _require_table_name(self, table_name: str) -> None: + """Refuse a table name PostgreSQL would truncate, before it is sent.""" + reason = self.unstorable_table_name_reason(table_name) + if reason is not None: + raise UnstorableTableNameError( + database_kind=type(self).__name__, table_name=table_name, reason=reason + ) + def validate_records(self, records: pa.Table) -> None: """Reject tables carrying any Arrow field or schema metadata. diff --git a/src/orcapod/databases/spiraldb_connector.py b/src/orcapod/databases/spiraldb_connector.py index 68c39783c..f88e5c455 100644 --- a/src/orcapod/databases/spiraldb_connector.py +++ b/src/orcapod/databases/spiraldb_connector.py @@ -377,6 +377,25 @@ def _about_table(table: str) -> Iterator[None]: # --------------------------------------------------------------------------- +def _holds_zone_aware_timestamp(arrow_type: pa.DataType) -> bool: + """Whether ``arrow_type`` is or nests a timestamp with a time zone (ITL-1162). + + Args: + arrow_type: The column's Arrow type. + + Returns: + ``True`` for a zone-aware timestamp at any depth of lists, structs and maps. + """ + if pa.types.is_timestamp(arrow_type): + return arrow_type.tz is not None + if isinstance(arrow_type, pa.ExtensionType): + return _holds_zone_aware_timestamp(arrow_type.storage_type) + return any( + _holds_zone_aware_timestamp(arrow_type.field(i).type) + for i in range(arrow_type.num_fields) + ) + + class SpiralDBConnector: """DBConnectorProtocol implementation backed by SpiralDB (pyspiral). @@ -947,6 +966,55 @@ def write_rows(rows: pa.Table) -> None: if len(novel) > 0: write_rows(novel) + def unstorable_column_reason( + self, name: str, arrow_type: pa.DataType, *, primary_key: bool + ) -> str | None: + """Say why SpiralDB cannot store a column, or ``None`` when it can (ITL-1162). + + SpiralDB stores Arrow tables natively, but not every column as written + (ITL-1164). It reserves ``.`` as its column-group separator in every name, + and the ``__`` prefix on value (column group) columns; a key column may + start with ``__`` (``__record_id``). It drops a timestamp's time zone, so a + time-zone-aware timestamp column (``timestamp[us, tz=UTC]``) would read + back as another type and is refused. + + Args: + name: The column's name. + arrow_type: The column's Arrow type. + primary_key: Whether the column is the table's primary key. + + Returns: + A short reason, or ``None`` when SpiralDB stores the column. + """ + if "." in name: + return "SpiralDB reserves '.' in column names as its column-group separator" + if name.startswith("__") and not primary_key: + return "SpiralDB reserves the '__' prefix on value columns" + if pa.types.is_timestamp(arrow_type) and arrow_type.tz is not None: + return ( + "SpiralDB drops a timestamp's time zone, so it reads back as " + f"timestamp[{arrow_type.unit}]" + ) + if _holds_zone_aware_timestamp(arrow_type): + return ( + "SpiralDB drops the time zone of a timestamp nested in it, so it " + "reads back as another type" + ) + return None + + def unstorable_table_name_reason(self, table_name: str) -> str | None: + """Say why SpiralDB cannot hold a table of this name: it holds every name. + + The database's table names are letters, digits and ``_``. + + Args: + table_name: The table name. + + Returns: + Always ``None``. + """ + return None + def validate_records(self, records: pa.Table) -> None: """No-op: ``SpiralDBConnector`` preserves Arrow field metadata via the native KV store.""" diff --git a/src/orcapod/databases/sqlite_connector.py b/src/orcapod/databases/sqlite_connector.py index bd0e4253f..72c89236b 100644 --- a/src/orcapod/databases/sqlite_connector.py +++ b/src/orcapod/databases/sqlite_connector.py @@ -65,17 +65,23 @@ def _sqlite_type_to_arrow(declared_type: str) -> pa.DataType: return pa.float64() -def _arrow_type_to_sqlite_sql(arrow_type: pa.DataType) -> str: - """Map an Arrow DataType to a SQLite SQL type string for CREATE TABLE. +def _sqlite_sql_type(arrow_type: pa.DataType) -> str | None: + """Return the SQLite column type for an Arrow type, or ``None`` when there is none. Args: arrow_type: The Arrow DataType to convert. Returns: - A SQLite SQL type string (e.g. "INTEGER", "TEXT"). + A SQLite SQL type string (e.g. "INTEGER", "TEXT"), or ``None`` for a type + SQLite has no column type for (lists, structs, dates, timestamps, extension + types and the rest). """ import pyarrow as _pa # noqa: PLC0415 — needed at call time + from orcapod.utils.arrow_utils import normalize_to_large_types # noqa: PLC0415 + + # View layouts map as their large variants, as the storability rule reads them. + arrow_type = normalize_to_large_types(arrow_type, keep_null=True) if arrow_type == _pa.bool_(): return "BOOLEAN" if _pa.types.is_integer(arrow_type): @@ -86,8 +92,52 @@ def _arrow_type_to_sqlite_sql(arrow_type: pa.DataType) -> str: return "TEXT" if _pa.types.is_binary(arrow_type) or _pa.types.is_large_binary(arrow_type): return "BLOB" - logger.warning("Unsupported Arrow type %r; mapping to TEXT", arrow_type) - return "TEXT" + return None + + +def _arrow_type_to_sqlite_sql(arrow_type: pa.DataType) -> str: + """Map an Arrow DataType to a SQLite SQL type string for CREATE TABLE. + + Args: + arrow_type: The Arrow DataType to convert. + + Returns: + A SQLite SQL type string (e.g. "INTEGER", "TEXT"). + + Raises: + ValueError: If SQLite has no column type for ``arrow_type``. There is no + fallback to TEXT: a value of such a type would fail to bind or read + back as text (ITL-1162). + """ + sql_type = _sqlite_sql_type(arrow_type) + if sql_type is None: + raise ValueError(f"SQLite has no column type for Arrow type {arrow_type}") + return sql_type + + +def _sqlite_unstorable_reason(arrow_type: pa.DataType) -> str | None: + """Say why SQLite cannot store a column of ``arrow_type``, or ``None`` when it can. + + A type is stored when the column type it is written as reads back as the same + type in its large variant: ``bool``, ``int64``, ``double``, ``large_string`` + and ``large_binary`` (and ``string`` / ``binary``, read back large). + + Args: + arrow_type: The column's Arrow type. + + Returns: + A short reason, or ``None`` when SQLite stores the type. + """ + from orcapod.utils.arrow_utils import normalize_to_large_types # noqa: PLC0415 + + arrow_type = normalize_to_large_types(arrow_type, keep_null=True) + sql_type = _sqlite_sql_type(arrow_type) + if sql_type is None: + return "SQLite has no column type for it" + read_back = _sqlite_type_to_arrow(sql_type) + if read_back != arrow_type: + return f"SQLite stores it as {sql_type}, which reads back as {read_back}" + return None def _coerce_column(values: list[Any], arrow_type: pa.DataType) -> list[Any]: @@ -391,6 +441,44 @@ def upsert_records( rows = (tuple(row[c] for c in cols) for row in records.to_pylist()) conn.executemany(sql, rows) + def unstorable_column_reason( + self, name: str, arrow_type: pa.DataType, *, primary_key: bool + ) -> str | None: + """Say why SQLite cannot store a column, or ``None`` when it can (ITL-1162). + + SQLite stores a column only when it reads it back as written: + ``bool``, ``int64``, ``double``, ``large_string`` and ``large_binary`` + (``string`` and ``binary`` read back in their large variants). Narrower + integers and floats read back widened, and lists, structs, dates, + timestamps and extension types have no SQLite column type. The name and + the key role do not matter to SQLite. + + Args: + name: The column's name. + arrow_type: The column's Arrow type. + primary_key: Whether the column is the table's primary key. + + Returns: + A short reason, or ``None`` when SQLite stores the column. + """ + return _sqlite_unstorable_reason(arrow_type) + + def unstorable_table_name_reason(self, table_name: str) -> str | None: + """Say why SQLite cannot hold a table of this name, or ``None`` (ITL-1162). + + SQLite has no identifier length limit, but reserves table names that + start with ``sqlite_`` (in any case) for itself. + + Args: + table_name: The table name. + + Returns: + A short reason, or ``None`` when SQLite holds the name. + """ + if table_name.lower().startswith("sqlite_"): + return "SQLite reserves table names that start with 'sqlite_'" + return None + def validate_records(self, records: pa.Table) -> None: """Reject tables carrying any Arrow field or schema metadata. diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 737b56228..be3fcd042 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -85,6 +85,154 @@ class UnsupportedValueError(ValueError): """ +class UnstorableColumnError(ValueError): + """A database cannot store a column of a table it is asked to hold (ITL-1162). + + Raised by a database whose backend states which columns it stores faithfully (a + ``ConnectorArrowDatabase``: column types on the SQLite and PostgreSQL connectors, + reserved column names on SpiralDB), before anything of the table is written. A + job raises it when the database is attached to a node whose stored tables hold + such a column, before anything runs, for the columns known from declared + schemas alone: a node's own columns and a function's outputs. The columns a + node carries from its input (tags, system tags, source info) and an operator's + output columns are refused when their table is first written, before anything + of it is stored. + + Attributes: + database_kind: The kind of database that refused, e.g. ``"SQLiteConnector"``. + column: Name of the column. + arrow_type: The column's Arrow type. + reason: Why the database cannot store it. + node_label: Label of the node whose table holds the column, or ``None`` when + the refusal is not tied to one node. + """ + + def __init__( + self, + *, + database_kind: str, + column: str, + arrow_type: object, + reason: str, + node_label: str | None = None, + ) -> None: + """Create the error. + + Args: + database_kind: The kind of database that refused. + column: Name of the column. + arrow_type: The column's Arrow type. + reason: Why the database cannot store it. + node_label: Label of the node whose table holds the column, if known. + """ + self.database_kind = database_kind + self.column = column + self.arrow_type = arrow_type + self.reason = reason + self.node_label = node_label + super().__init__(database_kind, column, arrow_type, reason) + + def __reduce__(self) -> tuple[Any, ...]: + """Rebuild from the keyword fields, so pickling and copying keep them.""" + return ( + _rebuild_unstorable_column_error, + ( + self.database_kind, + self.column, + self.arrow_type, + self.reason, + self.node_label, + ), + ) + + def add_context(self, node_label: str | None = None) -> None: + """Fill in the node label where the raising site did not know it. + + Args: + node_label: Label of the node whose table holds the column. + """ + if self.node_label is None: + self.node_label = node_label + + def __str__(self) -> str: + node = f"node {self.node_label!r}: " if self.node_label is not None else "" + return ( + f"{node}a {self.database_kind} database cannot store column " + f"{self.column!r} of Arrow type {self.arrow_type}: {self.reason}. " + "Use an InMemoryArrowDatabase or a DeltaTableDatabase instead." + ) + + +class UnstorableTableNameError(ValueError): + """A database cannot hold a table under the name it is asked to use (ITL-1162). + + PostgreSQL truncates a table name longer than 63 bytes, so two Orcapod tables + whose names share their first 63 bytes would silently become one. The + connector states which names it holds (``unstorable_table_name_reason``), and + a ``ConnectorArrowDatabase`` refuses any other name at every entry that + creates, writes, reads or checks a table, so a truncated name is never sent. + + Attributes: + database_kind: The kind of database that refused, e.g. + ``"PostgreSQLConnector"``. + table_name: The table name the database was asked to use. + reason: Why the database cannot hold it. + """ + + def __init__(self, *, database_kind: str, table_name: str, reason: str) -> None: + """Create the error. + + Args: + database_kind: The kind of database that refused. + table_name: The table name the database was asked to use. + reason: Why the database cannot hold it. + """ + self.database_kind = database_kind + self.table_name = table_name + self.reason = reason + super().__init__(database_kind, table_name, reason) + + def __reduce__(self) -> tuple[Any, ...]: + """Rebuild from the keyword fields, so pickling and copying keep them.""" + return ( + _rebuild_unstorable_table_name_error, + (self.database_kind, self.table_name, self.reason), + ) + + def __str__(self) -> str: + return ( + f"a {self.database_kind} database cannot hold a table named " + f"{self.table_name!r}: {self.reason}. Use an InMemoryArrowDatabase or a " + "DeltaTableDatabase instead." + ) + + +def _rebuild_unstorable_table_name_error( + database_kind: str, table_name: str, reason: str +) -> UnstorableTableNameError: + """Rebuild an ``UnstorableTableNameError`` from its fields (``__reduce__``).""" + return UnstorableTableNameError( + database_kind=database_kind, table_name=table_name, reason=reason + ) + + +def _rebuild_unstorable_column_error( + database_kind: str, + column: str, + arrow_type: object, + reason: str, + node_label: str | None, +) -> UnstorableColumnError: + """Rebuild an ``UnstorableColumnError`` from its fields (``__reduce__``).""" + return UnstorableColumnError( + database_kind=database_kind, + column=column, + arrow_type=arrow_type, + reason=reason, + node_label=node_label, + ) + + class MissingOptionalDependencyError(ImportError): """A stored value needs an optional dependency that is not installed. diff --git a/src/orcapod/pipeline/base.py b/src/orcapod/pipeline/base.py index 3236c1c2c..c9ee88e93 100644 --- a/src/orcapod/pipeline/base.py +++ b/src/orcapod/pipeline/base.py @@ -8,6 +8,7 @@ from dataclasses import dataclass from typing import TYPE_CHECKING, Any, Generic, TypeVar +from orcapod.core.nodes.function_node import FunctionNodeBase from orcapod.core.tracker import AutoRegisteringContextBasedTracker from orcapod.pipeline.dag import OrcaDAG from orcapod.pipeline.pod_invocation import ( @@ -154,7 +155,17 @@ def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> No Args: database: An ``ArrowDatabaseProtocol`` instance to attach, or ``None`` to detach. + + Raises: + UnstorableColumnError: If ``database`` cannot store a function node's + result table (ITL-1162); no node is changed. """ + if database is not None: + # Every function node is checked before any changes, so a database that + # cannot store a result table leaves every node as it was (ITL-1162). + for node in self._nodes.values(): + if isinstance(node, FunctionNodeBase): + node.require_storable_results(database) for node in self._nodes.values(): if node.node_type not in ("source", "side_effect"): node.set_ephemeral_database(database) diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 40408853f..9e3e26e39 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -419,6 +419,8 @@ def bind( SourceSpecMismatchError: If any source's columns differ from its slot's declared names, roles or types. ValueError: If a source key has no matching ``SourceJobNode`` slot. + UnstorableColumnError: If *database* cannot store a table of a node + (ITL-1162); the job is left as it was. """ from orcapod.core.nodes.source_node import SourceNodeBase @@ -441,6 +443,11 @@ def bind( if isinstance(node, SourceNodeBase) and node.name in sources: node.validate(sources[node.name]) + # A database that cannot store a node's tables is refused before anything + # changes too (ITL-1162). + if database_changed and database is not None: + self._require_storable_tables(database) + # Every source is validated before anything changes: a refused source leaves # the job as it was, its database included (ITL-1112). if database is not None: @@ -476,6 +483,30 @@ def bind( # _distribute_databases() # ------------------------------------------------------------------ + def _require_storable_tables(self, database: "ArrowDatabaseProtocol") -> None: + """Refuse ``database`` if it cannot store a table of one of the job's nodes. + + The same views as ``_distribute_databases`` gives the nodes. Nothing + changes (ITL-1162). + + Args: + database: The job database to check. + + Raises: + UnstorableColumnError: For the first node whose table the database + cannot store, naming the node. + """ + pipeline_db = database.at(*self._name) + result_db = pipeline_db.at("_result") + for node in (self._persistent_node_map or {}).values(): + if isinstance(node, FunctionJobNode): + node.require_storable_tables(pipeline_db, result_db) + elif isinstance(node, OperatorJobNode): + op_cache_mode = getattr(node, "_cache_mode", None) or CacheMode.OFF + node.require_storable_tables(pipeline_db, op_cache_mode) + elif isinstance(node, SideEffectJobNode): + node.require_storable_tables(pipeline_db) + def _distribute_databases(self) -> None: """Wire live DB references to all FunctionJobNode and OperatorJobNode objects. @@ -491,6 +522,9 @@ def _distribute_databases(self) -> None: "Call bind(database=...) or from_pipeline(..., database=...) first." ) + # Every node is checked before any is attached, so a refused database + # leaves every node as it was (ITL-1162). + self._require_storable_tables(self._database) pipeline_db = self._database.at(*self._name) result_db = pipeline_db.at("_result") diff --git a/src/orcapod/protocols/database_protocols.py b/src/orcapod/protocols/database_protocols.py index fb585495c..a8c15765b 100644 --- a/src/orcapod/protocols/database_protocols.py +++ b/src/orcapod/protocols/database_protocols.py @@ -227,6 +227,21 @@ def flush_batch(self, record_path: tuple[str, ...]) -> None: ... +class ColumnCheckingDatabaseProtocol(Protocol): + """A database that states which columns it can store (ITL-1162). + + Such a database (``ConnectorArrowDatabase``) refuses a table with a column its + backend cannot store faithfully, before anything of it is written. A job node + asks it about its stored tables when the database is attached, so a pipeline + the database cannot hold is refused before anything runs. Check with + ``orcapod.contexts.stored.require_storable``. + """ + + def require_storable(self, schema: pa.Schema) -> None: + """Raise ``UnstorableColumnError`` for a column of ``schema`` it cannot store.""" + ... + + @runtime_checkable class DatabaseRegistryProtocol(Protocol): """Protocol for a database config registry used during pipeline save/load. diff --git a/src/orcapod/protocols/db_connector_protocol.py b/src/orcapod/protocols/db_connector_protocol.py index 7cd0ee8d3..6ed2f94bd 100644 --- a/src/orcapod/protocols/db_connector_protocol.py +++ b/src/orcapod/protocols/db_connector_protocol.py @@ -118,6 +118,44 @@ def upsert_records( """ ... + def unstorable_column_reason( + self, name: str, arrow_type: pa.DataType, *, primary_key: bool + ) -> str | None: + """Say why this connector cannot store a column, or ``None`` when it can. + + The one place a connector states which columns it stores (ITL-1162). A + connector stores a column only when it reads it back as written: the same + name, and the same Arrow type in its large variant (``large_string``, + ``large_binary``). A type the backend would widen, turn into text or fail to + bind is refused, and so is a name the backend reserves. + ``ConnectorArrowDatabase.require_storable`` asks this for every column of + a table before anything of it is written. + + Args: + name: The column's name. + arrow_type: The column's Arrow type. + primary_key: Whether the column is the table's primary key. + + Returns: + A short reason, or ``None`` when the connector stores the column. + """ + ... + + def unstorable_table_name_reason(self, table_name: str) -> str | None: + """Say why this connector cannot hold a table of this name, or ``None``. + + The one place a connector states which table names it holds (ITL-1162): a + name the backend would change (PostgreSQL truncates past 63 bytes) is + refused, so two tables can never collide under one stored name. + + Args: + table_name: The table name. + + Returns: + A short reason, or ``None`` when the connector holds the name. + """ + ... + def validate_records(self, records: pa.Table) -> None: """Validate that ``records`` are safe to write through this connector. diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index f87e838f3..a81613653 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -37,12 +37,15 @@ OrcapodDatabaseError, OrcapodDatabaseReadError, OrcapodDatabaseWriteError, + UnstorableColumnError, ) from orcapod.contexts.stored import ( StaleTables, account_for_stale_tables, + data_context_field, find_stale_tables, require_same_data_context, + require_storable, with_data_context, ) from orcapod.invocation import ( @@ -471,6 +474,22 @@ def _execute_side_effect_row( return (tag, data) +def _invocation_log_schema() -> pa.Schema: + """The columns of an invocation log row, before ``__data_context`` is appended.""" + return pa.schema( + [ + pa.field("record_id_hash", pa.large_binary()), + pa.field("pipeline_run_id", pa.large_string()), + pa.field("executed_at", pa.timestamp("us", tz="UTC")), + ] + ) + + +def _stored_invocation_log_schema() -> pa.Schema: + """The columns of an invocation log row as stored, ``__data_context`` included.""" + return _invocation_log_schema().append(data_context_field()) + + def _write_invocation_row( *, pipeline_database: ArrowDatabaseProtocol, @@ -498,15 +517,15 @@ def _write_invocation_row( ``record_id``; stored on the row as ``__data_context``. """ executed_at = datetime.datetime.now(datetime.timezone.utc) - record = pa.table( - { - "record_id_hash": pa.array([record_id_hash_bytes], type=pa.large_binary()), - "pipeline_run_id": pa.array([run_id], type=pa.large_string()), - "executed_at": pa.array( - [executed_at], - type=pa.timestamp("us", tz="UTC"), - ), - } + record = pa.Table.from_pylist( + [ + { + "record_id_hash": record_id_hash_bytes, + "pipeline_run_id": run_id, + "executed_at": executed_at, + } + ], + schema=_invocation_log_schema(), ) try: pipeline_database.add_record( @@ -1112,6 +1131,24 @@ def _refuse_log_of_another_data_context(self) -> None: # DB attachment # ------------------------------------------------------------------ + def require_storable_tables(self, pipeline_database: ArrowDatabaseProtocol) -> None: + """Refuse a database that cannot store this node's invocation log (ITL-1162). + + Nothing is delivered and nothing changes. + + Args: + pipeline_database: The database for the invocation log. + + Raises: + UnstorableColumnError: If the database cannot store a column, naming + this node. + """ + try: + require_storable(pipeline_database, _stored_invocation_log_schema()) + except UnstorableColumnError as exc: + exc.add_context(node_label=self.label) + raise + def attach_databases( self, pipeline_database: ArrowDatabaseProtocol | None = None, @@ -1126,6 +1163,10 @@ def attach_databases( pipeline_database: Pre-scoped pipeline DB (at pipeline root), or ``None`` to detach. """ + if pipeline_database is not None: + # A database that cannot store the invocation log refuses it now, + # before anything is delivered or changes (ITL-1162). + self.require_storable_tables(pipeline_database) self._pipeline_database = pipeline_database self._unlisted_store_reported = False self._current_log_checked = False diff --git a/tests/test_core/sources/test_db_table_source.py b/tests/test_core/sources/test_db_table_source.py index 1b11e7271..22b4d1d3f 100644 --- a/tests/test_core/sources/test_db_table_source.py +++ b/tests/test_core/sources/test_db_table_source.py @@ -77,6 +77,14 @@ def create_table_if_not_exists(self, *args: Any, **kwargs: Any) -> None: def upsert_records(self, *args: Any, **kwargs: Any) -> None: pass # not used by DBTableSource + def unstorable_table_name_reason(self, table_name: str) -> str | None: + return None # every name is held + + def unstorable_column_reason( + self, name: str, arrow_type: Any, *, primary_key: bool + ) -> str | None: + return None # not used by DBTableSource + def validate_records(self, records: Any) -> None: pass # not used by DBTableSource diff --git a/tests/test_databases/fake_pyspiral.py b/tests/test_databases/fake_pyspiral.py index 887baff8c..88dd25328 100644 --- a/tests/test_databases/fake_pyspiral.py +++ b/tests/test_databases/fake_pyspiral.py @@ -557,14 +557,25 @@ def write( **kwargs: Accepted for signature compatibility; unused. Raises: - ValueError: If a key column is missing, a struct value is null without - ``push_down_nulls``, or the column set differs from the stored table. + ValueError: If a key column is missing, a column name is one SpiralDB + reserves (a ``.`` anywhere, or a ``__`` prefix on a value column), a + struct value is null without ``push_down_nulls``, or the column set + differs from the stored table. """ b = self._store.behaviour keys = self.key_schema missing = [k for k in keys.names if k not in table.column_names] if missing: raise ValueError(f"write is missing key columns {missing}") + for name in table.column_names: + # SpiralDB's own refusals, with its messages as observed live (ITL-1164). + if "." in name: + raise ValueError(f"Column '{name}' contains a reserved dot-separator") + if name.startswith("__") and name not in keys.names: + raise ValueError( + f"Column name '{name}' has reserved prefix: column group columns " + "must not start with '__'" + ) if b.reject_struct_nulls and not push_down_nulls: for name in table.column_names: if _has_struct_nulls(table.column(name)): diff --git a/tests/test_databases/test_connector_arrow_database.py b/tests/test_databases/test_connector_arrow_database.py index f2d83f3f6..5fdb3934a 100644 --- a/tests/test_databases/test_connector_arrow_database.py +++ b/tests/test_databases/test_connector_arrow_database.py @@ -155,6 +155,15 @@ def upsert_records( kept = existing.filter(mask) self._tables[table_name] = pa.concat_tables([kept, records]) + def unstorable_table_name_reason(self, table_name: str) -> str | None: + return None # every name is held + + def unstorable_column_reason( + self, name: str, arrow_type: pa.DataType, *, primary_key: bool + ) -> str | None: + """Store every column: this in-memory mock keeps Arrow tables as they are.""" + return None + def validate_records(self, records: pa.Table) -> None: """Reject any Arrow field or schema metadata — mirrors real SQL connectors.""" problem_fields = [f.name for f in records.schema if _field_has_metadata(f)] diff --git a/tests/test_databases/test_postgresql_connector.py b/tests/test_databases/test_postgresql_connector.py index 41045c3c0..545d7b096 100644 --- a/tests/test_databases/test_postgresql_connector.py +++ b/tests/test_databases/test_postgresql_connector.py @@ -139,10 +139,15 @@ def test_large_list_raises(self): with pytest.raises(ValueError, match="not supported"): _arrow_type_to_pg_sql(pa.large_list(pa.int32())) - def test_unknown_falls_back_to_text(self): - # pa.null() is an unusual type not in the mapping - result = _arrow_type_to_pg_sql(pa.null()) - assert result == "TEXT" + def test_unknown_raises_instead_of_text(self): + """No silent TEXT fallback (ITL-1162).""" + for arrow_type in (pa.null(), pa.struct([("a", pa.int64())])): + with pytest.raises(ValueError, match="not supported"): + _arrow_type_to_pg_sql(arrow_type) + + def test_boolean_information_schema_name(self): + """``information_schema`` names a bool column ``boolean``.""" + assert _pg_type_to_arrow("boolean", "boolean") == pa.bool_() class TestCoercePgValue: diff --git a/tests/test_databases/test_postgresql_connector_integration.py b/tests/test_databases/test_postgresql_connector_integration.py index e37c379ea..ae5436de4 100644 --- a/tests/test_databases/test_postgresql_connector_integration.py +++ b/tests/test_databases/test_postgresql_connector_integration.py @@ -21,6 +21,7 @@ from __future__ import annotations +import datetime import os import uuid from collections.abc import Iterator @@ -475,3 +476,148 @@ def test_schema_mismatch_raises(self, db: ConnectorArrowDatabase) -> None: db.add_record(("fn", "mismatch"), record_id="r2", record=r2) with pytest.raises(ValueError, match="Schema mismatch"): db.flush() + + +# --------------------------------------------------------------------------- +# What the connector states it stores reads back as written (ITL-1162) +# --------------------------------------------------------------------------- + + +@pytest.mark.postgres +@pytest.mark.parametrize( + "arrow_type, values", + [ + pytest.param(pa.bool_(), [True, None], id="bool"), + pytest.param(pa.int32(), [1, None], id="int32"), + pytest.param(pa.int64(), [1, None], id="int64"), + pytest.param(pa.float32(), [1.5, None], id="float32"), + pytest.param(pa.float64(), [1.5, None], id="float64"), + pytest.param(pa.large_string(), ["a", None], id="large_string"), + pytest.param(pa.large_binary(), [b"a", None], id="large_binary"), + pytest.param(pa.date32(), [datetime.date(2026, 1, 1), None], id="date32"), + pytest.param( + pa.timestamp("us"), + [datetime.datetime(2026, 1, 1, 12), None], + id="timestamp", + ), + pytest.param( + pa.timestamp("us", tz="UTC"), + [datetime.datetime(2026, 1, 1, 12, tzinfo=datetime.timezone.utc), None], + id="timestamp_utc", + ), + ], +) +def test_storable_types_read_back_as_written( + connector: PostgreSQLConnector, arrow_type: pa.DataType, values: list[object] +) -> None: + """Each type ``unstorable_column_reason`` accepts round-trips, twice (ITL-1162).""" + assert ( + connector.unstorable_column_reason("v", arrow_type, primary_key=False) is None + ) + db = ConnectorArrowDatabase(connector) + table = pa.table({"v": pa.array(values, type=arrow_type)}) + db.add_record(("t",), b"1", table.slice(0, 1), flush=True) + db.add_record(("t",), b"2", table.slice(1, 1), flush=True) + + stored = db.get_all_records(("t",)) + assert stored is not None + assert stored.schema.field("v").type == arrow_type + assert sorted(stored["v"].to_pylist(), key=lambda v: v is None) == values + + +@pytest.mark.postgres +def test_function_node_job_refused(connector: PostgreSQLConnector) -> None: + """A function node's result table cannot be stored; the job refuses it (ITL-1162).""" + from orcapod import PipelineJob, function_pod + from orcapod.errors import UnstorableColumnError + from orcapod.sources import DictSource + + @function_pod(output_keys="double") + def double(x: int) -> int: + return 2 * x + + job = PipelineJob(name="p", database=ConnectorArrowDatabase(connector)) + source = DictSource( + data=[{"k": "a", "x": 1}], + tag_columns=["k"], + tag_only_columns=True, + source_id="src", + ) + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = double.pod(source, label="dbl") + assert exc_info.value.column == "__pf_exec_executor_info" + assert connector.get_table_names() == [] + + +@pytest.mark.postgres +def test_long_column_name_refused_and_nothing_written( + connector: PostgreSQLConnector, +) -> None: + """A name PostgreSQL would truncate is refused before the table exists (ITL-1162).""" + from orcapod.errors import UnstorableColumnError + + db = ConnectorArrowDatabase(connector) + name = "_tag_source_id::" + "0" * 64 + table = pa.table({name: pa.array(["s"], type=pa.large_string())}) + with pytest.raises(UnstorableColumnError): + db.add_record(("t",), b"1", table) + db.flush() + assert connector.get_table_names() == [] + + +@pytest.mark.postgres +def test_operator_cache_refused_at_write_and_nothing_written( + connector: PostgreSQLConnector, +) -> None: + """An operator cache's system-tag column names exceed 63 bytes (ITL-1162).""" + from orcapod import PipelineJob + from orcapod.core.operators import SelectDataColumns + from orcapod.errors import UnstorableColumnError, UnstorableTableNameError + from orcapod.sources import DictSource + from orcapod.types import CacheMode + + db = ConnectorArrowDatabase(connector) + job = PipelineJob(name="p", database=db) + source = DictSource( + data=[{"k": "a", "x": 1, "y": 2}], + tag_columns=["k"], + tag_only_columns=True, + source_id="src", + ) + with job: + _ = SelectDataColumns(["x"])(source, label="sel") + job.nodes["sel"].attach_databases( + pipeline_database=db.at("p"), cache_mode=CacheMode.LOG + ) + with pytest.raises(Exception) as exc_info: + _ = job.run() + # The cache's table name (over 63 bytes) or its system-tag column names are + # refused, whichever entry comes first; nothing reaches the server. + chain: list[BaseException] = [] + err: BaseException | None = exc_info.value + while err is not None: + chain.append(err) + err = err.__cause__ or err.__context__ + assert any( + isinstance(e, (UnstorableColumnError, UnstorableTableNameError)) for e in chain + ), chain + assert connector.get_table_names() == [] + + +@pytest.mark.postgres +def test_long_table_name_refused_and_nothing_written( + connector: PostgreSQLConnector, +) -> None: + """A table name PostgreSQL would truncate is never sent to it (ITL-1162).""" + from orcapod.errors import UnstorableTableNameError + + db = ConnectorArrowDatabase(connector) + path = ("fn", "x" * 70) + table = pa.table({"v": pa.array([1], type=pa.int64())}) + with pytest.raises(UnstorableTableNameError): + db.add_record(path, b"1", table) + with pytest.raises(UnstorableTableNameError): + db.get_all_records(path) + db.flush() + assert connector.get_table_names() == [] diff --git a/tests/test_databases/test_spiraldb_connector_integration.py b/tests/test_databases/test_spiraldb_connector_integration.py index adb8833da..c04b1b82d 100644 --- a/tests/test_databases/test_spiraldb_connector_integration.py +++ b/tests/test_databases/test_spiraldb_connector_integration.py @@ -386,3 +386,69 @@ def test_nested_struct_field_metadata_round_trip(self, connector): assert inner.metadata.get(b"unit") == b"volts" finally: connector.delete_table(table_name) + + +class TestReservedNamesRefusedLive: + """ITL-1162 / ITL-1164 against the live service (opt-in, like every test here). + + The refusal itself is decided by ``SpiralDBConnector.unstorable_column_reason`` + before any request; the second test checks the rule against SpiralDB's own + refusals, so the fake's model of them stays honest. + """ + + def test_function_node_job_refused_before_anything_runs(self, connector): + from orcapod import PipelineJob, function_pod + from orcapod.errors import UnstorableColumnError + from orcapod.sources import DictSource + + calls: list[int] = [] + + @function_pod(output_keys="double") + def double(x: int) -> int: + calls.append(x) + return 2 * x + + job = PipelineJob( + name=_unique_table("itl1162"), database=ConnectorArrowDatabase(connector) + ) + source = DictSource( + data=[{"k": "a", "x": 1}], + tag_columns=["k"], + tag_only_columns=True, + source_id="src", + ) + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = double.pod(source, label="dbl") + assert exc_info.value.column == "__input_data_hash" + assert calls == [] + + @pytest.mark.parametrize("name", ["__value", "a.b"]) + def test_spiraldb_itself_refuses_the_names_the_connector_refuses( + self, connector, name + ): + assert ( + connector.unstorable_column_reason(name, pa.int64(), primary_key=False) + is not None + ) + table = _unique_table("itl1162_names") + try: + connector.create_table_if_not_exists( + table, + [ + ColumnInfo("k", pa.large_string(), nullable=False), + ColumnInfo(name, pa.int64(), nullable=True), + ], + pk_column="k", + ) + rows = pa.table( + { + "k": pa.array(["a"], type=pa.large_string()), + name: pa.array([1], type=pa.int64()), + } + ) + with pytest.raises(Exception, match="reserved"): + connector.upsert_records(table, rows, id_column="k") + finally: + if table in connector.get_table_names(): + connector.delete_table(table) diff --git a/tests/test_databases/test_sqlite_connector.py b/tests/test_databases/test_sqlite_connector.py index 729706111..b3fc9c908 100644 --- a/tests/test_databases/test_sqlite_connector.py +++ b/tests/test_databases/test_sqlite_connector.py @@ -93,13 +93,15 @@ def test_binary(self): def test_bool(self): assert _arrow_type_to_sqlite_sql(pa.bool_()) == "BOOLEAN" - def test_unsupported_type_logs_warning_and_returns_text(self, caplog): - import logging - - with caplog.at_level(logging.WARNING): - result = _arrow_type_to_sqlite_sql(pa.list_(pa.int64())) - assert result == "TEXT" - assert "Unsupported Arrow type" in caplog.text + def test_unsupported_type_raises_instead_of_text(self): + """No silent TEXT fallback: such a value would fail to bind (ITL-1162).""" + for arrow_type in ( + pa.list_(pa.int64()), + pa.timestamp("us", tz="UTC"), + pa.date32(), + ): + with pytest.raises(ValueError, match="no column type"): + _arrow_type_to_sqlite_sql(arrow_type) class TestSQLiteConnectorScaffold: diff --git a/tests/test_databases/test_unstorable_columns.py b/tests/test_databases/test_unstorable_columns.py new file mode 100644 index 000000000..833d2e113 --- /dev/null +++ b/tests/test_databases/test_unstorable_columns.py @@ -0,0 +1,1023 @@ +"""A connector database refuses a table it cannot store, before anything is written (ITL-1162). + +Each connector states which columns it stores: those it reads back as written +(``unstorable_column_reason``). ``ConnectorArrowDatabase`` refuses any other column +before anything of the table is buffered, and a job refuses a function or +side-effect node whose stored table holds one when the database is attached, before +anything runs. A function node's result table always holds list-of-struct executor +columns and a UTC timestamp, so no function node runs on SQLite or PostgreSQL. +""" + +from __future__ import annotations + +import datetime +import pathlib +from typing import Any +from unittest.mock import MagicMock, patch + +import pyarrow as pa +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.operators import SelectDataColumns +from orcapod.databases import ( + ConnectorArrowDatabase, + InMemoryArrowDatabase, + PostgreSQLConnector, + SQLiteConnector, +) +from orcapod.errors import OrcapodDatabaseWriteError, UnstorableColumnError +from orcapod.side_effects import SideEffectPod, SideEffectPodConfig +from orcapod.sources import ArrowTableSource, DictSource +from orcapod.types import CacheMode + +CALLS: list[int] = [] +DELIVERED: list[int] = [] + +EXECUTOR_INFO = "__pf_exec_executor_info" +EXECUTOR_INFO_TYPE = pa.large_list( + pa.struct([("key", pa.large_string()), ("value", pa.large_string())]) +) + + +@pytest.fixture(autouse=True) +def _reset() -> None: + """Start every test with no recorded calls or deliveries.""" + CALLS.clear() + DELIVERED.clear() + + +@function_pod(output_keys="double") +def double(x: int) -> int: + """Double ``x``, recording the call.""" + CALLS.append(x) + return 2 * x + + +def _deliver(x: int, ctx: Any) -> None: + """A side effect: record the delivered value.""" + DELIVERED.append(x) + + +deliver = SideEffectPod(_deliver, SideEffectPodConfig(track_completion=True)) + + +def _source(rows: list[dict[str, Any]] | None = None) -> DictSource: + """A small in-memory source with tag ``k``.""" + return DictSource( + data=rows if rows is not None else [{"k": "a", "x": 1}, {"k": "b", "x": 2}], + tag_columns=["k"], + tag_only_columns=True, + source_id="src", + ) + + +def _sqlite_database(tmp_path: pathlib.Path) -> ConnectorArrowDatabase: + return ConnectorArrowDatabase(SQLiteConnector(str(tmp_path / "store.db"))) + + +# --------------------------------------------------------------------------- +# A job refuses a node up front +# --------------------------------------------------------------------------- + + +class TestJobRefusesUpFront: + def test_function_node_on_sqlite_refused_before_anything_runs( + self, tmp_path: pathlib.Path + ): + """The reported case: one function node on a SQLite job database.""" + database = _sqlite_database(tmp_path) + job = PipelineJob(name="p", database=database) + + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = double.pod(_source(), label="dbl") + + err = exc_info.value + assert err.node_label == "dbl" + assert err.database_kind == "SQLiteConnector" + assert err.column == EXECUTOR_INFO + assert err.arrow_type == EXECUTOR_INFO_TYPE + message = str(err) + assert "node 'dbl'" in message + assert "SQLiteConnector" in message + assert EXECUTOR_INFO in message + assert "InMemoryArrowDatabase" in message + assert "DeltaTableDatabase" in message + # Nothing ran and nothing was written. + assert CALLS == [] + assert database.list_table_paths() == [] + + def test_side_effect_node_on_sqlite_refused_before_delivery( + self, tmp_path: pathlib.Path + ): + """The invocation log's UTC timestamp is refused before anything is delivered.""" + database = _sqlite_database(tmp_path) + job = PipelineJob(name="p", database=database) + + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = deliver.process(_source(), label="out") + + err = exc_info.value + assert err.node_label == "out" + assert err.column == "executed_at" + assert err.arrow_type == pa.timestamp("us", tz="UTC") + assert DELIVERED == [] + assert database.list_table_paths() == [] + + def test_function_node_on_postgresql_refused(self): + """PostgreSQL has no column type for the list-of-struct executor columns.""" + with patch("psycopg.connect") as mock_connect: + mock_connect.return_value = MagicMock() + connector = PostgreSQLConnector("postgresql://localhost/test") + try: + job = PipelineJob(name="p", database=ConnectorArrowDatabase(connector)) + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = double.pod(_source(), label="dbl") + finally: + connector._conn = None # nothing to close + err = exc_info.value + assert err.node_label == "dbl" + assert err.database_kind == "PostgreSQLConnector" + assert err.column == EXECUTOR_INFO + assert CALLS == [] + + def test_in_memory_database_still_runs_function_node(self): + """A database that states no rule stores every column.""" + job = PipelineJob(name="p", database=InMemoryArrowDatabase()) + with job: + _ = double.pod(_source(), label="dbl") + _ = job.run() + assert sorted(CALLS) == [1, 2] + + +# --------------------------------------------------------------------------- +# The write itself refuses, before anything is buffered +# --------------------------------------------------------------------------- + + +UNSTORABLE_ON_SQLITE = [ + pytest.param(pa.int8(), [1], id="int8"), + pytest.param(pa.int32(), [1], id="int32"), + pytest.param(pa.uint64(), [1], id="uint64"), + pytest.param(pa.float32(), [1.5], id="float32"), + pytest.param(pa.date32(), [datetime.date(2026, 1, 1)], id="date32"), + pytest.param( + pa.timestamp("us", tz="UTC"), + [datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc)], + id="timestamp_utc", + ), + pytest.param( + pa.timestamp("us"), [datetime.datetime(2026, 1, 1)], id="timestamp_naive" + ), + pytest.param(pa.large_list(pa.int64()), [[1]], id="large_list"), + pytest.param(EXECUTOR_INFO_TYPE, [[{"key": "a", "value": "b"}]], id="list_struct"), + pytest.param(pa.struct([("a", pa.int64())]), [{"a": 1}], id="struct"), + pytest.param(pa.null(), [None], id="null"), +] + +STORABLE_ON_SQLITE = [ + pytest.param(pa.bool_(), [True], id="bool"), + pytest.param(pa.int64(), [1], id="int64"), + pytest.param(pa.float64(), [1.5], id="float64"), + pytest.param(pa.string(), ["a"], id="string"), + pytest.param(pa.large_string(), ["a"], id="large_string"), + pytest.param(pa.binary(), [b"a"], id="binary"), + pytest.param(pa.large_binary(), [b"a"], id="large_binary"), +] + + +class TestWriteRefuses: + @pytest.mark.parametrize("arrow_type, values", UNSTORABLE_ON_SQLITE) + def test_sqlite_refuses_column_before_buffering( + self, tmp_path: pathlib.Path, arrow_type, values + ): + database = _sqlite_database(tmp_path) + table = pa.table({"v": pa.array(values, type=arrow_type)}) + + with pytest.raises(UnstorableColumnError) as exc_info: + database.add_record(("t",), b"1", table) + + err = exc_info.value + assert err.column == "v" + assert err.arrow_type == arrow_type + assert err.node_label is None + assert str(err).startswith("a SQLiteConnector database cannot store column") + # Nothing was buffered, so nothing is written or read back. + database.flush() + assert database.get_all_records(("t",)) is None + assert database.list_table_paths() == [] + + @pytest.mark.parametrize("arrow_type, values", STORABLE_ON_SQLITE) + def test_sqlite_stores_and_reads_back( + self, tmp_path: pathlib.Path, arrow_type, values + ): + """What SQLite accepts reads back as written, and a second write succeeds.""" + from orcapod.utils.arrow_utils import normalize_to_large_types + + database = _sqlite_database(tmp_path) + table = pa.table({"v": pa.array(values, type=arrow_type)}) + database.add_record(("t",), b"1", table, flush=True) + database.add_record(("t",), b"2", table, flush=True) + + stored = database.get_all_records(("t",)) + assert stored is not None + assert stored.schema.field("v").type == normalize_to_large_types(arrow_type) + assert stored["v"].to_pylist() == values * 2 + + def test_operator_cache_with_storable_columns_runs_on_sqlite( + self, tmp_path: pathlib.Path + ): + """Operator caches hold only the operator's columns plus binary/text ones.""" + database = _sqlite_database(tmp_path) + job = PipelineJob(name="p", database=database) + with job: + _ = SelectDataColumns(["x"])(_source(), label="sel") + job.nodes["sel"].attach_databases( + pipeline_database=database.at("p"), cache_mode=CacheMode.LOG + ) + _ = job.run() + job.nodes["sel"].clear_cache() + _ = job.run() # the second write meets the stored table's read-back types + assert sorted(job.nodes["sel"].as_table()["x"].to_pylist()) == [1, 2] + + def test_operator_cache_with_unstorable_column_refused_at_write( + self, tmp_path: pathlib.Path + ): + """An operator cache holding an int32 column is refused when it is written.""" + database = _sqlite_database(tmp_path) + source = ArrowTableSource( + pa.table( + { + "k": pa.array(["a"], type=pa.large_string()), + "x": pa.array([1], type=pa.int32()), + } + ), + tag_columns=["k"], + tag_only_columns=True, + source_id="src", + ) + job = PipelineJob(name="p", database=database) + with job: + _ = SelectDataColumns(["x"])(source, label="sel") + job.nodes["sel"].attach_databases( + pipeline_database=database.at("p"), cache_mode=CacheMode.LOG + ) + + with pytest.raises(OrcapodDatabaseWriteError) as exc_info: + _ = job.run() + + assert exc_info.value.node_label == "sel" + cause = exc_info.value.__cause__ + assert isinstance(cause, UnstorableColumnError) + assert cause.column == "x" + assert cause.arrow_type == pa.int32() + assert database.list_table_paths() == [] + + +# --------------------------------------------------------------------------- +# The rule each connector states +# --------------------------------------------------------------------------- + + +class TestConnectorRule: + def test_sqlite_rule(self): + connector = SQLiteConnector(":memory:") + try: + for arrow_type in ( + pa.bool_(), + pa.int64(), + pa.float64(), + pa.string(), + pa.large_string(), + pa.binary(), + pa.large_binary(), + ): + assert ( + connector.unstorable_column_reason( + "c", arrow_type, primary_key=False + ) + is None + ), arrow_type + reason = connector.unstorable_column_reason( + "c", pa.int32(), primary_key=False + ) + assert reason == "SQLite stores it as INTEGER, which reads back as int64" + reason = connector.unstorable_column_reason( + "c", pa.timestamp("us", tz="UTC"), primary_key=False + ) + assert reason == "SQLite has no column type for it" + finally: + connector.close() + + @pytest.mark.parametrize( + "arrow_type, storable", + [ + (pa.bool_(), True), + (pa.int8(), False), + (pa.int16(), False), + (pa.int32(), True), + (pa.int64(), True), + (pa.uint32(), False), + (pa.float32(), True), + (pa.float64(), True), + (pa.string(), True), + (pa.large_string(), True), + (pa.binary(), True), + (pa.large_binary(), True), + (pa.date32(), True), + (pa.timestamp("us"), True), + (pa.timestamp("us", tz="UTC"), True), + (pa.timestamp("ns", tz="UTC"), False), + (pa.timestamp("us", tz="America/New_York"), False), + (pa.large_list(pa.int64()), False), + (EXECUTOR_INFO_TYPE, False), + (pa.struct([("a", pa.int64())]), False), + (pa.null(), False), + ], + ) + def test_postgresql_rule(self, arrow_type, storable): + """By code reading: the type it writes reads back as the same type.""" + with patch("psycopg.connect") as mock_connect: + mock_connect.return_value = MagicMock() + connector = PostgreSQLConnector("postgresql://localhost/test") + try: + reason = connector.unstorable_column_reason( + "c", arrow_type, primary_key=False + ) + finally: + connector._conn = None + assert (reason is None) is storable, reason + + +# --------------------------------------------------------------------------- +# SpiralDB: reserved column names (ITL-1164), through the pyspiral fake +# --------------------------------------------------------------------------- + + +@pytest.fixture +def spiral_database(tmp_path: pathlib.Path): + """A ``ConnectorArrowDatabase`` over ``SpiralDBConnector``, stored by the fake.""" + from orcapod.databases import SpiralDBConnector + from tests.test_databases import fake_pyspiral + + with fake_pyspiral.install(tmp_path / "spiral"): + connector = SpiralDBConnector("p") + try: + yield ConnectorArrowDatabase(connector) + finally: + connector.close() + + +class TestSpiralDBRefusesReservedNames: + def test_function_node_refused_before_anything_runs(self, spiral_database): + job = PipelineJob(name="p", database=spiral_database) + + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = double.pod(_source(), label="dbl") + + err = exc_info.value + assert err.node_label == "dbl" + assert err.database_kind == "SpiralDBConnector" + assert err.column == "__input_data_hash" + assert "'__' prefix" in err.reason + assert "DeltaTableDatabase" in str(err) + assert CALLS == [] + assert spiral_database.list_table_paths() == [] + + def test_side_effect_node_refused_before_delivery(self, spiral_database): + job = PipelineJob(name="p", database=spiral_database) + + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = deliver.process(_source(), label="out") + + assert exc_info.value.node_label == "out" + # The first column it cannot store: the time-zone-aware timestamp. + assert exc_info.value.column == "executed_at" + assert DELIVERED == [] + + def test_operator_cache_refused_when_attached(self, spiral_database): + job = PipelineJob(name="p", database=spiral_database) + with job: + _ = SelectDataColumns(["x"])(_source(), label="sel") + + with pytest.raises(UnstorableColumnError) as exc_info: + job.nodes["sel"].attach_databases( + pipeline_database=spiral_database.at("p"), cache_mode=CacheMode.LOG + ) + + assert exc_info.value.node_label == "sel" + assert exc_info.value.column == "__node_content_hash" + + def test_operator_without_cache_runs(self, spiral_database): + """An operator whose cache is off stores nothing, so nothing is refused.""" + job = PipelineJob(name="p", database=spiral_database) + with job: + _ = SelectDataColumns(["x"])(_source(), label="sel") + _ = job.run() + assert sorted(job.nodes["sel"].as_table()["x"].to_pylist()) == [1, 2] + + def test_cached_source_refused_when_built(self, spiral_database): + from orcapod.core.sources.cached_source import CachedSource + + with pytest.raises(UnstorableColumnError) as exc_info: + _ = CachedSource(_source(), cache_database=spiral_database) + + assert exc_info.value.column == "__data_context" + + @pytest.mark.parametrize( + "name, reason", + [ + ("__value", "'__' prefix"), + ("a.b", "'.'"), + ], + ) + def test_write_refuses_reserved_name(self, spiral_database, name, reason): + table = pa.table({name: pa.array([1], type=pa.int64())}) + + with pytest.raises(UnstorableColumnError) as exc_info: + spiral_database.add_record(("t",), b"1", table) + + assert exc_info.value.column == name + assert reason in exc_info.value.reason + spiral_database.flush() + assert spiral_database.list_table_paths() == [] + + def test_write_refuses_time_zone_aware_timestamp(self, spiral_database): + """SpiralDB drops the time zone, so the column would read back as another type.""" + table = pa.table( + { + "ts": pa.array( + [datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc)], + type=pa.timestamp("us", tz="UTC"), + ) + } + ) + + with pytest.raises(UnstorableColumnError) as exc_info: + spiral_database.add_record(("t",), b"1", table) + + assert exc_info.value.column == "ts" + assert "time zone" in exc_info.value.reason + assert "timestamp[us]" in exc_info.value.reason + spiral_database.flush() + assert spiral_database.list_table_paths() == [] + + def test_key_column_may_start_with_double_underscore(self, spiral_database): + """``__record_id`` is the key, which SpiralDB accepts; lists are stored.""" + table = pa.table( + { + "v": pa.array([[{"key": "a", "value": "b"}]], type=EXECUTOR_INFO_TYPE), + "ts": pa.array( + [datetime.datetime(2026, 1, 1)], type=pa.timestamp("us") + ), + } + ) + spiral_database.add_record(("t",), b"1", table, flush=True) + stored = spiral_database.get_all_records(("t",)) + assert stored is not None + assert stored.num_rows == 1 + + +class TestSpiralFakeRefusesLikeSpiralDB: + """The fake refuses the names SpiralDB refuses, with its messages (ITL-1164).""" + + @pytest.mark.parametrize( + "name, message", + [ + ( + "__input_data_hash", + "Column name '__input_data_hash' has reserved prefix: column group " + "columns must not start with '__'", + ), + ("a.b", "Column 'a.b' contains a reserved dot-separator"), + ], + ) + def test_connector_write_refused(self, tmp_path: pathlib.Path, name, message): + from orcapod.databases import SpiralDBConnector + from orcapod.types import ColumnInfo + from tests.test_databases import fake_pyspiral + + with fake_pyspiral.install(tmp_path / "spiral"): + connector = SpiralDBConnector("p") + connector.create_table_if_not_exists( + "t", + [ + ColumnInfo("k", pa.large_string(), nullable=False), + ColumnInfo(name, pa.int64(), nullable=True), + ], + pk_column="k", + ) + rows = pa.table( + { + "k": pa.array(["a"], type=pa.large_string()), + name: pa.array([1], type=pa.int64()), + } + ) + with pytest.raises(ValueError, match="reserved") as exc_info: + connector.upsert_records("t", rows, id_column="k") + connector.close() + assert message in str(exc_info.value) + + +# --------------------------------------------------------------------------- +# Every stored table of a node is checked, also through a wrapper +# --------------------------------------------------------------------------- + + +class TestEveryTableChecked: + def test_function_node_pipeline_database_checked(self, tmp_path: pathlib.Path): + """A SQLite pipeline database is refused even when results go elsewhere.""" + job = PipelineJob(name="p", database=InMemoryArrowDatabase()) + with job: + _ = double.pod(_source(), label="dbl") + node = job.nodes["dbl"] + + with pytest.raises(UnstorableColumnError) as exc_info: + node.attach_databases( + pipeline_database=_sqlite_database(tmp_path), + result_database=InMemoryArrowDatabase(), + ) + + assert exc_info.value.node_label == "dbl" + assert exc_info.value.column == "__pipeline_recomputation_index" + assert exc_info.value.arrow_type == pa.int32() + + def test_extension_aware_wrapper_checked_before_delivery( + self, tmp_path: pathlib.Path + ): + """The check reaches a connector database inside ``ExtensionAwareDatabase``.""" + from orcapod.contexts import get_default_context + from orcapod.databases import ExtensionAwareDatabase + + # The wrapper's annotations do not match the protocols (existing debt). + converter: Any = get_default_context().type_converter + wrapped: Any = ExtensionAwareDatabase( + _sqlite_database(tmp_path), converter=converter + ) + job = PipelineJob(name="p", database=wrapped) + + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = deliver.process(_source(), label="out") + + assert exc_info.value.node_label == "out" + assert exc_info.value.column == "executed_at" + assert DELIVERED == [] + + def test_extension_aware_wrapper_over_in_memory_not_checked(self): + """A wrapped database that states no rule stores every column.""" + from orcapod.contexts import get_default_context + from orcapod.databases import ExtensionAwareDatabase + + converter: Any = get_default_context().type_converter + wrapped: Any = ExtensionAwareDatabase( + InMemoryArrowDatabase(), converter=converter + ) + job = PipelineJob(name="p", database=wrapped) + with job: + _ = double.pod(_source(), label="dbl") + _ = job.run() + assert sorted(CALLS) == [1, 2] + + +class TestRefusedBindLeavesJobAsItWas: + def test_refused_database_bind_changes_nothing(self, tmp_path: pathlib.Path): + """Every node is checked before any is attached (CodeRabbit, ITL-1162).""" + in_memory = InMemoryArrowDatabase() + job = PipelineJob(name="p", database=in_memory) + with job: + out = double.pod(_source(), label="dbl") + _ = SelectDataColumns(["double"])(out, label="sel") + _ = job.run() + assert sorted(CALLS) == [1, 2] + sqlite = _sqlite_database(tmp_path) + + with pytest.raises(UnstorableColumnError) as exc_info: + job.bind(database=sqlite) + + assert exc_info.value.node_label == "dbl" + # The job keeps its database: a new run reads the stored results. + job.nodes["dbl"].clear_cache() + CALLS.clear() + _ = job.run() + assert CALLS == [] + assert sorted(job.nodes["dbl"].as_table()["double"].to_pylist()) == [2, 4] + assert sqlite.list_table_paths() == [] + + +class TestWriteChecksOrder: + def test_extension_type_refused_as_unstorable_column(self, tmp_path: pathlib.Path): + """The type rule runs before the metadata check (CodeRabbit, ITL-1162).""" + database = _sqlite_database(tmp_path) + table = pa.table({"u": pa.array([b"0123456789abcdef"], type=pa.uuid())}) + + with pytest.raises(UnstorableColumnError) as exc_info: + database.add_record(("t",), b"1", table) + + assert exc_info.value.column == "u" + + def test_null_column_is_not_a_large_string_column(self, spiral_database): + """The schema comparison keeps ``null``: a later ``large_string`` is a change.""" + null_table = pa.table({"v": pa.array([None], type=pa.null())}) + text_table = pa.table({"v": pa.array(["a"], type=pa.large_string())}) + spiral_database.add_record(("t",), b"1", null_table, flush=True) + + spiral_database.add_record(("t",), b"2", text_table) + with pytest.raises(ValueError, match="Schema mismatch"): + spiral_database.flush() + + def test_buffered_text_reads_with_committed_text(self, tmp_path: pathlib.Path): + """Buffered ``string`` rows read together with committed ``large_string`` rows.""" + database = _sqlite_database(tmp_path) + table = pa.table({"v": pa.array(["a"], type=pa.string())}) + database.add_record(("t",), b"1", table, flush=True) + database.add_record(("t",), b"2", table) + + stored = database.get_all_records(("t",)) + + assert stored is not None + assert stored.schema.field("v").type == pa.large_string() + assert stored["v"].to_pylist() == ["a", "a"] + + @pytest.mark.parametrize( + "arrow_type, values", + [ + pytest.param(pa.string_view(), ["a"], id="string_view"), + pytest.param(pa.binary_view(), [b"a"], id="binary_view"), + ], + ) + def test_view_types_stored_in_large_variant( + self, tmp_path: pathlib.Path, arrow_type, values + ): + """View layouts are normalized before the check, as every table is.""" + database = _sqlite_database(tmp_path) + table = pa.table({"v": pa.array(values, type=arrow_type)}) + database.add_record(("t",), b"1", table, flush=True) + + stored = database.get_all_records(("t",)) + + assert stored is not None + assert stored["v"].to_pylist() == values + + +# --------------------------------------------------------------------------- +# PostgreSQL truncates identifiers to 63 bytes; SQLite has no such limit +# --------------------------------------------------------------------------- + + +def _mocked_postgresql() -> tuple[PostgreSQLConnector, MagicMock]: + """A ``PostgreSQLConnector`` over a mocked connection (no server).""" + with patch("psycopg.connect") as mock_connect: + conn = MagicMock() + mock_connect.return_value = conn + return PostgreSQLConnector("postgresql://localhost/test"), conn + + +class TestIdentifierLength: + @pytest.mark.parametrize( + "name, storable", + [ + ("a" * 63, True), + ("a" * 64, False), + ("é" * 31, True), # 62 bytes + ("é" * 32, False), # 64 bytes + ("_tag_source_id::" + "0" * 64, False), # a system-tag column name + ], + ) + def test_postgresql_rule(self, name, storable): + connector, _ = _mocked_postgresql() + try: + reason = connector.unstorable_column_reason( + name, pa.int64(), primary_key=False + ) + finally: + connector._conn = None + assert (reason is None) is storable, reason + if not storable: + assert reason is not None and "63 bytes" in reason + + def test_postgresql_refuses_long_name_before_anything_is_written(self): + connector, conn = _mocked_postgresql() + try: + database = ConnectorArrowDatabase(connector) + name = "_tag_source_id::" + "0" * 64 + table = pa.table({name: pa.array(["s"], type=pa.large_string())}) + calls_before = len(conn.mock_calls) + + with pytest.raises(UnstorableColumnError) as exc_info: + database.add_record(("t",), b"1", table) + database.flush() # nothing was buffered + + assert exc_info.value.column == name + assert exc_info.value.database_kind == "PostgreSQLConnector" + assert len(conn.mock_calls) == calls_before # no statement was sent + finally: + connector._conn = None + + def test_sqlite_keeps_long_names(self, tmp_path: pathlib.Path): + """SQLite has no identifier length limit: a long name reads back as written.""" + database = _sqlite_database(tmp_path) + name = "_tag_source_id::" + "0" * 200 + table = pa.table({name: pa.array(["s"], type=pa.large_string())}) + database.add_record(("t",), b"1", table, flush=True) + + stored = database.get_all_records(("t",)) + + assert stored is not None + assert name in stored.column_names + + +class TestPipelineRecordsAndEphemeralResults: + def test_ephemeral_database_checked_before_anything_changes( + self, tmp_path: pathlib.Path + ): + job = PipelineJob(name="p", database=InMemoryArrowDatabase()) + with job: + _ = double.pod(_source(), label="dbl") + + with pytest.raises(UnstorableColumnError) as exc_info: + job.set_ephemeral_database(_sqlite_database(tmp_path)) + + assert exc_info.value.node_label == "dbl" + assert exc_info.value.column == EXECUTOR_INFO + # No node took the database: a run computes and stores as before. + assert job.nodes["dbl"].ephemeral_result_database is None + _ = job.run() + assert sorted(CALLS) == [1, 2] + + def test_attach_with_refused_ephemeral_database_changes_nothing( + self, tmp_path: pathlib.Path + ): + in_memory = InMemoryArrowDatabase() + job = PipelineJob(name="p", database=in_memory) + with job: + _ = double.pod(_source(), label="dbl") + _ = job.run() + node = job.nodes["dbl"] + + with pytest.raises(UnstorableColumnError): + node.attach_databases( + pipeline_database=InMemoryArrowDatabase(), + ephemeral_database=_sqlite_database(tmp_path), + ) + + # The node kept its databases and its run: nothing is recomputed. + CALLS.clear() + assert sorted(node.as_table()["double"].to_pylist()) == [2, 4] + assert node.ephemeral_result_database is None + assert CALLS == [] + + +# --------------------------------------------------------------------------- +# In-PR CodeRabbit review on 807166aa +# --------------------------------------------------------------------------- + + +class TestReviewFollowUps: + def test_error_survives_pickling(self): + import copy + import pickle + + err = UnstorableColumnError( + database_kind="SQLiteConnector", + column="v", + arrow_type=pa.int32(), + reason="r", + node_label="dbl", + ) + for again in (pickle.loads(pickle.dumps(err)), copy.copy(err)): + assert isinstance(again, UnstorableColumnError) + assert (again.column, again.arrow_type, again.node_label) == ( + "v", + pa.int32(), + "dbl", + ) + assert str(again) == str(err) + + @pytest.mark.parametrize( + "name", ["xmin", "xmax", "cmin", "cmax", "ctid", "tableoid"] + ) + def test_postgresql_refuses_system_column_names(self, name): + connector, _ = _mocked_postgresql() + try: + reason = connector.unstorable_column_reason( + name, pa.int64(), primary_key=False + ) + finally: + connector._conn = None + assert reason is not None and "system column" in reason + + @pytest.mark.parametrize( + "arrow_type", + [ + pa.large_list(pa.timestamp("us", tz="UTC")), + pa.struct([("t", pa.timestamp("us", tz="UTC"))]), + pa.map_(pa.large_string(), pa.timestamp("ms", tz="UTC")), + ], + ) + def test_spiraldb_refuses_nested_time_zone_aware_timestamp( + self, spiral_database, arrow_type + ): + table = pa.table({"v": pa.array([None], type=arrow_type)}) + + with pytest.raises(UnstorableColumnError) as exc_info: + spiral_database.add_record(("t",), b"1", table) + + assert "time zone" in exc_info.value.reason + + def test_union_refused_before_any_cast(self, tmp_path: pathlib.Path): + """A refused column raises the refusal, not an Arrow cast error.""" + database = _sqlite_database(tmp_path) + union = pa.UnionArray.from_dense( + pa.array([0], type=pa.int8()), + pa.array([0], type=pa.int32()), + [pa.array(["a"], type=pa.string())], + ) + table = pa.table({"u": union}) + + with pytest.raises(UnstorableColumnError) as exc_info: + database.add_record(("t",), b"1", table) + + assert exc_info.value.column == "u" + + def test_nested_cached_pod_checked(self, tmp_path: pathlib.Path): + """A pod with its own cache database is checked with the job's (ITL-1162).""" + sqlite = _sqlite_database(tmp_path) + + @function_pod(output_keys="double", pod_cache_database=sqlite) + def cached_double(x: int) -> int: + CALLS.append(x) + return 2 * x + + job = PipelineJob(name="p", database=InMemoryArrowDatabase()) + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = cached_double.pod(_source(), label="dbl") + + assert exc_info.value.node_label == "dbl" + assert exc_info.value.database_kind == "SQLiteConnector" + assert CALLS == [] + + def test_sql_type_maps_accept_view_types(self, tmp_path: pathlib.Path): + """CREATE TABLE maps view layouts as the rule reads them (CLI run 12).""" + from orcapod.types import ColumnInfo + + connector = SQLiteConnector(str(tmp_path / "views.db")) + try: + connector.create_table_if_not_exists( + "t", + [ + ColumnInfo("k", pa.string_view(), nullable=False), + ColumnInfo("b", pa.binary_view(), nullable=True), + ], + pk_column="k", + ) + types = {c.name: c.arrow_type for c in connector.get_column_info("t")} + finally: + connector.close() + assert types == {"k": pa.large_string(), "b": pa.large_binary()} + from orcapod.databases.postgresql_connector import _arrow_type_to_pg_sql + + assert _arrow_type_to_pg_sql(pa.string_view()) == "TEXT" + assert _arrow_type_to_pg_sql(pa.binary_view()) == "BYTEA" + + +# --------------------------------------------------------------------------- +# PostgreSQL table names over 63 bytes (coordinator ruling, ITL-1162) +# --------------------------------------------------------------------------- + + +class TestTableNameLength: + def test_connector_rules(self, tmp_path: pathlib.Path): + connector, _ = _mocked_postgresql() + try: + assert connector.unstorable_table_name_reason("t" * 63) is None + reason = connector.unstorable_table_name_reason("t" * 64) + assert reason is not None and "63 bytes" in reason + finally: + connector._conn = None + sqlite = SQLiteConnector(str(tmp_path / "names.db")) + try: + assert sqlite.unstorable_table_name_reason("t" * 500) is None + finally: + sqlite.close() + + def test_every_table_entry_refuses_a_long_name(self): + """Write, read and check all refuse before anything is sent to the server.""" + from orcapod.errors import UnstorableTableNameError + + connector, conn = _mocked_postgresql() + try: + database = ConnectorArrowDatabase(connector) + path = ("fn", "x" * 70) + table = pa.table({"v": pa.array([1], type=pa.int64())}) + calls_before = len(conn.mock_calls) + + with pytest.raises(UnstorableTableNameError) as exc_info: + database.add_record(path, b"1", table) + with pytest.raises(UnstorableTableNameError): + database.get_all_records(path) + with pytest.raises(UnstorableTableNameError): + database.table_exists(path) + database.flush() # nothing was buffered + + assert len(conn.mock_calls) == calls_before + err = exc_info.value + assert err.database_kind == "PostgreSQLConnector" + assert err.table_name == "fn__" + "x" * 70 + assert "DeltaTableDatabase" in str(err) + finally: + connector._conn = None + + def test_connector_entries_refuse_a_long_name(self): + from orcapod.errors import UnstorableTableNameError + from orcapod.types import ColumnInfo + + connector, conn = _mocked_postgresql() + name = "t" * 64 + try: + calls_before = len(conn.mock_calls) + with pytest.raises(UnstorableTableNameError): + connector.create_table_if_not_exists( + name, [ColumnInfo("k", pa.large_string(), False)], pk_column="k" + ) + with pytest.raises(UnstorableTableNameError): + connector.get_column_info(name) + with pytest.raises(UnstorableTableNameError): + connector.get_pk_columns(name) + with pytest.raises(UnstorableTableNameError): + connector.upsert_records(name, pa.table({"k": ["a"]}), id_column="k") + assert len(conn.mock_calls) == calls_before + finally: + connector._conn = None + + def test_cached_source_on_postgresql_refused(self): + """A cached source's table name holds a hash: over 63 bytes on PostgreSQL.""" + from orcapod.core.sources.cached_source import CachedSource + from orcapod.errors import UnstorableTableNameError + + connector, conn = _mocked_postgresql() + try: + cached = CachedSource( + _source(), cache_database=ConnectorArrowDatabase(connector) + ) + calls_before = len(conn.mock_calls) + with pytest.raises(Exception) as exc_info: + _ = cached.as_table() + chain: list[BaseException] = [] + err: BaseException | None = exc_info.value + while err is not None: + chain.append(err) + err = err.__cause__ or err.__context__ + assert any(isinstance(e, UnstorableTableNameError) for e in chain), chain + assert len(conn.mock_calls) == calls_before + finally: + connector._conn = None + + def test_pickling(self): + import pickle + + from orcapod.errors import UnstorableTableNameError + + err = UnstorableTableNameError( + database_kind="PostgreSQLConnector", table_name="t", reason="r" + ) + again = pickle.loads(pickle.dumps(err)) + assert str(again) == str(err) + + +class TestInPRReviewOn45e5e44e: + def test_data_function_result_database_checked(self, tmp_path: pathlib.Path): + """``function_pod(result_database=...)`` caches in its own database too.""" + sqlite = _sqlite_database(tmp_path) + + @function_pod(output_keys="double", result_database=sqlite) + def cached_double(x: int) -> int: + CALLS.append(x) + return 2 * x + + job = PipelineJob(name="p", database=InMemoryArrowDatabase()) + with pytest.raises(UnstorableColumnError) as exc_info: + with job: + _ = cached_double.pod(_source(), label="dbl") + + assert exc_info.value.node_label == "dbl" + assert exc_info.value.database_kind == "SQLiteConnector" + assert CALLS == [] + + @pytest.mark.parametrize("name", ["sqlite_results", "SQLite_x", "SQLITE_"]) + def test_sqlite_reserved_table_prefix(self, tmp_path: pathlib.Path, name): + from orcapod.errors import UnstorableTableNameError + + database = _sqlite_database(tmp_path) + table = pa.table({"v": pa.array([1], type=pa.int64())}) + + with pytest.raises(UnstorableTableNameError): + database.add_record((name,), b"1", table) + database.flush() + assert database.list_table_paths() == [] diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index fb9a7cfe8..ffd8e1757 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -20,7 +20,7 @@ from orcapod import PipelineJob, function_pod from orcapod.channels import Channel from orcapod.core.operators import SelectDataColumns -from orcapod.errors import OrcapodDatabaseWriteError +from orcapod.errors import OrcapodDatabaseWriteError, UnstorableColumnError from orcapod.databases import ( ConnectorArrowDatabase, InMemoryArrowDatabase, @@ -158,7 +158,8 @@ def double_or_fail(x: int) -> int: @function_pod(output_keys="where") def to_path(x: int) -> pathlib.Path: - """Return a ``pathlib.Path`` output, a logical type SQLite cannot store.""" + """Return a ``pathlib.Path`` output, a logical type SQLite cannot store; record the call.""" + calls.append(x) return pathlib.Path(f"/data/{x}.txt") @@ -460,24 +461,24 @@ def test_final_flush_failure_warns_pending_output(self, orchestrator_cls): # --------------------------------------------------------------------------- -# The reported trigger: a logical-type output on SQLite (ITL-877 is the limit itself) +# The reported trigger: a logical-type output on SQLite (ITL-877 is the limit itself). +# Since ITL-1162 a SQLite database refuses every function node when the job is built. # --------------------------------------------------------------------------- class TestSQLiteTrigger: - """The reported trigger: a logical-type output on SQLite.""" + """The reported trigger: a function node on SQLite, refused before it runs.""" - def test_logical_type_output_on_sqlite_raises(self, orchestrator_cls, tmp_path): - """A ``pathlib.Path`` output on SQLite raises, naming the node.""" + def test_function_node_on_sqlite_refused_naming_the_node(self, tmp_path): + """The result table's executor columns cannot be stored; the node is named.""" store = ConnectorArrowDatabase(SQLiteConnector(str(tmp_path / "store.db"))) - job = build_job(store, ROWS, pod=to_path) - with pytest.raises(OrcapodDatabaseWriteError) as exc_info: - job.run(orchestrator=orchestrator_cls()) + with pytest.raises(UnstorableColumnError) as exc_info: + build_job(store, ROWS, pod=to_path) assert exc_info.value.node_label == "dbl" assert "node 'dbl'" in str(exc_info.value) - assert "does not preserve Arrow metadata" in str(exc_info.value) + assert calls == [] # ---------------------------------------------------------------------------