Repository navigation
fix(databases): refuse a table a connector database cannot store (ITL-1162, ITL-1164) #408
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f430c4c
859a5bf
9243230
d03ef6f
c0a43bb
7307f95
42e4377
50cec09
06b8f8b
807166a
a8ecac1
d464ace
5e92c00
25e8d93
45e5e44
70a02b0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| ### Fixed | ||
|
|
||
| - A `ConnectorArrowDatabase` over `SQLiteConnector` or `PostgreSQLConnector` now refuses a table it cannot store before anything of it is written, with `UnstorableColumnError` naming the connector, the column, its Arrow type and what to use instead (ITL-1162). A connector stores a column only when it reads it back as written: on SQLite `bool`, `int64`, `double`, `large_string` and `large_binary`; on PostgreSQL also `int32`, `float`, `date32`, `timestamp[us]` and `timestamp[us, tz=UTC]`. Before, a function node on such a database ran its function and then failed at commit (`sqlite3.ProgrammingError: ... type 'list' is not supported`), timestamps were stored as text and read back as strings, and narrower integers and floats read back widened, failing the next write. A job now refuses a node whose stored table such a database cannot hold (every function node; a side-effect node on SQLite) when the job is built, before anything runs, naming the node. The connectors no longer fall back to a `TEXT` column for a type they have no column type for. | ||
| - `PostgreSQLConnector` now refuses a column name longer than 63 bytes (UTF-8), which PostgreSQL truncates, so it would read back as another name (ITL-1162). Before, such a column was stored under the truncated name. A table name longer than 63 bytes is refused too, with `UnstorableTableNameError`, at every entry that creates, writes, reads or checks a table, so a truncated name is never sent: two tables whose names share their first 63 bytes would otherwise silently be one. `SQLiteConnector` refuses a table name starting with `sqlite_`, which SQLite reserves. | ||
| - A `ConnectorArrowDatabase` over `SpiralDBConnector` now refuses, by the same rule, a column SpiralDB cannot store: a value column whose name starts with `__` (the `__record_id` key is allowed) or any column whose name contains `.`, and a timestamp column with a time zone, which SpiralDB reads back without it (ITL-1164). Before, a function node on such a database ran and then failed at commit (`Column name '__input_data_hash' has reserved prefix`). A job refuses every function node, side-effect node, operator cache and cached source on such a database when it is built. | ||
|
|
||
| ### Known limitations | ||
|
|
||
| - A connector job database cannot hold function nodes in v0.3.0: neither SQL connector (SQLite, PostgreSQL) nor SpiralDB. A function node's result table holds list-of-struct executor columns, which neither SQL connector stores, and `__`-prefixed value columns, which SpiralDB reserves (ITL-1164). A side-effect node is refused on SQLite (its log's UTC timestamp) and on SpiralDB, and an operator cache or cached source on SpiralDB (`__data_context`). Use a Delta database (`DeltaTableDatabase`) or `InMemoryArrowDatabase`. Storing such columns on connector databases is planned for a later release (ITL-1162, ITL-1164). | ||
| - On PostgreSQL, an operator cache with caching on is refused in v0.3.0 when it is written (before anything is stored): its system-tag column names (`_tag_source_id::` and a 64-character hash) are longer than PostgreSQL's 63-byte limit. Storing them under a mapped name is planned for a later release (ITL-1166). | ||
| - On PostgreSQL in v0.3.0, a table whose Orcapod path gives a name longer than 63 bytes is refused. Every node table and cached source path holds a 64-character hash, so only tables written directly through `ConnectorArrowDatabase` under short paths fit; reading an existing table with `PostgreSQLTableSource` is unaffected. Mapping long names is planned for a later release (ITL-1166). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| ## `src/orcapod/databases/connector_arrow_database.py` | ||
|
|
||
| ### CA3 — Connector databases cannot store Orcapod's own stored tables (list, struct and timestamp columns; SpiralDB reserved names) | ||
| **Status:** open | ||
| **Severity:** high | ||
| **Issue:** ITL-1162 | ||
|
|
||
| A `ConnectorArrowDatabase` over `SQLiteConnector` or `PostgreSQLConnector` cannot store a | ||
| function node's result table. Its `__pf_exec_executor_info` and `__pf_exec_extra_info` | ||
| columns are `large_list<struct<key, value>>`. The SQLite connector logged "Unsupported Arrow | ||
| type ...; mapping to TEXT" and then failed to bind a Python list (`sqlite3.ProgrammingError`) | ||
| at commit, after the function had run. The same table's `__pod_ts` (`timestamp[us, tz=UTC]`) | ||
| and a side-effect log's `executed_at` were stored as TEXT and read back as `large_string`. | ||
| `int8`, `int32` and `float32` columns read back widened, so the second write to the table | ||
| failed its schema comparison. PostgreSQL raises for list columns at `CREATE TABLE` and has no | ||
| column type for structs, and truncates a column name to 63 bytes, so an operator cache's | ||
| system-tag columns (`_tag_source_id::<64-hex>`) read back under other names. Its table names | ||
| (the node path joined by `__`, 109-173 characters) are truncated the same way, so two tables | ||
| sharing their first 63 bytes would silently be one. SpiralDB refuses value columns named with a `__` prefix and any column | ||
| name containing `.`, and drops a timestamp's time zone (ITL-1164), so it cannot hold a function | ||
| result table (`__input_data_hash`, ...) or any table's `__data_context`. | ||
|
|
||
| **Fix (refusal; developer ruling, November 16, 2026):** each connector states which columns | ||
| it stores, in one place (`DBConnectorProtocol.unstorable_column_reason(name, arrow_type, | ||
| primary_key=...)`): a column it reads back as written, with the same Arrow type in its large | ||
| variant. `ConnectorArrowDatabase.require_storable` raises `UnstorableColumnError` (connector, | ||
| column, Arrow type, reason, what to use instead) in `add_records`, before anything is | ||
| buffered. A function job node (its result table) and a side-effect job node (its invocation | ||
| log) ask the database when it is attached, so a job refuses them before anything runs, naming | ||
| the node. An operator job node with a cache and a cached source ask about their bookkeeping | ||
| columns (`__node_content_hash`, `__data_context`) when attached or built; their user columns | ||
| are checked at the write. `SpiralDBConnector` states its name rule the same way (developer | ||
| ruling, November 16, 2026). `PostgreSQLConnector` also states which table names it holds | ||
| (`unstorable_table_name_reason`, 63 bytes), checked at every table entry | ||
| (`UnstorableTableNameError`). The connectors' silent TEXT fallback in `CREATE TABLE` now | ||
| raises. | ||
|
|
||
| **Open (the real fix, a later release):** store list and struct columns (for example as JSON | ||
| text read back as the same Arrow type) and timestamps on SQLite, and rename SpiralDB's | ||
| reserved column names on write and recover them on read, and restore a timestamp's time | ||
| zone (ITL-1164); map PostgreSQL column and table names over 63 bytes the same way (ITL-1166), so a function node can | ||
| run on a connector database. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,6 +8,7 @@ | |
| from collections.abc import Iterator | ||
| from typing import TYPE_CHECKING, Any | ||
|
|
||
| from orcapod.core.data_function import CachedDataFunction | ||
| from orcapod.core.datagrams import Datagram | ||
| from orcapod.core.function_pod import ( | ||
| FunctionPodStream, | ||
|
|
@@ -17,8 +18,12 @@ | |
| from orcapod.core.streams.view import StreamView | ||
| from orcapod.system_constants import constants | ||
| from orcapod.types import ColumnConfig, Schema | ||
| from orcapod.contexts.stored import StaleTables | ||
| from orcapod.core.result_cache import ResultCache | ||
| from orcapod.contexts.stored import ( | ||
| StaleTables, | ||
| data_context_field, | ||
| require_storable, | ||
| ) | ||
| from orcapod.core.result_cache import ResultCache, pod_timestamp_type | ||
| from orcapod.hooks import InvocationStatus | ||
| from orcapod.protocols.core_protocols import ( | ||
| FunctionPodProtocol, | ||
|
|
@@ -71,6 +76,53 @@ def __init__( | |
| name_path=self.uri[: 2 if function_pod.ctx_arg_name is not None else 1], | ||
| ) | ||
|
|
||
| def require_storable_results(self) -> None: | ||
| """Refuse a result database that cannot store this pod's result table (ITL-1162). | ||
|
|
||
| The result table holds the input data hash, the function's variation and | ||
| execution columns (``__pf_var_*``, ``__pf_exec_*``), its output columns, | ||
| the ``__pod_ts`` timestamp and ``__data_context``, typed from their declared | ||
| schemas; nothing is computed. (The source-info columns are text.) A | ||
| database that states which columns it stores (``ConnectorArrowDatabase``) | ||
| refuses here, before the function runs; any other database accepts. | ||
|
|
||
| Raises: | ||
| UnstorableColumnError: If the result database cannot store a column. | ||
| """ | ||
| import pyarrow as pa | ||
|
|
||
| pf = self._function_pod.data_function | ||
| python_schema = { | ||
| **{ | ||
| f"{constants.PF_VARIATION_PREFIX}{name}": hint | ||
| for name, hint in pf.get_function_variation_data_schema().items() | ||
| }, | ||
| **{ | ||
| f"{constants.PF_EXECUTION_PREFIX}{name}": hint | ||
| for name, hint in pf.get_execution_data_schema().items() | ||
| }, | ||
| **pf.output_data_schema, | ||
| } | ||
| declared = self.data_context.type_converter.python_schema_to_arrow_schema( | ||
| python_schema | ||
| ) | ||
| schema = pa.schema( | ||
| [ | ||
| pa.field(constants.INPUT_DATA_HASH_COL, pa.large_binary()), | ||
| *declared, | ||
| pa.field(constants.POD_TIMESTAMP, pod_timestamp_type()), | ||
| data_context_field(), | ||
| ] | ||
| ) | ||
| require_storable(self._cache.result_database, schema) | ||
| # A function with its own result database (``function_pod(result_database=)``) | ||
| # stores the same table there. | ||
| if isinstance(pf, CachedDataFunction): | ||
| require_storable(pf.result_database, schema) | ||
| # A pod with its own cache database (``pod_cache_database``) stores there too. | ||
| if isinstance(self._function_pod, CachedFunctionPod): | ||
| self._function_pod.require_storable_results() | ||
|
Comment on lines
+123
to
+124
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| @property | ||
| def result_database(self) -> ArrowDatabaseProtocol: | ||
| """The underlying result database.""" | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.