Skip to content

fix(databases): refuse a table a connector database cannot store (ITL-1162, ITL-1164) - #408

Merged
eywalker merged 16 commits into
mainfrom
eywalker/itl-1162-sqlite-connector-database-cannot-store-a
Nov 17, 2026
Merged

eywalker merged 16 commits into
mainfrom
eywalker/itl-1162-sqlite-connector-database-cannot-store-a

Conversation

@eywalker

@eywalker eywalker commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes ITL-1162
Related: ITL-1164 (SpiralDB reserved names: refused here; the real fix, renaming on write, is not in this PR)
Related: ITL-1166 (PostgreSQL column and table names over 63 bytes: refused here; the real fix, name mapping, is not in this PR)

What

A ConnectorArrowDatabase now refuses a table its connector cannot store before anything is written, and a job refuses a node whose stored table holds such a column before anything runs. Developer rulings, 2026-11-16: refuse now (SQL types, and SpiralDB reserved names); storing these columns is for a later release.

  • One rule, stated by the connector: DBConnectorProtocol.unstorable_column_reason(name, arrow_type, *, primary_key) returns a reason, or None when the column can be stored.

    • SQLite / PostgreSQL store a column only when it reads back as written (same Arrow type in its large variant). This is computed from each connector's own write map and read map.
      • SQLite: bool, int64, double, large_string, large_binary.
      • PostgreSQL (by code reading): those plus int32, float, date32, timestamp[us], timestamp[us, tz=UTC]. PostgreSQL also refuses a column name longer than 63 bytes, which it truncates (_PG_MAX_IDENTIFIER_BYTES; coordinator ruling on a CodeRabbit finding). SQLite has no identifier length limit (tested with a 216-byte name).
    • SpiralDB refuses a value column named with a __ prefix (the __record_id key is allowed), any column name containing ., and a time-zone-aware timestamp column (SpiralDB drops the zone; coordinator decision on a CodeRabbit finding).
  • ConnectorArrowDatabase.require_storable(schema) raises UnstorableColumnError, naming the connector, the column, its Arrow type, the reason, and what to use instead (InMemoryArrowDatabase / DeltaTableDatabase). It runs in add_records before anything is buffered. ExtensionAwareDatabase forwards it.

  • Up front (types and names from declared schemas; nothing is computed):

    • a function job node checks its result table and its pipeline records at attach;
    • a side-effect job node checks its invocation log at attach;
    • an operator job node with caching on checks its bookkeeping columns at attach;
    • a cached source checks its bookkeeping columns when it is built.

    The node label is added by the caller (add_context). Each node kind states its check once (require_storable_tables), and PipelineJob checks every node before attaching any, in bind and at compile. A refused database leaves the job as it was.

  • Table names: each connector states which table names it holds (unstorable_table_name_reason). PostgreSQL refuses a name over 63 bytes, which it would truncate, so two tables sharing their first 63 bytes would be one. ConnectorArrowDatabase refuses such a name with UnstorableTableNameError at every entry that creates, writes, reads or checks a table, and PostgreSQLConnector guards its own table entries, so a truncated name is never sent (coordinator ruling). Node tables and cached sources hold a 64-character hash in their path, so on PostgreSQL only short, directly written paths fit. No in-suite PostgreSQL scenario that worked before is affected.

  • No silent TEXT fallback: the SQLite and PostgreSQL CREATE TABLE type maps raise for a type they have no column type for.

  • Records are normalized to their large variants (view types included) before the check and buffering. The flush schema check compares large-normalized types (keep_null=True), so a string column no longer fails its second write, and buffered and committed rows read together.

  • tests/test_databases/fake_pyspiral.py refuses the reserved names on write, with SpiralDB's messages. 0 of 495 fake-based tests relied on the fake accepting them.

Which nodes are refused

Stored table SQLite PostgreSQL SpiralDB
Function result (rdb_v2) + pipeline (pdb_v2) refused up front (__pf_exec_executor_info list) refused up front (same) refused up front (__input_data_hash)
Side-effect log (tdb_v2) refused up front (executed_at timestamp) accepted refused up front (executed_at time zone, __data_context)
Operator cache (odb_v1), caching on bookkeeping OK; a user column it cannot store is refused at the write refused at the write, before anything is stored (system-tag names over 63 bytes) refused up front (__node_content_hash)
Cached source (sdb_v1) bookkeeping OK; a user column it cannot store is refused at the write refused at its first read or write (table name over 63 bytes) refused when built (__data_context)

Timestamps

timestamp[us, tz=UTC] does not round-trip on SQLite (stored as TEXT, read back as large_string). Neither does date32, and int8 / int32 / float32 read back widened, so the next write failed the schema check. All of these now fall under the same refusal. On PostgreSQL, timestamp[us] and timestamp[us, tz=UTC] are accepted by code reading; other units and time zones are refused.

Not in scope

  • Storing list/struct (and SQLite timestamp) columns: the open part of design_issues/ITL-1162-CA3.md.
  • ITL-1164's later fix must restore what SpiralDB refuses here: rename the reserved __ / . column names on write and recover them on read, and restore a timestamp's time zone.
  • Field metadata / the referential marker on SQL connectors.

Verification

  • tests/test_databases/test_unstorable_columns.py (public interface) covers:
    • up-front refusals with no call, no delivery and no table written: SQLite, PostgreSQL (mocked psycopg.connect) and SpiralDB (the fake);
    • per-type write refusals, and round-trips of what SQLite stores;
    • operator caches and cached sources;
    • the ExtensionAwareDatabase path;
    • the fake's messages.
  • PostgreSQL: the rule has unit tests (mocked connection). Live tests in test_postgresql_connector_integration.py run in CI's Run PostgreSQL Tests workflow, not locally: every type the connector accepts round-trips twice, a function node job is refused, and a long name / operator cache is refused with no table created.
  • SpiralDB: verified against the fake only; the coordinator runs the live check.
  • No golden file touched; golden suite passes.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Jobs and database attachments now check storage compatibility before execution or state changes, reporting unsupported columns or table names.
    • Documentation describes connector storage limits and configurations that cannot be used with connector-backed job databases.
  • Bug Fixes

    • PostgreSQL rejects unsupported types, reserved system column names, and names exceeding its 63-byte limit instead of silently converting unsupported types to text.
    • SQLite rejects unsupported types and reserved table names.
    • SpiralDB rejects reserved column names and timezone-aware timestamps it cannot preserve.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant