Repository navigation
fix(databases): refuse a table a connector database cannot store (ITL-1162, ITL-1164) - #408
Merged
eywalker merged 16 commits intoNov 17, 2026
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes ITL-1162
Related: ITL-1164 (SpiralDB reserved names: refused here; the real fix, renaming on write, is not in this PR)
Related: ITL-1166 (PostgreSQL column and table names over 63 bytes: refused here; the real fix, name mapping, is not in this PR)
What
A
ConnectorArrowDatabasenow refuses a table its connector cannot store before anything is written, and a job refuses a node whose stored table holds such a column before anything runs. Developer rulings, 2026-11-16: refuse now (SQL types, and SpiralDB reserved names); storing these columns is for a later release.One rule, stated by the connector:
DBConnectorProtocol.unstorable_column_reason(name, arrow_type, *, primary_key)returns a reason, orNonewhen the column can be stored.bool,int64,double,large_string,large_binary.int32,float,date32,timestamp[us],timestamp[us, tz=UTC]. PostgreSQL also refuses a column name longer than 63 bytes, which it truncates (_PG_MAX_IDENTIFIER_BYTES; coordinator ruling on a CodeRabbit finding). SQLite has no identifier length limit (tested with a 216-byte name).__prefix (the__record_idkey is allowed), any column name containing., and a time-zone-aware timestamp column (SpiralDB drops the zone; coordinator decision on a CodeRabbit finding).ConnectorArrowDatabase.require_storable(schema)raisesUnstorableColumnError, naming the connector, the column, its Arrow type, the reason, and what to use instead (InMemoryArrowDatabase/DeltaTableDatabase). It runs inadd_recordsbefore anything is buffered.ExtensionAwareDatabaseforwards it.Up front (types and names from declared schemas; nothing is computed):
The node label is added by the caller (
add_context). Each node kind states its check once (require_storable_tables), andPipelineJobchecks every node before attaching any, inbindand at compile. A refused database leaves the job as it was.Table names: each connector states which table names it holds (
unstorable_table_name_reason). PostgreSQL refuses a name over 63 bytes, which it would truncate, so two tables sharing their first 63 bytes would be one.ConnectorArrowDatabaserefuses such a name withUnstorableTableNameErrorat every entry that creates, writes, reads or checks a table, andPostgreSQLConnectorguards its own table entries, so a truncated name is never sent (coordinator ruling). Node tables and cached sources hold a 64-character hash in their path, so on PostgreSQL only short, directly written paths fit. No in-suite PostgreSQL scenario that worked before is affected.No silent TEXT fallback: the SQLite and PostgreSQL
CREATE TABLEtype maps raise for a type they have no column type for.Records are normalized to their large variants (view types included) before the check and buffering. The flush schema check compares large-normalized types (
keep_null=True), so astringcolumn no longer fails its second write, and buffered and committed rows read together.tests/test_databases/fake_pyspiral.pyrefuses the reserved names on write, with SpiralDB's messages. 0 of 495 fake-based tests relied on the fake accepting them.Which nodes are refused
rdb_v2) + pipeline (pdb_v2)__pf_exec_executor_infolist)__input_data_hash)tdb_v2)executed_attimestamp)executed_attime zone,__data_context)odb_v1), caching on__node_content_hash)sdb_v1)__data_context)Timestamps
timestamp[us, tz=UTC]does not round-trip on SQLite (stored as TEXT, read back aslarge_string). Neither doesdate32, andint8/int32/float32read back widened, so the next write failed the schema check. All of these now fall under the same refusal. On PostgreSQL,timestamp[us]andtimestamp[us, tz=UTC]are accepted by code reading; other units and time zones are refused.Not in scope
design_issues/ITL-1162-CA3.md.__/.column names on write and recover them on read, and restore a timestamp's time zone.Verification
tests/test_databases/test_unstorable_columns.py(public interface) covers:psycopg.connect) and SpiralDB (the fake);ExtensionAwareDatabasepath;test_postgresql_connector_integration.pyrun in CI'sRun PostgreSQL Testsworkflow, not locally: every type the connector accepts round-trips twice, a function node job is refused, and a long name / operator cache is refused with no table created.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes