diff --git a/changelog.d/ITL-928.md b/changelog.d/ITL-928.md new file mode 100644 index 000000000..d14762b01 --- /dev/null +++ b/changelog.d/ITL-928.md @@ -0,0 +1,15 @@ +### Breaking changes and migration + +#### The Delta store keeps every column's Arrow type; rebuild a Delta store written by an earlier v0.3 build (ITL-928) + +Delta Lake has no unsigned integer, no `timestamp` in seconds, milliseconds or nanoseconds, no zone other than UTC, and no `date64`, `time` or `duration` type. `DeltaTableDatabase` used to let the Delta writer convert such a column, so a read gave back another type: `uint32` as `int32` (and a value above the signed maximum failed the write), `timestamp[s]`, `[ms]` and `[ns]` as `timestamp[us]` (losing nanosecond digits), `timestamp[us, tz=America/New_York]` as UTC; `date64`, `time` and `duration` columns could not be stored at all. A tag read back with another type changes the record ids computed from it, so on a Delta store a re-run over such a tag wrote the downstream nodes' pipeline rows a second time (a fresh job then read 6 rows for 3). + +Now such a column's original Arrow type is recorded in the Delta field's metadata, an unsigned integer is written as the signed integer of the same width bit for bit, and `timestamp[ns]`, `date64`, `time` and `duration` are written as their integer value; every read casts the column back to its recorded type. Nested columns (`list`, a struct with a `timestamp[ms]` field) are covered. Columns of every other type, and logical-type columns, are stored as before. + +A Delta store written by an earlier unreleased v0.3 build has no recorded types: its tables keep reading back Delta's types, a re-run over such a tag still writes duplicate pipeline rows, rows it already duplicated stay, a function node whose rows mix rows read from it with newly computed rows raises `InputValidationError` naming the column and both types, and appending a `timestamp[ns]` column to one of its tables is refused (the stored `timestamp[us]` differs from the new integer form). Rows with an unsigned column are still appended to such a table's signed column, as before, and read back signed; a value the signed type cannot hold is refused with `InputValidationError` and nothing is written. Nothing is repaired automatically: rebuild such a store by running the pipeline into a new store. Stores holding none of these column types are not affected. No content hash, pipeline hash, record id or table path changes. + +### Fixed + +- A function node's `as_table()` keeps the Arrow type of every tag column. Before, it rebuilt the table from Python values, so after `job.run()` an `int8` or `uint32` tag came back as `int64`, `float32` as `double`, a naive `timestamp[us]` as `timestamp[us, tz=UTC]` and `date64` as `date32`, and a `time`, `duration`, `decimal`, `uuid` or `Path` tag made it raise. Joining a function node's output with another source on such a tag no longer fails with "Join inputs disagree on the Arrow type". `as_table(all_info=True)` of a function node now also has the `_source_` and `_context_key` columns, as the same pod's stream outside a job does. When two of a node's rows disagree on a column's Arrow type, `as_table()` raises `InputValidationError` naming the node, the column and both types, instead of returning a type neither has. +- The table of a function pod's stream (the pod applied outside a job) no longer raises for a `uuid` or `Path` tag ("Expected bytes, got a 'UUID' object"). +- On a Delta store, a re-run over a tag of an unsigned, non-microsecond or zoned timestamp type writes no pipeline row again, and a fresh job, a loaded job, an operator node's replay and a `CachedSource` read the tag's own type (see above). diff --git a/design_issues/ITL-928-DT1.md b/design_issues/ITL-928-DT1.md new file mode 100644 index 000000000..e1a1424cd --- /dev/null +++ b/design_issues/ITL-928-DT1.md @@ -0,0 +1,27 @@ +## `src/orcapod/databases/delta_lake_databases.py` + +### DT1 — the Delta store gave back other Arrow types than it was given +**Status:** resolved +**Severity:** critical +**Issue:** ITL-928 + +The Delta writer converts what Delta Lake has no type for: an unsigned integer to the signed +integer of the same width (a value above the signed maximum failed the flush), `timestamp[s]`, +`[ms]` and `[ns]` to `timestamp[us]` (nanosecond digits silently dropped), a zone other than UTC +to UTC; it cannot store `date64`, `time32/64` or `duration` at all. Every read of a Delta store +(function node hot-load and `get_all_records`, a fresh or loaded job, operator `REPLAY`, +`CachedSource`) gave back the converted type. A tag read back that way changes the record ids +computed from it (the base entry id hashes the tag's Arrow table), so a re-run of +`source(uint32 tag) -> pod -> pod` found its results by input hash, computed nothing, and wrote +the downstream node's pipeline rows again: 6 rows for 3. + +**Fix:** `databases/delta_arrow_types.py`. Before a flush, a column with such a type at any depth +gets its whole Arrow type recorded in its top-level field metadata (`orcapod:arrow_type`, an +IPC-serialized schema; Delta keeps top-level field metadata, not nested), an unsigned integer is +written as the same-width signed integer bit for bit, and `timestamp[ns]`, `date64`, `time` and +`duration` as their integer value; s/ms timestamps and zones go through Delta's own lossless +conversion. Every read path (`get_all_records`, the snapshot scan, the snapshot extension after a +commit) casts recorded columns back. Logical-type columns are left to their extension metadata. +Tables written before the fix have no recorded type and read as before; such a store must be +rebuilt (developer, November 12, 2026). Not covered: Delta also gives back `fixed_size_binary` +as `binary` and a dictionary column as its value type; those are unchanged. diff --git a/design_issues/ITL-928-FN1.md b/design_issues/ITL-928-FN1.md new file mode 100644 index 000000000..9da2a27b9 --- /dev/null +++ b/design_issues/ITL-928-FN1.md @@ -0,0 +1,27 @@ +## `src/orcapod/core/nodes/function_node.py` + +### FN1 — a function node's table rebuilt its tags from Python values with inferred types +**Status:** resolved +**Severity:** critical +**Issue:** ITL-928 + +`FunctionNodeBase.as_table` built its table from `tag.as_dict()` and `data.as_dict()` Python +values, typed by the node's Python output schema through the type converter. A function pod +never touches tags, yet the tag columns came back with types inferred from Python: every +integer width as `int64`, `float32` as `double`, every naive or zoned timestamp as +`timestamp[us, tz=UTC]`, `date64` as `date32`; a `time`, `duration`, `decimal`, `uuid` or +`Path` tag made it raise. It happened on every store, sync and async, while `iter_data()`, +`get_all_records()` and the stored pipeline rows kept the input types. Joining a node's output +with another source on such a tag raised "Join inputs disagree on the Arrow type". The same +build also dropped the `_source_` and `_context_key` columns that the pod's stream +outside a job has. `FunctionPodStream.as_table`, the same pod outside a job, built its tag +table from Python values too (typed with the tag's own Arrow schema), so only a `uuid` or +`Path` tag broke it ("Expected bytes, got a 'UUID' object"). + +**Fix:** both build their tag table from each row's own Arrow table (`tag.as_table()`), and the +node its data table from `data.as_table()` (source info and context, no meta columns), +concatenated by name; rows that disagree on a column's Arrow type raise +`InputValidationError` naming the column and both types, never a widened type. No stored value, record id or hash changes. The zero-row table of a +node that has not run is still built from the Python output schema (the ITL-912 problem, for +function nodes). +The Delta store's own type changes are entry DT1 (`design_issues/ITL-928-DT1.md`). diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index d9774df4e..7e613ff79 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -385,8 +385,11 @@ print("function calls:", len(calls)) `InputValidationError`, which names the table, the column and both types, and nothing is written to that table. They are not converted to the stored type. `string` and `large_string`, and the other large and non-large layouts of one type, count as the same - type, and so do types the Delta writer converts on the way in (`uint64` is stored as - `int64`, every timestamp in microseconds). New columns and new struct fields pass this + type. A column of a type Delta Lake has no equivalent for (an unsigned integer, a + timestamp in seconds, milliseconds or nanoseconds or with a zone other than UTC, `date64`, + `time`, `duration`, at any depth) is written in a form Delta has, with its Arrow type + recorded in the table, and every read gives it back with that type and its exact values. + New columns and new struct fields pass this check; whether they are added follows the database's `allow_schema_evolution` setting. In a job run, a refused append fails the run as a `StoreWriteError` whose cause is the `InputValidationError`. diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index cdbac803e..4337be2b1 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1096,6 +1096,44 @@ async def _gather() -> list[tuple[TagProtocol, DataProtocol | None]]: if data is not None: yield tag, data + def _table_from_pairs( + self, pairs: list[tuple[TagProtocol, DataProtocol]] + ) -> pa.Table: + """Return the output rows ``pairs`` as one table with every column group.""" + tag_tables: list[pa.Table] = [] + all_data = [] + data_schema = None + for tag, data in pairs: + if data_schema is None: + data_schema = data.arrow_schema(all_info=True) + # Tags are passed through untouched: each row's own Arrow table keeps + # the input's tag types, a logical type included (ITL-928). + tag_tables.append(tag.as_table(columns={"system_tags": True})) + all_data.append(data.as_dict(all_info=True)) + + # TODO: re-verify the implemetation of this conversion + converter = self.data_context.type_converter + + # The data columns have declared types: convert each value as its column's + # type, never as the type the value happens to have. A subclass instance + # returned for a base-class column is then refused instead of being stored + # as the base without its own fields (ITL-972). Only the remaining columns + # (meta, source info) are typed from their values. + python_schema = dict(infer_python_schema_from_pylist_data(all_data)) + python_schema.update(self.output_schema()[1]) + struct_data = converter.python_dicts_to_struct_dicts( + all_data, python_schema=python_schema + ) + if tag_tables: + all_tags_as_tables = arrow_utils.concat_tables_by_name(tag_tables) + else: + # An empty stream has no row to take a tag table from. + all_tags_as_tables = pa.Table.from_pylist([]) + all_data_as_tables: pa.Table = pa.Table.from_pylist( + struct_data, schema=data_schema + ) + return arrow_utils.hstack_tables(all_tags_as_tables, all_data_as_tables) + def as_table( self, *, @@ -1103,44 +1141,7 @@ def as_table( all_info: bool = False, ) -> pa.Table: if self._cached_output_table is None: - all_tags = [] - all_data = [] - tag_schema, data_schema = None, None - for tag, data in self.iter_data(): - if tag_schema is None: - tag_schema = tag.arrow_schema(all_info=True) - if data_schema is None: - data_schema = data.arrow_schema(all_info=True) - # TODO: make use of arrow_compat dict - all_tags.append(tag.as_dict(all_info=True)) - all_data.append(data.as_dict(all_info=True)) - - # TODO: re-verify the implemetation of this conversion - converter = self.data_context.type_converter - - # The data columns have declared types: convert each value as its column's - # type, never as the type the value happens to have. A subclass instance - # returned for a base-class column is then refused instead of being stored - # as the base without its own fields (ITL-972). Only the remaining columns - # (meta, source info) are typed from their values. - python_schema = dict(infer_python_schema_from_pylist_data(all_data)) - python_schema.update(self.output_schema()[1]) - struct_data = converter.python_dicts_to_struct_dicts( - all_data, python_schema=python_schema - ) - all_tags_as_tables: pa.Table = pa.Table.from_pylist( - all_tags, schema=tag_schema - ) - # drop context key column from tags table (guard: column absent on empty stream) - if constants.CONTEXT_KEY in all_tags_as_tables.column_names: - all_tags_as_tables = all_tags_as_tables.drop([constants.CONTEXT_KEY]) - all_data_as_tables: pa.Table = pa.Table.from_pylist( - struct_data, schema=data_schema - ) - - self._cached_output_table = arrow_utils.hstack_tables( - all_tags_as_tables, all_data_as_tables - ) + self._cached_output_table = self._table_from_pairs(list(self.iter_data())) assert self._cached_output_table is not None, ( "_cached_output_table should not be None here." ) diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 888d0b0df..e3560d2d9 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -132,6 +132,58 @@ def _build_record_id_preimage( ) +def _large_layout(table: "pa.Table") -> "pa.Table": + """Return ``table`` with every column in its large layout, ``null`` kept ``null``. + + Args: + table: The table to normalize. + + Returns: + The table with large-normalized column types; names, nullability and + metadata kept. + """ + schema = pa.schema( + [ + field.with_type( + arrow_utils.normalize_to_large_types(field.type, keep_null=True) + ) + for field in table.schema + ] + ) + return table if schema.equals(table.schema) else table.cast(schema) + + +def _concat_row_tables(tables: "list[pa.Table]") -> "pa.Table": + """Concatenate one-row tables by column name, refusing a column whose types differ. + + Rows never make a column of a type one of them does not have: two rows whose + Arrow types for one column differ (other than by their large variants) are an + error, never widened or cast. + + Args: + tables: The rows' tables, in row order; not empty. + + Returns: + The concatenated table, with the first table's column order. + + Raises: + InputValidationError: If two rows give one column different Arrow types, + naming the column and both types. + """ + # Large and non-large layouts of one type are one type: give every row the + # large layout, so they also concatenate. Layouts only: ``null`` stays ``null``. + tables = [_large_layout(t) for t in tables] + # The first type seen for each column, whichever row first has the column. + seen: dict[str, pa.DataType] = {} + for table in tables: + for name, row_type in zip(table.schema.names, table.schema.types): + expected = seen.setdefault(name, row_type) + if arrow_utils.shared_arrow_type(expected, row_type) is None: + message = f"Rows disagree on the Arrow type of column {name!r}: {expected} and {row_type}" + raise InputValidationError(message) + return arrow_utils.concat_tables_by_name(tables) + + def _bare_digest(hash_string: str) -> str: """Return the bare hex digest of a stored ``"