Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions changelog.d/ITL-928.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
### Breaking changes and migration

#### The Delta store keeps every column's Arrow type; rebuild a Delta store written by an earlier v0.3 build (ITL-928)

Delta Lake has no unsigned integer, no `timestamp` in seconds, milliseconds or nanoseconds, no zone other than UTC, and no `date64`, `time` or `duration` type. `DeltaTableDatabase` used to let the Delta writer convert such a column, so a read gave back another type: `uint32` as `int32` (and a value above the signed maximum failed the write), `timestamp[s]`, `[ms]` and `[ns]` as `timestamp[us]` (losing nanosecond digits), `timestamp[us, tz=America/New_York]` as UTC; `date64`, `time` and `duration` columns could not be stored at all. A tag read back with another type changes the record ids computed from it, so on a Delta store a re-run over such a tag wrote the downstream nodes' pipeline rows a second time (a fresh job then read 6 rows for 3).

Now such a column's original Arrow type is recorded in the Delta field's metadata, an unsigned integer is written as the signed integer of the same width bit for bit, and `timestamp[ns]`, `date64`, `time` and `duration` are written as their integer value; every read casts the column back to its recorded type. Nested columns (`list<uint32>`, a struct with a `timestamp[ms]` field) are covered. Columns of every other type, and logical-type columns, are stored as before.

A Delta store written by an earlier unreleased v0.3 build has no recorded types: its tables keep reading back Delta's types, a re-run over such a tag still writes duplicate pipeline rows, rows it already duplicated stay, a function node whose rows mix rows read from it with newly computed rows raises `InputValidationError` naming the column and both types, and appending a `timestamp[ns]` column to one of its tables is refused (the stored `timestamp[us]` differs from the new integer form). Rows with an unsigned column are still appended to such a table's signed column, as before, and read back signed; a value the signed type cannot hold is refused with `InputValidationError` and nothing is written. Nothing is repaired automatically: rebuild such a store by running the pipeline into a new store. Stores holding none of these column types are not affected. No content hash, pipeline hash, record id or table path changes.

### Fixed

- A function node's `as_table()` keeps the Arrow type of every tag column. Before, it rebuilt the table from Python values, so after `job.run()` an `int8` or `uint32` tag came back as `int64`, `float32` as `double`, a naive `timestamp[us]` as `timestamp[us, tz=UTC]` and `date64` as `date32`, and a `time`, `duration`, `decimal`, `uuid` or `Path` tag made it raise. Joining a function node's output with another source on such a tag no longer fails with "Join inputs disagree on the Arrow type". `as_table(all_info=True)` of a function node now also has the `_source_<output>` and `_context_key` columns, as the same pod's stream outside a job does. When two of a node's rows disagree on a column's Arrow type, `as_table()` raises `InputValidationError` naming the node, the column and both types, instead of returning a type neither has.
- The table of a function pod's stream (the pod applied outside a job) no longer raises for a `uuid` or `Path` tag ("Expected bytes, got a 'UUID' object").
- On a Delta store, a re-run over a tag of an unsigned, non-microsecond or zoned timestamp type writes no pipeline row again, and a fresh job, a loaded job, an operator node's replay and a `CachedSource` read the tag's own type (see above).
27 changes: 27 additions & 0 deletions design_issues/ITL-928-DT1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
## `src/orcapod/databases/delta_lake_databases.py`

### DT1 — the Delta store gave back other Arrow types than it was given
**Status:** resolved
**Severity:** critical
**Issue:** ITL-928

The Delta writer converts what Delta Lake has no type for: an unsigned integer to the signed
integer of the same width (a value above the signed maximum failed the flush), `timestamp[s]`,
`[ms]` and `[ns]` to `timestamp[us]` (nanosecond digits silently dropped), a zone other than UTC
to UTC; it cannot store `date64`, `time32/64` or `duration` at all. Every read of a Delta store
(function node hot-load and `get_all_records`, a fresh or loaded job, operator `REPLAY`,
`CachedSource`) gave back the converted type. A tag read back that way changes the record ids
computed from it (the base entry id hashes the tag's Arrow table), so a re-run of
`source(uint32 tag) -> pod -> pod` found its results by input hash, computed nothing, and wrote
the downstream node's pipeline rows again: 6 rows for 3.

**Fix:** `databases/delta_arrow_types.py`. Before a flush, a column with such a type at any depth
gets its whole Arrow type recorded in its top-level field metadata (`orcapod:arrow_type`, an
IPC-serialized schema; Delta keeps top-level field metadata, not nested), an unsigned integer is
written as the same-width signed integer bit for bit, and `timestamp[ns]`, `date64`, `time` and
`duration` as their integer value; s/ms timestamps and zones go through Delta's own lossless
conversion. Every read path (`get_all_records`, the snapshot scan, the snapshot extension after a
commit) casts recorded columns back. Logical-type columns are left to their extension metadata.
Tables written before the fix have no recorded type and read as before; such a store must be
rebuilt (developer, November 12, 2026). Not covered: Delta also gives back `fixed_size_binary`
as `binary` and a dictionary column as its value type; those are unchanged.
27 changes: 27 additions & 0 deletions design_issues/ITL-928-FN1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
## `src/orcapod/core/nodes/function_node.py`

### FN1 — a function node's table rebuilt its tags from Python values with inferred types
**Status:** resolved
**Severity:** critical
**Issue:** ITL-928

`FunctionNodeBase.as_table` built its table from `tag.as_dict()` and `data.as_dict()` Python
values, typed by the node's Python output schema through the type converter. A function pod
never touches tags, yet the tag columns came back with types inferred from Python: every
integer width as `int64`, `float32` as `double`, every naive or zoned timestamp as
`timestamp[us, tz=UTC]`, `date64` as `date32`; a `time`, `duration`, `decimal`, `uuid` or
`Path` tag made it raise. It happened on every store, sync and async, while `iter_data()`,
`get_all_records()` and the stored pipeline rows kept the input types. Joining a node's output
with another source on such a tag raised "Join inputs disagree on the Arrow type". The same
build also dropped the `_source_<output>` and `_context_key` columns that the pod's stream
outside a job has. `FunctionPodStream.as_table`, the same pod outside a job, built its tag
table from Python values too (typed with the tag's own Arrow schema), so only a `uuid` or
`Path` tag broke it ("Expected bytes, got a 'UUID' object").

**Fix:** both build their tag table from each row's own Arrow table (`tag.as_table()`), and the
node its data table from `data.as_table()` (source info and context, no meta columns),
concatenated by name; rows that disagree on a column's Arrow type raise
`InputValidationError` naming the column and both types, never a widened type. No stored value, record id or hash changes. The zero-row table of a
node that has not run is still built from the Python output schema (the ITL-912 problem, for
function nodes).
The Delta store's own type changes are entry DT1 (`design_issues/ITL-928-DT1.md`).
7 changes: 5 additions & 2 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,8 +385,11 @@ print("function calls:", len(calls))
`InputValidationError`, which names the table, the column and both types, and nothing is
written to that table. They are not converted to the stored type. `string` and
`large_string`, and the other large and non-large layouts of one type, count as the same
type, and so do types the Delta writer converts on the way in (`uint64` is stored as
`int64`, every timestamp in microseconds). New columns and new struct fields pass this
type. A column of a type Delta Lake has no equivalent for (an unsigned integer, a
timestamp in seconds, milliseconds or nanoseconds or with a zone other than UTC, `date64`,
`time`, `duration`, at any depth) is written in a form Delta has, with its Arrow type
recorded in the table, and every read gives it back with that type and its exact values.
New columns and new struct fields pass this
check; whether they are added follows
the database's `allow_schema_evolution` setting. In a job run, a refused append fails the
run as a `StoreWriteError` whose cause is the `InputValidationError`.
Expand Down
77 changes: 39 additions & 38 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -1096,51 +1096,52 @@ async def _gather() -> list[tuple[TagProtocol, DataProtocol | None]]:
if data is not None:
yield tag, data

def _table_from_pairs(
self, pairs: list[tuple[TagProtocol, DataProtocol]]
) -> pa.Table:
"""Return the output rows ``pairs`` as one table with every column group."""
tag_tables: list[pa.Table] = []
all_data = []
data_schema = None
for tag, data in pairs:
if data_schema is None:
data_schema = data.arrow_schema(all_info=True)
# Tags are passed through untouched: each row's own Arrow table keeps
# the input's tag types, a logical type included (ITL-928).
tag_tables.append(tag.as_table(columns={"system_tags": True}))
all_data.append(data.as_dict(all_info=True))

# TODO: re-verify the implemetation of this conversion
converter = self.data_context.type_converter

# The data columns have declared types: convert each value as its column's
# type, never as the type the value happens to have. A subclass instance
# returned for a base-class column is then refused instead of being stored
# as the base without its own fields (ITL-972). Only the remaining columns
# (meta, source info) are typed from their values.
python_schema = dict(infer_python_schema_from_pylist_data(all_data))
python_schema.update(self.output_schema()[1])
struct_data = converter.python_dicts_to_struct_dicts(
all_data, python_schema=python_schema
)
if tag_tables:
all_tags_as_tables = arrow_utils.concat_tables_by_name(tag_tables)
else:
# An empty stream has no row to take a tag table from.
all_tags_as_tables = pa.Table.from_pylist([])
all_data_as_tables: pa.Table = pa.Table.from_pylist(
struct_data, schema=data_schema
)
return arrow_utils.hstack_tables(all_tags_as_tables, all_data_as_tables)

def as_table(
self,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> pa.Table:
if self._cached_output_table is None:
all_tags = []
all_data = []
tag_schema, data_schema = None, None
for tag, data in self.iter_data():
if tag_schema is None:
tag_schema = tag.arrow_schema(all_info=True)
if data_schema is None:
data_schema = data.arrow_schema(all_info=True)
# TODO: make use of arrow_compat dict
all_tags.append(tag.as_dict(all_info=True))
all_data.append(data.as_dict(all_info=True))

# TODO: re-verify the implemetation of this conversion
converter = self.data_context.type_converter

# The data columns have declared types: convert each value as its column's
# type, never as the type the value happens to have. A subclass instance
# returned for a base-class column is then refused instead of being stored
# as the base without its own fields (ITL-972). Only the remaining columns
# (meta, source info) are typed from their values.
python_schema = dict(infer_python_schema_from_pylist_data(all_data))
python_schema.update(self.output_schema()[1])
struct_data = converter.python_dicts_to_struct_dicts(
all_data, python_schema=python_schema
)
all_tags_as_tables: pa.Table = pa.Table.from_pylist(
all_tags, schema=tag_schema
)
# drop context key column from tags table (guard: column absent on empty stream)
if constants.CONTEXT_KEY in all_tags_as_tables.column_names:
all_tags_as_tables = all_tags_as_tables.drop([constants.CONTEXT_KEY])
all_data_as_tables: pa.Table = pa.Table.from_pylist(
struct_data, schema=data_schema
)

self._cached_output_table = arrow_utils.hstack_tables(
all_tags_as_tables, all_data_as_tables
)
self._cached_output_table = self._table_from_pairs(list(self.iter_data()))
assert self._cached_output_table is not None, (
"_cached_output_table should not be None here."
)
Expand Down
141 changes: 95 additions & 46 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,58 @@ def _build_record_id_preimage(
)


def _large_layout(table: "pa.Table") -> "pa.Table":
"""Return ``table`` with every column in its large layout, ``null`` kept ``null``.

Args:
table: The table to normalize.

Returns:
The table with large-normalized column types; names, nullability and
metadata kept.
"""
schema = pa.schema(
[
field.with_type(
arrow_utils.normalize_to_large_types(field.type, keep_null=True)
)
for field in table.schema
]
)
return table if schema.equals(table.schema) else table.cast(schema)


def _concat_row_tables(tables: "list[pa.Table]") -> "pa.Table":
"""Concatenate one-row tables by column name, refusing a column whose types differ.

Rows never make a column of a type one of them does not have: two rows whose
Arrow types for one column differ (other than by their large variants) are an
error, never widened or cast.

Args:
tables: The rows' tables, in row order; not empty.

Returns:
The concatenated table, with the first table's column order.

Raises:
InputValidationError: If two rows give one column different Arrow types,
naming the column and both types.
"""
# Large and non-large layouts of one type are one type: give every row the
# large layout, so they also concatenate. Layouts only: ``null`` stays ``null``.
tables = [_large_layout(t) for t in tables]
# The first type seen for each column, whichever row first has the column.
seen: dict[str, pa.DataType] = {}
for table in tables:
for name, row_type in zip(table.schema.names, table.schema.types):
expected = seen.setdefault(name, row_type)
if arrow_utils.shared_arrow_type(expected, row_type) is None:
message = f"Rows disagree on the Arrow type of column {name!r}: {expected} and {row_type}"
raise InputValidationError(message)
return arrow_utils.concat_tables_by_name(tables)


def _bare_digest(hash_string: str) -> str:
"""Return the bare hex digest of a stored ``"<label>:<hex>"`` hash string.

Expand Down Expand Up @@ -455,52 +507,7 @@ def as_table(
all_info: bool = False,
) -> "pa.Table":
if self._cached_output_table is None:
all_tags: list[dict] = []
all_data: list[dict] = []
# Derive Arrow schemas from the declared output_schema upfront so that
# the cached table always has a consistent, well-formed schema regardless
# of whether any rows were produced.
tag_python_schema, data_python_schema = self.output_schema(all_info=True)
for tag, data in self.iter_data():
all_tags.append(tag.as_dict(all_info=True))
all_data.append(data.as_dict(all_info=True))

converter = self.data_context.type_converter
tag_arrow_schema = converter.python_schema_to_arrow_schema(
tag_python_schema
)
data_arrow_schema = converter.python_schema_to_arrow_schema(
data_python_schema
)

if not all_tags:
# No data was produced (node has not been run yet or produced zero
# outputs). Build a zero-row table whose schema is derived from the
# pod's declared output so that callers always receive a well-formed
# schema regardless of whether the node has actually executed.
self._cached_output_table = pa.Table.from_pylist(
[],
schema=converter.python_schema_to_arrow_schema(
tag_python_schema + data_python_schema
),
)
else:
struct_data = converter.python_dicts_to_struct_dicts(
all_data, python_schema=data_python_schema
)
all_tags_as_tables: pa.Table = pa.Table.from_pylist(
all_tags, schema=tag_arrow_schema
)
if constants.CONTEXT_KEY in all_tags_as_tables.column_names:
all_tags_as_tables = all_tags_as_tables.drop(
[constants.CONTEXT_KEY]
)
all_data_as_tables: pa.Table = pa.Table.from_pylist(
struct_data, schema=data_arrow_schema
)
self._cached_output_table = arrow_utils.hstack_tables(
all_tags_as_tables, all_data_as_tables
)
self._cached_output_table = self._table_from_pairs(list(self.iter_data()))

column_config = ColumnConfig.handle_config(columns, all_info=all_info)
output_table = arrow_utils.apply_column_config(
Expand All @@ -527,6 +534,48 @@ def as_table(

return output_table

def _table_from_pairs(
self, pairs: "list[tuple[TagProtocol, DataProtocol]]"
) -> "pa.Table":
"""Return the output rows ``pairs`` as one table with every column group.

Built from each row's own Arrow tables, so every column keeps the type its
row has: a tag the type of the input's tag, never a type inferred from
Python values (ITL-928). A row's meta columns (a computed row's
``__computed`` flag) are its own bookkeeping, not part of the node's
output, as for the pod's stream outside a job. With no rows the table has
the declared columns, typed from the declared output schema.

Raises:
InputValidationError: If two rows give one column different Arrow
types, naming this node, the column and both types.
"""
if not pairs:
# No data was produced (node has not been run yet or produced zero
# outputs). Build a zero-row table whose schema is derived from the
# pod's declared output so that callers always receive a well-formed
# schema regardless of whether the node has actually executed.
tag_python_schema, data_python_schema = self.output_schema(all_info=True)
return pa.Table.from_pylist(
[],
schema=self.data_context.type_converter.python_schema_to_arrow_schema(
tag_python_schema + data_python_schema
),
)
try:
tag_table = _concat_row_tables(
[tag.as_table(columns={"system_tags": True}) for tag, _ in pairs]
)
data_table = _concat_row_tables(
[
data.as_table(columns={"source": True, "context": True})
for _, data in pairs
]
)
except InputValidationError as exc:
raise InputValidationError(f"Function node {self.label!r}: {exc}") from exc
return arrow_utils.hstack_tables(tag_table, data_table)

def __repr__(self) -> str:
return (
f"{type(self).__name__}(data_function={self._data_function!r}, "
Expand Down
Loading
Loading