Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -712,6 +712,20 @@ side when its inputs' types disagree; a result's type never depends on data valu
order or the path the data took; and inferring a type from Python values is a last resort for
when no type is known.

Large variants everywhere (developer, November 14, 2026, ITL-1122): every column of every table
Orcapod presents or stores has its string, binary and list types in their large variants
(large_string, large_binary, large_list, recursively), for every supported Arrow type, with
no exception: a user's own Arrow table (ArrowTableSource, a raw ArrowTableStream) and an
explicit Arrow schema= / data_schema= are normalized too. The one place is
ArrowTableStream.__init__ (arrow_utils.normalize_table_to_large_types(table, keep_null=True)),
where every source and every operator result becomes a stream; never add a per-source or
per-operator normalization. Names, nullability (top level and nested), field metadata, widths and
logical (extension) types are untouched, and large and non-large variants hash identically, so no
hash or default source id moves. Not covered: what is inside a library value (numpy, pandas,
polars, torch) keeps its library's own distinction. list_view and large_list_view are not
supported: a source refuses them (the Arrow hasher cannot hash them) and a raw ArrowTableStream
holds them as given (ITL-1129).

Respect the Arrow types the inputs already have, normalized to their large variants (developer,
November 10, 2026: large and non-large variants hash identically, so always normalize to large
for consistency). Infer an Arrow type from Python values only when no Arrow type is known, and
Expand Down Expand Up @@ -756,6 +770,9 @@ data. The rules:
operator that round-trips through Polars casts its columns back
to the normalized input types (arrow_utils.restore_column_types, or
restore_schema_nullability when nullability is restored too).
This holds for every operator, also one that passes its rows through unchanged (select,
map, a no-op filter, Join, SemiJoin), on a plain stream as inside a job: its result
is an ArrowTableStream (ITL-1122).
- List columns built by Batch, GroupBy and MergeJoin are typed by
arrow_utils.aggregated_list_type (directly or through build_aggregated_table):
large_list<normalized T>, or for a logical T the logical list type from the type converter
Expand Down Expand Up @@ -851,7 +868,7 @@ data. The rules:
nullability (`T` against `T | None`), because a null key matches nothing (developer,
November 13, 2026, ITL-829). `Join` and `MergeJoin` give the key `T` when any input has it
`T`, and `T | None` only when every input has it nullable; `SemiJoin`'s output is the left
input's schema as it is. The rule is `core/operators/key_nullability.py`
input's schema as it is (its nullability; types in their large variants, ITL-1122). The rule is `core/operators/key_nullability.py`
(`reconcile_key_nullability` on tag schemas before they are combined, `key_nullable` for the
Arrow field flag, set by re-wrapping arrays, never by a cast). Equal nullability takes the
unchanged path; any other type difference, and every data column, behaves as before.
Expand Down
19 changes: 18 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -754,6 +754,20 @@ side when its inputs' types disagree; a result's type never depends on data valu
order or the path the data took; and inferring a type from Python values is a last resort for
when no type is known.

Large variants everywhere (developer, November 14, 2026, ITL-1122): every column of every table
Orcapod presents or stores has its string, binary and list types in their large variants
(`large_string`, `large_binary`, `large_list`, recursively), for every supported Arrow type, with
no exception: a user's own Arrow table (`ArrowTableSource`, a raw `ArrowTableStream`) and an
explicit Arrow `schema=` / `data_schema=` are normalized too. The one place is
`ArrowTableStream.__init__` (`arrow_utils.normalize_table_to_large_types(table, keep_null=True)`),
where every source and every operator result becomes a stream; never add a per-source or
per-operator normalization. Names, nullability (top level and nested), field metadata, widths and
logical (extension) types are untouched, and large and non-large variants hash identically, so no
hash or default source id moves. Not covered: what is inside a library value (numpy, pandas,
polars, torch) keeps its library's own distinction. `list_view` and `large_list_view` are not
supported: a source refuses them (the Arrow hasher cannot hash them) and a raw `ArrowTableStream`
holds them as given (ITL-1129).

Respect the Arrow types the inputs already have, normalized to their large variants (developer,
November 10, 2026: large and non-large variants hash identically, so always normalize to large
for consistency). Infer an Arrow type from Python values only when no Arrow type is known, and
Expand Down Expand Up @@ -798,6 +812,9 @@ data. The rules:
operator that round-trips through Polars casts its columns back
to the normalized input types (`arrow_utils.restore_column_types`, or
`restore_schema_nullability` when nullability is restored too).
This holds for every operator, also one that passes its rows through unchanged (select,
map, a no-op filter, `Join`, `SemiJoin`), on a plain stream as inside a job: its result
is an `ArrowTableStream` (ITL-1122).
- List columns built by `Batch`, `GroupBy` and `MergeJoin` are typed by
`arrow_utils.aggregated_list_type` (directly or through `build_aggregated_table`):
`large_list<normalized T>`, or for a logical `T` the logical list type from the type converter
Expand Down Expand Up @@ -895,7 +912,7 @@ data. The rules:
nullability (`T` against `T | None`), because a null key matches nothing (developer,
November 13, 2026, ITL-829). `Join` and `MergeJoin` give the key `T` when any input has it
`T`, and `T | None` only when every input has it nullable; `SemiJoin`'s output is the left
input's schema as it is. The rule is `core/operators/key_nullability.py`
input's schema as it is (its nullability; types in their large variants, ITL-1122). The rule is `core/operators/key_nullability.py`
(`reconcile_key_nullability` on tag schemas before they are combined, `key_nullable` for the
Arrow field flag, set by re-wrapping arrays, never by a cast). Equal nullability takes the
unchanged path; any other type difference, and every data column, behaves as before.
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/ITL-1122.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
### Breaking changes and migration

- Every table Orcapod presents or stores has the large Arrow variants of string, binary and list types (`large_string`, `large_binary`, `large_list`, recursively), with no exception (ITL-1122). A `string`, `binary` or `list` column of an Arrow table you give `ArrowTableSource` (or a raw `ArrowTableStream`), or of an Arrow schema you pass as `schema=` to `CSVSource` or `DataFrameSource` or as `data_schema=` to `DictSource`, now comes back as `large_string`, `large_binary` or `large_list`. Names, nullability, widths and logical types are kept. Code that compared an Arrow type to `pa.string()` compares it to `pa.large_string()` now. No hash moves: large and non-large variants hash identically, so row hashes, content hashes, pipeline hashes, record ids and default source ids stay as they were, and a Delta table written with either variant accepts rows of the other.

### Changed

- Large types no longer depend on the source kind or the path (ITL-1122). `CSVSource` gave `string` from a file. `CachedSource` gave the types its cache store kept: `string` from an in-memory cache, `large_string` from a Delta cache and with no rows. On a plain stream, the operators that pass rows through unchanged (`SelectTagColumns`, `SelectDataColumns`, `DropTagColumns`, `DropDataColumns`, `SelectColumns`, `DropColumns`, `MapTags`, `MapData`, `PolarsFilter` with no predicate, `Join`, `SemiJoin`, and `GroupBy`'s key columns) kept a `string` column, while the same operator inside a job gave `large_string`. `Join` of a `string` and a `large_string` input on one tag gave the type of whichever input came first in its canonical order. All of them now give large types, with rows and without, sync and async.
24 changes: 24 additions & 0 deletions design_issues/ITL-1122-LN1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## Cross-cutting

### LN1 — Large normalization applied unevenly across sources and operators
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1122

The rule (developer, November 14, 2026, replacing an earlier form with exceptions for a user's
Arrow table and for a store that keeps the distinction): every column of every table Orcapod
presents or stores has its string, binary and list types in their large variants, with no
exception. Measured on `main` 3a7f3850: `ArrowTableSource` and an explicit Arrow schema kept
`string`; `CSVSource` gave `string` (a CSV file keeps no distinction); `CachedSource` gave the
types its cache store kept (`string` from an in-memory cache, `large_string` from Delta and with
no rows); on a plain stream, the operators that pass rows through (select, drop, map, a filter
with nothing to filter, `Join`, `SemiJoin`, `GroupBy`'s key) kept a source's `string` while every
operator inside a job gave `large_string`, and `Join` of a `string` and a `large_string` input on
a shared tag took the type of whichever input came first in canonical order. No shipped source
store (CSV, Delta Lake, SQLite, PostgreSQL, SpiralDB) keeps the distinction.

**Fix:** one place, `ArrowTableStream.__init__`, normalizes the table every source and every
operator result becomes a stream from (`arrow_utils.normalize_table_to_large_types(...,
keep_null=True)`). Swept by `tests/test_core/operators/test_large_normalization.py` and
`tests/test_core/sources/test_large_normalization_sources.py` (every source kind x operator x
plain / job sync / job async x populated / zero-row).
4 changes: 4 additions & 0 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,10 @@ silent widening; convert explicitly.
- **Types are normalized to their large variants.** `string` becomes `large_string`,
`binary` becomes `large_binary` and `list` becomes `large_list`, recursively (also inside
structs). The Arrow hasher treats each pair as identical, so this never changes a hash.
Every operator does this, also one that passes its rows through unchanged (a select, a
rename, a filter with nothing to filter, a join), on a plain stream as inside a job. Every
source gives large types too, so an operator's inputs already have them (see
[Arrow types of a source](sources.md#arrow-types-of-a-source)).
- **Width, nullability and logical types are kept.** An `int8` column stays `int8`, a
`float32` column stays `float32`, and a logical type such as `Path` keeps its logical type.
- **List columns built by `MergeJoin`, `Batch` and `GroupBy`** are `large_list<T>`, where `T`
Expand Down
11 changes: 11 additions & 0 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,17 @@ Data with no rows has no columns to read, so `DictSource(data=[], ...)` raises
`ValueError: Table is empty`. To build a source with no rows, pass `data_schema=` so the
columns are known.

### Arrow types of a source

Every source gives the large variants of string, binary and list types (`large_string`,
`large_binary`, `large_list`, recursively), whatever its input: an Arrow table of your own
(`ArrowTableSource`), an Arrow schema you pass (`schema=` or `data_schema=`), Python values,
a data frame, a CSV file or a database table. A `string` column you give comes back as
`large_string`. Only the layout changes: names, nullability, widths (`int8` stays `int8`) and
logical types are kept, and the two variants hash identically, so no hash depends on which
one you gave. The Arrow `list_view` and `large_list_view` types are not supported: a source
refuses a column of either type.

### SpiralDB tables

`SpiralDBTableSource` exposes a table in a [SpiralDB](https://spiraldb.com) project as a
Expand Down
13 changes: 0 additions & 13 deletions src/orcapod/core/operators/filters.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,19 +46,6 @@ def __init__(
self.constraints = constraints if constraints is not None else {}
super().__init__(**kwargs)

def zero_row_result(self, *inputs: StreamProtocol) -> StreamProtocol:
"""This filter's result over a zero-row input, for a table with no rows (ITL-912).

A filter with no predicate and no constraint passes its input through
unaltered (``unary_static_process``), so its result is the input with the types
it has. Any other filter gives its input's columns in the large-normalized
form a run gives them, without evaluating the predicate.
"""
if len(self.predicates) == 0 and len(self.constraints) == 0:
(only,) = inputs
return only
return super().zero_row_result(*inputs)

def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol:
if len(self.predicates) == 0 and len(self.constraints) == 0:
logger.info(
Expand Down
10 changes: 10 additions & 0 deletions src/orcapod/core/streams/arrow_table_stream.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,13 @@ class ArrowTableStream(StreamBase):
This is useful for ensuring that the data in the stream remains consistent and unchanging.

The types of the tag and data columns are inferred from the PyArrow Table schema.

Every column takes the large variant of its Arrow type (developer, November 14,
2026, ITL-1122): ``string`` is ``large_string``, ``binary`` is ``large_binary``,
``list`` is ``large_list``, recursively. This is the one place a table becomes a
stream, so every source (each builds its stream here, from a user's Arrow table
too) and every operator result presents large types. Width, nullability, field
metadata and logical types are kept; large and non-large variants hash identically.
"""

# The table's rows are held, not computed: a run() view says "all N rows", as
Expand All @@ -60,6 +67,9 @@ def __init__(
self._producer = producer
self._upstreams = upstreams

# Every table Orcapod presents is in its large variants (ITL-1122).
table = arrow_utils.normalize_table_to_large_types(table, keep_null=True)

data_table, data_context_table = arrow_utils.split_by_column_groups(
table, [constants.CONTEXT_KEY]
)
Expand Down
26 changes: 20 additions & 6 deletions src/orcapod/utils/arrow_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,9 @@ def _normalize_field_to_large_types(
return field.with_type(normalize_to_large_types(field.type, keep_null=keep_null))


def normalize_schema_to_large_types(schema: "pa.Schema") -> "pa.Schema":
def normalize_schema_to_large_types(
schema: "pa.Schema", *, keep_null: bool = False
) -> "pa.Schema":
"""
Convert a schema to use large variants of data types.

Expand All @@ -203,6 +205,9 @@ def normalize_schema_to_large_types(schema: "pa.Schema") -> "pa.Schema":

Args:
schema: Arrow schema to normalize
keep_null: Passed to ``normalize_to_large_types``. ``True`` keeps a
``null`` column ``null``, as operators and sources do; the default
``False`` maps it to ``large_string``, as ``Data`` does.

Returns:
New schema with large type variants, or same schema if no changes needed
Expand All @@ -220,7 +225,7 @@ def normalize_schema_to_large_types(schema: "pa.Schema") -> "pa.Schema":
schema_changed = False

for field in schema:
normalized_type = normalize_to_large_types(field.type)
normalized_type = normalize_to_large_types(field.type, keep_null=keep_null)

# Check if the type actually changed
if normalized_type != field.type:
Expand Down Expand Up @@ -275,16 +280,22 @@ def shared_arrow_type(
return None


def normalize_table_to_large_types(table: "pa.Table") -> "pa.Table":
def normalize_table_to_large_types(
table: "pa.Table", *, keep_null: bool = False
) -> "pa.Table":
"""
Normalize table schema to use large type variants.

Uses cast() which should be zero-copy for large variant conversions
since they have identical binary representations, but ensures proper
type validation and handles any edge cases safely.
type validation and handles any edge cases safely. Only layouts change:
width, nullability, field metadata and logical (extension) types are kept.

Args:
table: Arrow table to normalize
keep_null: Passed to ``normalize_to_large_types``. ``True`` keeps a
``null`` column ``null``, as operators and sources do; the default
``False`` maps it to ``large_string``, as ``Data`` does.

Returns:
Table with normalized schema, or same table if no changes needed
Expand All @@ -296,10 +307,13 @@ def normalize_table_to_large_types(table: "pa.Table") -> "pa.Table":
name: large_string
age: int64
"""
normalized_schema = normalize_schema_to_large_types(table.schema)
# ``table.schema`` makes a new Schema object on every access: read it once,
# so the identity check below can find an unchanged schema.
schema = table.schema
normalized_schema = normalize_schema_to_large_types(schema, keep_null=keep_null)

# If schema didn't change, return original table
if normalized_schema is table.schema:
if normalized_schema is schema:
return table

# Use cast() for safety - should be zero-copy for large variant conversions
Expand Down
Loading
Loading