Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -296,8 +296,10 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
both as `set[int]`, as for lists), so it is
registered by identifier only, never bound to a Python type (`binds_python_type`), and
`set[T]` maps to Arrow through `T`. `list[T]` of a primitive stays a plain
`large_list`: never wrap it. Inside a dict value or tuple field a set of a primitive keeps
its plain storage (Arrow has no extension type in a struct field). A container's elements
`large_list`: never wrap it. A native container, like any logical type, inside a dict key
or value or a tuple field is refused at definition (Arrow has no extension type in a struct
field; developer, November 15, 2026, ITL-1150): one guard, `nested_storage.require_storable_in_struct_field`,
never a silent strip to storage. A container's elements
are value-hashed only when its innermost element is a logical type. A saved schema rebuilds
an unregistered native container from its identifier (`serialization.parse_arrow_type_string`).
`frozenset[T]` is refused (ITL-1140).
Expand Down
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,8 +307,10 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
both as `set[int]`, as for lists), so it is
registered by identifier only, never bound to a Python type (`binds_python_type`), and
`set[T]` maps to Arrow through `T`. `list[T]` of a primitive stays a plain
`large_list`: never wrap it. Inside a dict value or tuple field a set of a primitive keeps
its plain storage (Arrow has no extension type in a struct field). A container's elements
`large_list`: never wrap it. A native container, like any logical type, inside a dict key
or value or a tuple field is refused at definition (Arrow has no extension type in a struct
field; developer, November 15, 2026, ITL-1150): one guard, `nested_storage.require_storable_in_struct_field`,
never a silent strip to storage. A container's elements
are value-hashed only when its innermost element is a logical type. A saved schema rebuilds
an unregistered native container from its identifier (`serialization.parse_arrow_type_string`).
`frozenset[T]` is refused (ITL-1140).
Expand Down
30 changes: 30 additions & 0 deletions changelog.d/ITL-1150.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
### Breaking changes and migration

#### A logical type or native container as a dict key or value is refused where it is declared (ITL-1150)

A dictionary is stored as a list of key/value structs, and Arrow cannot hold an extension type
inside a struct field. A set of a primitive there was stored silently as a plain list:
`dict[str, set[int]]` read back from `as_source()`, in a downstream pod and in a job loaded in
another process as `dict[str, list[int]]`, and a source declared with it took the schema
`dict[str, list[int]]`. Every other logical type in that position (`Path`, `UUID`, a user
class, `numpy.ndarray`, `list[Path]`, `set[Path]`) was already refused.

Now any logical type or native container as a dict key or value, at any depth, is refused
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
when it is declared: a function pod's parameter or output, a source's declared or inferred
schema, and a field of a dataclass, an `orcapod.dataclass` or a pydantic model. Refused, for
example: `dict[str, set[int]]`, `dict[str, set[str]]`, `dict[str, list[set[int]]]`,
`dict[set[int], int]`, `list[dict[str, set[int]]]`, a dataclass field typed
`dict[str, set[int]]`.

A tuple annotation was never accepted by a function pod or a source (registration rejects it
as an unsupported annotation, and still does). The only path that took a set inside a tuple
was a direct call of the type converter (`python_schema_to_arrow_schema({"x": tuple[int, set[int]]})`),
which stored the set as a plain list and now refuses it too.

The error, `NestedTypeNotStorableError` (a `ValueError`), names the
column or `Class.field`, the declared type and ITL-1150.

Migration: hold the set in a dataclass field (a field of a dataclass, an `orcapod.dataclass`
or a pydantic model keeps `set[int]`, `Path`, `UUID` and the other logical types on every
read path), or declare the plain type (`dict[str, list[int]]`). No stored data, stored path or
hash changes: only declarations that used to be accepted are refused.
3 changes: 0 additions & 3 deletions changelog.d/ITL-611.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,5 @@ form are combined. Delete the stored tables of such nodes and run again.

### Known limitations

- Inside a dict value or a tuple field (`dict[str, set[int]]`, `dict[str, list[set[int]]]`), a
set of a primitive keeps its plain `large_list` storage, because Arrow cannot hold an extension type inside a struct
field; read back without the declared type, such a value is a list. (ITL-611)
- `frozenset[T]` is not supported: declare `set[T]`. Support is to be evaluated in ITL-1140.
(ITL-611)
10 changes: 10 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,16 @@ before standalone iteration, table materialization, or sync/async storage can di
Errors name the function, output, declared type and returned type; nested locations show
structure without including returned dictionary keys or converter messages.

A declared type that storage cannot keep is refused when the pod is defined, before any value
is returned. A dictionary is stored as a list of key/value structs, and Arrow cannot hold a
logical type inside a struct field, so a logical type or native container (`Path`, `UUID`, a
user class, `numpy.ndarray`, `set[int]`, `list[set[int]]`, `list[Path]`) as a dictionary key
or value, at any depth (`dict[str, set[int]]`, `list[dict[str, set[int]]]`, a dataclass field
typed `dict[str, set[int]]`), raises `NestedTypeNotStorableError` naming the parameter or
output (ITL-1150). The same types are faithful as a column, inside a list, and as a field of a
dataclass, an `orcapod.dataclass` or a pydantic model: hold such a value in a dataclass
field, or declare the plain type (`dict[str, list[int]]`).

Outputs are validated against the declared storage schema immediately, and accepted numeric
exceptions are normalized so downstream pods, hashes and stored rows see the same value.
Eager validation never replaces a compatible user-class object; stored reads keep their
Expand Down
11 changes: 8 additions & 3 deletions docs/concepts/logical-type-identifiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,14 @@ print(labels_type.storage_type)
- The elements are stored in the canonical set order (see the hashing reference, section 1e),
so equal sets store, and hash, identically. A set row never hashes like the list row with
the same elements, because the container's identifier is part of the row hash.
- **Known limit:** inside a dict value or a tuple field (`dict[str, set[int]]`,
`dict[str, list[set[int]]]`), a set of a primitive keeps its plain `large_list` storage, because Arrow cannot hold an extension
type inside a struct field; read back without the declared type, such a value is a list.
- A native container, like any logical type, cannot be a dict key or value or a tuple field
yet (`dict[str, set[int]]`, `dict[str, list[set[int]]]`, `dict[set[int], int]`,
`list[dict[str, set[int]]]`): Arrow cannot hold an extension type inside a struct field, so
the value would read back as a list. Such a declared type is refused where it is declared
(a pod's parameter or output, a source's schema, a user-class field), with a
`NestedTypeNotStorableError` that names the column or `Class.field` (ITL-1150). Hold the set
in a dataclass field, where it keeps its type, or declare the plain type
(`dict[str, list[int]]`).
- `frozenset[T]` is not supported: declare `set[T]`. Support is to be evaluated in ITL-1140.

A container type you implement yourself is not a native container. It is an ordinary logical
Expand Down
7 changes: 5 additions & 2 deletions src/orcapod/logical_types/user_class_logical_type.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
require_restorable_none,
with_context,
)
from orcapod.semantic_types.nested_storage import NestedTypeNotStorableError
from orcapod.logical_types.registry import (
make_arrow_extension_type,
make_polars_extension_type,
Expand Down Expand Up @@ -314,6 +315,8 @@ def _storage(
RecursiveTypeError: If the class refers back to itself, naming the path.
NestedNoneNotRestorableError: If a field admits a None a read cannot
restore (ITL-1144), naming the field.
NestedTypeNotStorableError: If a field's type holds a logical type as a
dict key or value (ITL-1150), naming the field.
"""
hints = uci.type_hints(python_type)
arrow_fields = []
Expand All @@ -331,9 +334,9 @@ def _storage(
UserClassLogicalType,
),
)
except NestedNoneNotRestorableError as exc:
except (NestedNoneNotRestorableError, NestedTypeNotStorableError) as exc:
raise with_context(
f"{python_type.__qualname__}.{field_name}", exc
f"{python_type.__qualname__}.{field_name} ({annotation})", exc
) from exc
except TypeError as exc:
if isinstance(exc, RecursiveTypeError):
Expand Down
16 changes: 10 additions & 6 deletions src/orcapod/semantic_types/nested_none.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
import types
import typing
from collections.abc import Callable
from typing import TYPE_CHECKING, Any, get_args, get_origin
from typing import TYPE_CHECKING, Any, TypeVar, get_args, get_origin

if TYPE_CHECKING:
import pyarrow as pa
Expand Down Expand Up @@ -110,10 +110,14 @@ def unrestorable_none_error(arrow_type: pa.DataType) -> NestedNoneNotRestorableE
)


def with_context(
context: str, error: NestedNoneNotRestorableError
) -> NestedNoneNotRestorableError:
"""The same refusal with the column or field it concerns put first.
_Refusal = TypeVar("_Refusal", bound=ValueError)


def with_context(context: str, error: _Refusal) -> _Refusal:
"""The same refusal, of the same class, with the column or field it concerns put first.

Used for every refusal of a declared nested type: a None a read cannot restore
(ITL-1144) and a logical type storage cannot keep (ITL-1150).

Args:
context: What the caller knows the refusal concerns, e.g. ``"Column 'v'"``.
Expand All @@ -122,4 +126,4 @@ def with_context(
Returns:
A new error whose message starts with ``context``.
"""
return NestedNoneNotRestorableError(f"{context}: {error}")
return type(error)(f"{context}: {error}")
47 changes: 47 additions & 0 deletions src/orcapod/semantic_types/nested_storage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
"""Which nested positions storage can keep a logical type at (ITL-1150).

A Python ``dict`` is stored as a list of key/value structs and a tuple as a struct (or a
fixed-size list), and Arrow cannot hold an extension type inside a struct field or a list
value field. A logical type or a native container (``set[int]``, ``list[set[int]]``) as a
dict key or value or a tuple field would therefore be stored as its plain storage and read
back as something else than declared (a set as a list). Until such positions are stored
faithfully, a declared type with one is refused where it is declared, so no value is stored
lossily. A field of a user class is not such a position: the class decodes each field by its
declared type.
"""

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
import pyarrow as pa

_FOLLOW_UP = "ITL-1150"


class NestedTypeNotStorableError(ValueError):
"""A logical type is declared where storage cannot keep it (ITL-1150)."""


def require_storable_in_struct_field(arrow_type: pa.DataType) -> None:
"""Refuse a logical type or native container as a dict key or value or a tuple field.

Args:
arrow_type: The Arrow type the declared key, value or field type registers as.

Raises:
NestedTypeNotStorableError: If ``arrow_type`` is an extension type (a logical type
or a native container such as ``set[int]``).
"""
import pyarrow as pa

if isinstance(arrow_type, pa.ExtensionType):
raise NestedTypeNotStorableError(
f"a value of Arrow extension type {arrow_type.extension_name!r} cannot be a "
f"dict key or value or a tuple field yet: Arrow cannot hold an extension type "
f"inside a struct field, so the value would not read back as declared (a set "
f"would read as a list). Hold it in a dataclass field, where it keeps its type, "
f"or declare the plain type (e.g. dict[str, list[int]] for "
f"dict[str, set[int]]). Follow-up: {_FOLLOW_UP}."
)
Loading
Loading