Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions changelog.d/ITL-1144.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
### Breaking changes and migration

- A `None` inside a container is refused where a read could not give it back (ITL-1144). A list or set element, a dict value or a class field declared `X | None`, where `X` is a container (`list`, `set`, `dict`) or a logical type other than your own class (`Path`, `UPath`, `UUID`, `numpy.ndarray`, `torch.Tensor`, pandas and Polars values), now raises `ValueError` when the type is declared, as a function's parameter or return value, a source's `data_schema`, or a field of a dataclass or pydantic model. Examples: `list[set[int] | None]`, `list[list[int] | None]`, `dict[str, list[int] | None]`, `list[Path | None]`, `set[UUID | None]`, a field `x: UUID | None` or `x: list[int] | None`. The error names the column (or `Class.field`), the type and ITL-1144. Such a `None` was stored correctly, but read back as an empty container (`[]`, `set()`, `{}`), with a different row hash, or the read failed with a `TypeError`: through `as_source()`, a `DictSource`, a loaded job and a downstream function. An in-memory pod-to-pod chain that never stored or re-read the values did keep the `None`; it is refused too. Declare the inner type without `| None`, or hold the value in a dataclass and make that optional (`list[MyData | None]`). Still accepted, unchanged: a `None` inside a container where the inner type is a scalar (`list[int | None]`, `dict[str, float | None]`, a field `x: int | None`) or your own class (`list[MyData | None]`), and a `None` at the top level of any column (`list[Path] | None`). No hash, stored table or golden value changes.
- Reading a stored column that holds such a `None` raises `ValueError` naming the column, instead of returning a wrong value or a `TypeError` (ITL-1144). A null key/value entry of a stored dict raises too, instead of being dropped. This covers an Arrow table given to `ArrowTableSource` with a null inner list or struct, and `GroupBy` or `Batch` over a nullable list, set, `Path` or `UUID` column, whose output rows read as Python values now raise. `GroupBy` and `Batch` over a nullable scalar column (`list[int | None]`) are unchanged. Drop the nulls first (`DropNulls` on that column) or keep the value in a dataclass.
- A source given a `None` where its declared type does not allow one now refuses instead of storing an empty container or a wrong value (ITL-1144). A `None` for a list, set or dict whose declared type is not `| None`, inside a column's value (`[None, [1]]` for `list[list[int]]`, `{"k": None}` for `dict[str, list[int]]`, a dataclass field `xs: list[int]` given `None`) or as the column's own value, was stored as `[]`, `set()` or `{}`. A `None` for a logical type declared without `| None` reached the type's encoder: a `Path` was stored as the string `'None'` and read back as `Path('None')`, a `list[UUID]` as `[]`, and a `UUID` failed with an `AttributeError`; a user class raised `TypeError` and now raises the same `UnsupportedValueError`. `DictSource`, `ListSource` and any other source built from Python values now raise `UnsupportedValueError` (a `ValueError`) naming the column and the declared type. A function output already refused such a value (`OutputTypeError`). A `None` at a position declared `| None` is unchanged, and no hash, stored table or golden value of a valid value changes.
33 changes: 33 additions & 0 deletions design_issues/ITL-1144-NN1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## `src/orcapod/semantic_types/universal_converter.py`

### NN1 — A None inside a container is stored but not read back for a container or a logical type
**Status:** in progress
**Severity:** critical
**Issue:** ITL-1144

A `None` inside a container column (a list or set element, a dict value, a user-class field)
is stored faithfully as an Arrow null, and the in-memory row hash equals the stored row hash.
Reading the stored column back to Python restored it only where the position's type is a
scalar or a user class. A container read as an empty one (`list[set[int] | None]` holding
`[None, {1}]` read `[[], [1]]`; a dataclass field `x: list[int] | None` read `[]`), with a
different row hash; any other logical type failed (`list[UUID | None]`,
`list[Path | None]`, `list[np.ndarray | None]`: `TypeError`). Every read path agreed:
`as_source()`, `DictSource`, a loaded job, a downstream function, and `GroupBy` or `Batch`
over a nullable list, set, `Path` or `UUID` column, where a downstream function received
and stored the wrong value. An in-memory pod-to-pod chain kept the `None`. The decoders
map a null to an empty value (`if value else []`) or hand it to a logical type that does not
accept it.

**Fix (guard, v0.3.0):** `semantic_types/nested_none.py` states which nested None a read
restores. `register_python_class` and the user-class field registration refuse a declared
`X | None` at a nested position whose None cannot be restored
(`NestedNoneNotRestorableError`, naming the column or `Class.field`), and the nested
decoders raise on a stored null there instead of returning a wrong value (Arrow-sourced
columns, whose item nullability the Python type does not show, ITL-1136). A None given for a list,
set or dict whose declared type does not admit one, which the write converters stored as an
empty container, is refused at write (`UnsupportedValueError`, naming the column); an Arrow
map, whose decoder passes a null through, stays readable. Still open: read
these None values back faithfully (a read-side change; storage is already right, so no
stored value or hash moves) and lift the guard. Land ITL-1136's fix only after it: reading
item nullability into the Python type would make every default-nullable nested list read
from Arrow a refused type.
54 changes: 54 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -1123,6 +1123,60 @@ for tag, data in count_rows.pod(readings).iter_data():
# s2 3
```

### A `None` inside a container

A `None` inside a list, a set, a dict value or a class field is accepted where its type is
a scalar (`int`, `float`, `str`, `bool`, `bytes`, `datetime`, ...) or a class of your own
(a dataclass, an `orcapod.dataclass` or a pydantic model): `list[int | None]`,
`dict[str, float | None]`, `list[MyData | None]` and a field `x: int | None` store and read
back exactly.

Where the type inside is a container (`list`, `set`, `dict`) or one of the logical types
above (`Path`, `UUID`, `numpy.ndarray`, ...), a stored `None` is not yet read back
faithfully: it would read as an empty container or fail. Such a declaration is refused when
the type is declared: `list[set[int] | None]`, `list[Path | None]`, `dict[str, list[int] |
None]` or a dataclass field `x: UUID | None`, as a function's parameter or return value, a
source's schema or a class field. The error names the column (or the class field) and the
type. Declare the inner type without `| None`, or hold the value in a dataclass and make
that optional (`list[MyData | None]`). A `None` at the top level of a column
(`list[Path] | None`, `UUID | None`) is accepted as before. Lifting this restriction is
ITL-1144.

```python
try:
DictSource(
data=[{"sample": "s1", "labels": [{1, 2}, None]}],
tag_columns=["sample"],
data_schema={"sample": str, "labels": list[set[int] | None]},
)
except ValueError as error:
print(str(error).split(": ")[0])
# Column 'labels' (list[set[int] | None])

labels = DictSource(
data=[{"sample": "s1", "counts": [3, None, 1]}],
tag_columns=["sample"],
data_schema={"sample": str, "counts": list[int | None]},
)
for tag, data in labels.iter_data():
print(data["counts"])
# [3, None, 1]
```

A column that already holds such a `None` (an Arrow table given to `ArrowTableSource`, or
`GroupBy` or `Batch` over a column of a nullable list, set, `Path` or `UUID`) raises a
`ValueError` naming the column when its values are read as Python values, instead of
returning a wrong value. Before `GroupBy` or `Batch`, `DropNulls` on that column removes
the rows whose value is `None`, so no `None` reaches the lists they build. `DropNulls` reads
only whether a column's own value is null, so it cannot remove a `None` already nested inside
a list: remove it from an Arrow table before giving the table to `ArrowTableSource`. Keeping
the value in a dataclass (`list[MyData | None]`) works on both paths.

A `None` given where the declared type does not allow one (`[None, [1]]` for
`list[list[int]]`) is refused when a source is built from it, naming the column: it used to
be stored as an empty list. A function returning one is refused as described in
[Return values must match the declared type](#return-values-must-match-the-declared-type).

## Multiple input streams

If you pass multiple streams to a function pod, they are automatically joined (using
Expand Down
20 changes: 20 additions & 0 deletions src/orcapod/logical_types/user_class_logical_type.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@
import orcapod.logical_types.user_class_identity as uci
from orcapod.logical_types.base_logical_type import BaseLogicalType
from orcapod.logical_types.structural_class import class_for_record
from orcapod.semantic_types.nested_none import (
NestedNoneNotRestorableError,
require_restorable_none,
with_context,
)
from orcapod.logical_types.registry import (
make_arrow_extension_type,
make_polars_extension_type,
Expand Down Expand Up @@ -307,6 +312,8 @@ def _storage(
Raises:
ValueError: If the type hints cannot be resolved.
RecursiveTypeError: If the class refers back to itself, naming the path.
NestedNoneNotRestorableError: If a field admits a None a read cannot
restore (ITL-1144), naming the field.
"""
hints = uci.type_hints(python_type)
arrow_fields = []
Expand All @@ -315,6 +322,19 @@ def _storage(
annotation = hints.get(field_name, Any)
try:
arrow_type = converter.register_python_class(annotation)
# A field is a nested position: its None must be readable (ITL-1144).
require_restorable_none(
annotation,
arrow_type,
lambda name: isinstance(
converter.get_logical_type_by_arrow_extension_name(name),
UserClassLogicalType,
),
)
except NestedNoneNotRestorableError as exc:
raise with_context(
f"{python_type.__qualname__}.{field_name}", exc
) from exc
except TypeError as exc:
if isinstance(exc, RecursiveTypeError):
raise RecursiveTypeError(
Expand Down
125 changes: 125 additions & 0 deletions src/orcapod/semantic_types/nested_none.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
"""Which None at a nested position a read restores (ITL-1144).

A None inside a container column (a list or set element, a dict value, a user-class
field) is stored faithfully as an Arrow null. Reading the column back to Python restores
it only where the position's type is a primitive or a user class: a container read as an
empty container and any other logical type failed. Until the read restores every such
None (the ITL-1144 follow-up), a declared type with such a position is refused, and a
stored null at such a position raises when it is read, so no wrong value gets out.
"""

from __future__ import annotations

import types
import typing
from collections.abc import Callable
from typing import TYPE_CHECKING, Any, get_args, get_origin

if TYPE_CHECKING:
import pyarrow as pa

IsUserClass = Callable[[str], bool]
"""Whether an Arrow extension name is the logical type of a user class (a dataclass,
an ``orcapod.dataclass`` or a pydantic model). The caller supplies it, so this module
does not import the user-class logical types, which import it."""

_FOLLOW_UP = "ITL-1144"


class NestedNoneNotRestorableError(ValueError):
"""A None at a nested position cannot be read back faithfully (ITL-1144)."""


def none_is_restorable(arrow_type: pa.DataType, is_user_class: IsUserClass) -> bool:
"""Whether a read restores a null at a nested position of ``arrow_type`` as None.

A user class (dataclass, ``orcapod.dataclass``, pydantic model), every scalar type
and an Arrow map (its decoder passes a null through; a Python ``dict`` is stored as
a list of key/value structs, never as a map) are restored; a list or struct reads
as an empty value, and any other logical type fails.

Args:
arrow_type: The Arrow type of the nested position.
is_user_class: Whether an extension name is a user class's logical type.

Returns:
True when a null there is read back as None.
"""
import pyarrow as pa

if isinstance(arrow_type, pa.ExtensionType):
return is_user_class(arrow_type.extension_name)
return not (
pa.types.is_list(arrow_type)
or pa.types.is_large_list(arrow_type)
or pa.types.is_fixed_size_list(arrow_type)
or pa.types.is_list_view(arrow_type)
or pa.types.is_large_list_view(arrow_type)
or pa.types.is_struct(arrow_type)
)


def _is_optional(annotation: Any) -> bool:
"""Whether ``annotation`` is a union that includes None."""
origin = get_origin(annotation)
return (origin is typing.Union or origin is types.UnionType) and type(
None
) in get_args(annotation)


def require_restorable_none(
annotation: Any, arrow_type: pa.DataType, is_user_class: IsUserClass
) -> None:
"""Refuse a nested position declared ``X | None`` whose None a read cannot restore.

Args:
annotation: The declared type of the nested position (a list or set element, a
dict value, a user-class field).
arrow_type: The Arrow type ``annotation`` registers as.
is_user_class: Whether an extension name is a user class's logical type.

Raises:
NestedNoneNotRestorableError: If ``annotation`` admits None and a stored None
there would not be read back as None.
"""
if _is_optional(annotation) and not none_is_restorable(arrow_type, is_user_class):
raise NestedNoneNotRestorableError(
f"{annotation} is not supported inside a container or as a class field yet: "
f"a None there is stored but not read back faithfully (it would read as an "
f"empty value or fail). Declare it without '| None', or hold the value in a "
f"dataclass and make that optional (e.g. list[MyData | None]). Follow-up: "
f"{_FOLLOW_UP}."
)


def unrestorable_none_error(arrow_type: pa.DataType) -> NestedNoneNotRestorableError:
"""The error for a stored null read at a nested position it cannot be restored at.

Args:
arrow_type: The Arrow type of the nested position holding the null.

Returns:
The error to raise.
"""
return NestedNoneNotRestorableError(
f"a stored None inside a container, at a position of Arrow type {arrow_type}, "
f"cannot be read back faithfully yet (it would read as an empty value or fail). "
f"Remove the None before the value is put in a container (e.g. DropNulls on "
f"the column before GroupBy or Batch), or hold the value in a dataclass. "
f"Follow-up: {_FOLLOW_UP}."
)


def with_context(
context: str, error: NestedNoneNotRestorableError
) -> NestedNoneNotRestorableError:
"""The same refusal with the column or field it concerns put first.

Args:
context: What the caller knows the refusal concerns, e.g. ``"Column 'v'"``.
error: The refusal raised below the caller.

Returns:
A new error whose message starts with ``context``.
"""
return NestedNoneNotRestorableError(f"{context}: {error}")
Loading
Loading