Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
92e0f53
docs(specs): ITL-611 design for set of primitives (proposal)
Nov 14, 2026
da7d976
test(logical_types): failing tests for a set of primitives (ITL-611)
Nov 14, 2026
3e4c359
feat(logical_types): a set of a primitive is the native set container…
Nov 14, 2026
eebc063
fix(pipeline): a loaded job restores native containers from their ide…
Nov 14, 2026
4f1ba42
test(golden): add set[int], set[str], set[bytes] entries (ITL-611)
Nov 14, 2026
2b4916f
docs: a set of a primitive is a native container (ITL-611)
Nov 14, 2026
abe9d7a
refactor: type the native-container helpers (ITL-611)
Nov 14, 2026
e7709d2
docs(plans): ITL-611 follow-up dispatch steps
Nov 14, 2026
7c1d48f
Merge remote-tracking branch 'origin/main' into eywalker/itl-611-exte…
Nov 14, 2026
595d336
docs(plans): ITL-611 follow-up notes
Nov 14, 2026
b22cb6f
Merge remote-tracking branch 'origin/main' into eywalker/itl-611-exte…
Nov 14, 2026
2276d84
Merge remote-tracking branch 'origin/main' into eywalker/itl-611-exte…
Nov 15, 2026
3ec7738
docs(plans): ITL-611 CodeRabbit round 1 plan
Nov 15, 2026
8b75cce
fix(semantic_types): a set's element Python type comes from its Arrow…
Nov 15, 2026
52f26a7
Merge remote-tracking branch 'origin/main' into eywalker/itl-611-exte…
Nov 15, 2026
7a4797f
docs(plans): ITL-611 CodeRabbit round 2 finding
Nov 15, 2026
f74a939
fix(logical_types): a set of a plain element is found by its identifi…
Nov 15, 2026
077262d
docs(plans): ITL-611 set metadata element_kind arrow
Nov 15, 2026
24cc3d9
fix(logical_types): a set of a primitive writes element_kind arrow, e…
Nov 15, 2026
e7bf4e0
test(golden): ITL-611 set data hashes under the element_kind metadata
Nov 15, 2026
928482f
docs(logical_types): set of a primitive metadata is element_kind arro…
Nov 15, 2026
c4f4887
fix(semantic_types): a registered logical type refuses a stored colum…
Nov 15, 2026
e3baffa
Merge remote-tracking branch 'origin/main' into eywalker/itl-611-exte…
Nov 15, 2026
f548f5c
fix(logical_types): a list of sets of a plain element is found by its…
Nov 15, 2026
d091a74
Merge remote-tracking branch 'origin/main' into eywalker/itl-611-exte…
Nov 15, 2026
106648b
fix(logical_types): the list factory reads a set element as its Arrow…
Nov 15, 2026
76679d7
fix(logical_types): a set whose metadata names no element is refused …
Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,22 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
identifier is the container kind plus the element's identifier; the type hash treats them
like any generic alias (origin and arguments). A change to how they are stored or hashed is
a data context (global hash version) change, not a logical type change.
- `set[T]` of a primitive is a native container too (developer, November 14, 2026, ITL-611):
`ListLogicalType` over an `ArrowElement`, extension name `set[<large Arrow type text>]`
(`set[int64]`), metadata `{"category": "set", "element_kind": "arrow"}` (developer,
November 15, 2026: the name holds the element as it is and is the registry key; the
metadata never repeats it), elements in the canonical set order (`sort_set_elements`). A
reader takes the element from the storage (`large_list<int64>`) and refuses
(`LogicalTypeIdentifierError`) a storage that is not a list of the element the name names.
It reads as `set[<Python type of the element Arrow type>]` (`set[int8]` and `set[int64]`
both as `set[int]`, as for lists), so it is
registered by identifier only, never bound to a Python type (`binds_python_type`), and
`set[T]` maps to Arrow through `T`. `list[T]` of a primitive stays a plain
`large_list`: never wrap it. Inside a dict value or tuple field a set of a primitive keeps
its plain storage (Arrow has no extension type in a struct field). A container's elements
are value-hashed only when its innermost element is a logical type. A saved schema rebuilds
an unregistered native container from its identifier (`serialization.parse_arrow_type_string`).
`frozenset[T]` is refused (ITL-1140).
- On read, stored metadata must be one of the type's `readable_extension_metadata()`;
otherwise raise `LogicalTypeIdentifierError`. Never wrap a stored column by name alone and
never fall back to the storage type. List an older value in `also_reads` only when the
Expand Down
16 changes: 16 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,22 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
identifier is the container kind plus the element's identifier; the type hash treats them
like any generic alias (origin and arguments). A change to how they are stored or hashed is
a data context (global hash version) change, not a logical type change.
- `set[T]` of a primitive is a native container too (developer, November 14, 2026, ITL-611):
`ListLogicalType` over an `ArrowElement`, extension name `set[<large Arrow type text>]`
(`set[int64]`), metadata `{"category": "set", "element_kind": "arrow"}` (developer,
November 15, 2026: the name holds the element as it is and is the registry key; the
metadata never repeats it), elements in the canonical set order (`sort_set_elements`). A
reader takes the element from the storage (`large_list<int64>`) and refuses
(`LogicalTypeIdentifierError`) a storage that is not a list of the element the name names.
It reads as `set[<Python type of the element Arrow type>]` (`set[int8]` and `set[int64]`
both as `set[int]`, as for lists), so it is
registered by identifier only, never bound to a Python type (`binds_python_type`), and
`set[T]` maps to Arrow through `T`. `list[T]` of a primitive stays a plain
`large_list`: never wrap it. Inside a dict value or tuple field a set of a primitive keeps
its plain storage (Arrow has no extension type in a struct field). A container's elements
are value-hashed only when its innermost element is a logical type. A saved schema rebuilds
an unregistered native container from its identifier (`serialization.parse_arrow_type_string`).
`frozenset[T]` is refused (ITL-1140).
- On read, stored metadata must be one of the type's `readable_extension_metadata()`;
otherwise raise `LogicalTypeIdentifierError`. Never wrap a stored column by name alone and
never fall back to the storage type. List an older value in `also_reads` only when the
Expand Down
52 changes: 52 additions & 0 deletions changelog.d/ITL-611.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
### Breaking changes and migration

#### A set of a primitive is stored as the native set container; its rows hash differently (ITL-611)

A column declared `set[T]`, where `T` has no logical type (`set[int]`, `set[str]`,
`set[bytes]`, `set[datetime]`), is now stored under the Arrow extension type
`set[<element Arrow type>]` (for example `set[int64]`, metadata
`{"category": "set", "element_kind": "arrow"}`: the element type is in the name and is read
from the storage, `large_list<int64>`) instead of a plain `large_list`. This moves:

- the stored Arrow type of such a column (at the top level, as `set[T] | None`, and as the
element of a list: `list[set[int]]` is stored as `list[set[int64]]`);
- the row hash of every row that holds such a column, and with it the input hash of a function
pod that takes such a column (its results are computed again), and the default `source_id`
and content hash of an in-memory source with such a column.

Type, schema and pipeline hashes do not move, nor does anything about `list[T]` or about a set
of a logical type. No released version wrote such a row: only stores written by development
builds of v0.3 hold a set of a primitive as a plain list. Such a store is not refused: a node
serves its stored values as lists, and a run that adds a row raises when the stored and the new
form are combined. Delete the stored tables of such nodes and run again.

### Fixed

- **A `set[T]` of a primitive reads back as a set everywhere.** A downstream pod annotated
`set[int]` received a `list`, even inside the job that computed it; `node.as_source()` read
a `list` under a schema that said `set[int]`; and a job loaded in another process said
`list[int]`. The row `{1, 2, 3}` as `set[int]` also hashed exactly like `[1, 2, 3]` as
`list[int]`. A set now keeps its type on every path: the node, a downstream pod,
`as_source()`, `DerivedSource`, and a job loaded in a fresh process, sync and async, on
Delta and in memory. Its Python type is that of the element's Arrow type, as for a list:
`set[np.int8]` and `set[int]` both read as `set[int]`, each column keeps its own width, and
both can be declared in one process. `list[T]` of a primitive is unchanged (a plain
`large_list`). (ITL-611)
- **A job loaded in a fresh process restores `list[T]` and `set[T]` columns of a logical type**
(`set[Path]`, `set[UUID]`, `list[Path]`): their saved type could not be restored unless the
type had been used in that process first, and the stored view read the raw storage, with a
row hash that differed from the one in the process that wrote it. A saved schema now rebuilds
such a container from its identifier. (ITL-611)

- **A stored column whose storage is not its logical type's storage raises
`LogicalTypeIdentifierError`**, naming the column and both storage types, whether or not the
type was registered in the process before (it raised a `TypeError` from Arrow when it was).
(ITL-611)

### Known limitations

- Inside a dict value or a tuple field (`dict[str, set[int]]`, `dict[str, list[set[int]]]`), a
set of a primitive keeps its plain `large_list` storage, because Arrow cannot hold an extension type inside a struct
field; read back without the declared type, such a value is a list. (ITL-611)
- `frozenset[T]` is not supported: declare `set[T]`. Support is to be evaluated in ITL-1140.
(ITL-611)
38 changes: 38 additions & 0 deletions design_issues/ITL-611-NC1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
## `src/orcapod/logical_types/`

### NC1 — A `set[T]` of a primitive was stored, read back and hashed as a list
**Status:** resolved
**Severity:** high
**Issue:** ITL-611

`set[T]` where `T` has no logical type (`set[int]`, `set[str]`, `set[bytes]`,
`set[datetime]`) was stored as a plain `large_list<T>`, the same Arrow type as `list[T]`. The
set identity was in no stored table and in no row hash: a downstream pod annotated
`set[int]` received a `list` even inside the job that computed it, `node.as_source()` read a
`list` under a schema that said `set[int]`, a job loaded in another process said `list[int]`,
and the row `{1, 2, 3}` as `set[int]` hashed exactly like `[1, 2, 3]` as `list[int]`. A
`set[T]` of a logical type (`set[Path]`) was already the native set container, but a job
loaded in a fresh process could not restore it (`parse_arrow_type_string` resolved only
extension names already registered in the process) and read its raw storage, with a row hash
that differed from the in-process one.

**Fix:** the developer approved (November 14, 2026) storing a set of a primitive as the
native set container: `ListLogicalType` over an `ArrowElement`, extension name
`set[<element Arrow type in its large variants>]` (`set[int64]`), metadata
`{"category": "set", "element_kind": "arrow"}` (developer, November 15, 2026: the name
holds the element as it is and is the registry key; the metadata does not repeat it, and a
reader takes the element from the storage, refusing a storage that is not a list of the
element the name names), no version key, elements in the
canonical set order of `canonical_order.sort_set_elements` (unchanged, so the stored values
are as before and only the type identity moves). `list[T]` of a primitive stays a plain
`large_list`. A container's elements are value-hashed only when its innermost element is a
logical type (a `set[bytes]` is hashed as stored, as a `list[bytes]` column is). A saved
schema rebuilds an unregistered native container from its identifier through the registered
factory, so a loaded job restores `set[int]` and `set[Path]` alike. The row hash of every row
holding a set of a primitive moves (documented; no golden value moved, entries for
`set[int]`, `set[str]` and `set[bytes]` were added). Known limits: inside a dict value or a
tuple field a set of a primitive keeps its plain storage (Arrow has no extension type in a
struct field); `frozenset[T]` is refused, its support to be evaluated in ITL-1140; a store
written by a development build of v0.3 that holds a set of a primitive as a plain list is not
refused on read (it serves lists, and a run that adds a row to it raises when the two forms
are concatenated): clear such nodes.
43 changes: 42 additions & 1 deletion docs/concepts/logical-type-identifiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,8 @@ containers*. They are logical types only because Arrow cannot hold an extension
a list; conceptually they are an ordinary list or set of `T`. By design:

- **A native container has no version of its own.** Its metadata is JSON with `category`
(`list` or `set`) and the element's extension name and metadata, and nothing else. Its
(`list` or `set`) and the element's extension name and metadata (for a set of a primitive,
the element's Arrow type; see below), and nothing else. Its
identifier is the container kind plus the element's identifier, so it changes exactly when
the element's identifier changes. Because the element's metadata is written into that
JSON as text, a logical type used inside `list[T]` or `set[T]` must have extension
Expand All @@ -127,6 +128,46 @@ a list; conceptually they are an ordinary list or set of `T`. By design:
hashing**, recorded by the data context (the global hash version), never a change of a
logical type.

**A set of a primitive is a native container too** (ITL-611). `set[T]` where `T` has no
logical type (`set[int]`, `set[str]`, `set[bytes]`, `set[datetime]`) is stored under the
wrapper `set[<element Arrow type>]`, so a set reads back as a set everywhere: from the
node, in a downstream pod, from `as_source()` and from a job loaded in another process.
The element's identifier is its Arrow type in the large variants, as text, inside the
extension name, which is the key the type is registered under. The metadata does not repeat
it: `"element_kind": "arrow"` (never `element_ext_name`, which names a logical type) says
that the element is a plain Arrow type, which a reader takes from the storage
(`large_list<large_string>`). A stored column whose storage is not a list of the element its
name names is refused with `LogicalTypeIdentifierError`.

```python
from orcapod.sources import DictSource

source = DictSource(
data=[{"k": "r", "labels": {"b", "a"}}],
tag_columns=["k"],
data_schema={"k": str, "labels": set[str]},
)
labels_type = source.as_table().schema.field("labels").type
print(labels_type.extension_name)
# set[large_string]
print(labels_type.__arrow_ext_serialize__().decode())
# {"category": "set", "element_kind": "arrow"}
print(labels_type.storage_type)
# large_list<item: large_string>
```

- `list[T]` of a primitive is not a native container: it stays a plain `large_list`, because
a list is a list in Arrow too.
- `set[T | None]` has the identifier of `set[T]`, as `set[Path | None]` has that of
`set[Path]`: the element's nullability is not part of a native container's identifier.
- The elements are stored in the canonical set order (see the hashing reference, section 1e),
so equal sets store, and hash, identically. A set row never hashes like the list row with
the same elements, because the container's identifier is part of the row hash.
- **Known limit:** inside a dict value or a tuple field (`dict[str, set[int]]`,
`dict[str, list[set[int]]]`), a set of a primitive keeps its plain `large_list` storage, because Arrow cannot hold an extension
type inside a struct field; read back without the declared type, such a value is a list.
- `frozenset[T]` is not supported: declare `set[T]`. Support is to be evaluated in ITL-1140.

A container type you implement yourself is not a native container. It is an ordinary logical
type with its own identifier, which you declare and change like that of any other type.

Expand Down
15 changes: 14 additions & 1 deletion docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -582,7 +582,9 @@ counts column order. A Polars frame and the equal pandas frame never share a has

A `dict` and a `set` have no order, so equal values must hash equally whatever order they
were built in. Arrow stores both as lists: `dict[K, V]` as `large_list<struct<key, value>>`
and a plain `set[T]` as `large_list<T>`. Orcapod puts them in canonical order at the point
and a `set[T]` of a primitive as the native set container `set[<element Arrow type>]` over
`large_list<T>` (ITL-611; see [Logical Type Identifiers](../concepts/logical-type-identifiers.md)).
Orcapod puts them in canonical order at the point
where a Python value becomes Arrow, so the stored list, and therefore the row hash, does not
depend on insertion order (dicts) or on `PYTHONHASHSEED` (sets of `str` or `bytes`).

Expand Down Expand Up @@ -661,6 +663,17 @@ of affected inputs, and changing storage order would move row hashes and what a
`set[T]` of a logical type (`set[Path]`, `set[UUID]`) is ordered by `ListLogicalType`, not
by this rule.

**The set's identity is part of the row hash** (ITL-611). A set of a primitive is stored
under its native container type, and the Arrow hasher hashes an extension column's
identifier (extension name and metadata) with its stored values. So `{1, 2, 3}` as
`set[int]` and `[1, 2, 3]` as `list[int]` are two different rows, although both store the
list `[1, 2, 3]`. The elements of a set of a primitive are hashed as stored, never replaced
by their value hashes: only a container whose innermost element is a logical type is (a
`set[bytes]` hashes like its stored `large_list<large_binary>` plus the identifier, although
the semantic hasher has a handler for `bytes`). Before ITL-611 a set of a primitive was a
plain `large_list` and hashed exactly like the list row; such rows were written only by
development builds of v0.3.

---

## 2. Source Provenance & System Tags
Expand Down
58 changes: 35 additions & 23 deletions src/orcapod/hashing/visitors.py
Original file line number Diff line number Diff line change
Expand Up @@ -300,27 +300,47 @@ def _hashes_extension(self, extension_type: "pa.ExtensionType") -> bool:

The same type-level decision ``visit_extension`` takes for a non-null
value: a registered value hasher for the Python type, or for the innermost
element of a list- or set-backed extension type.
element of a list- or set-backed extension type when that element is a
logical type.
"""
python_type = self._type_converter.arrow_type_to_python_type(extension_type)
if typing.get_origin(python_type) in (list, set) and pa.types.is_large_list(
extension_type.storage_type
):
args = typing.get_args(python_type)
if args:
inner = args[0]
while typing.get_origin(inner) in (list, set):
inner_args = typing.get_args(inner)
if not inner_args:
break
inner = inner_args[0]
return isinstance(
inner, type
) and self._python_hasher.type_handler_registry.has_handler(inner)
return self._hashes_container_elements(args[0])
if python_type is typing.Any or not isinstance(python_type, type):
return False
return self._python_hasher.type_handler_registry.has_handler(python_type)

def _hashes_container_elements(self, element_python_type: Any) -> bool:
"""Return True if a native container's elements are hashed by value.

Unwraps list and set nesting to the innermost element type. Its values are
hashed by value only when it is a logical type with a registered value
hasher. A container of a plain Arrow element (``set[bytes]``, ITL-611) is
hashed as stored, like a plain ``list[bytes]`` column, even where the
semantic hasher has a handler for the element's Python type.

Args:
element_python_type: The Python type of the container's elements.

Returns:
Whether the elements are replaced by their value hashes.
"""
inner = element_python_type
while typing.get_origin(inner) in (list, set):
inner_args = typing.get_args(inner)
if not inner_args:
break
inner = inner_args[0]
return (
isinstance(inner, type)
and self._type_converter.get_logical_type(inner) is not None
and self._python_hasher.type_handler_registry.has_handler(inner)
)

def visit_extension(
self,
extension_type: "pa.ExtensionType",
Expand Down Expand Up @@ -370,19 +390,11 @@ def visit_extension(
elem_python_type = args[0]

# Type-driven hashability: unwrap list/set nesting to the innermost
# non-container type and check whether it has a semantic handler.
# Decision is made once per column (not per row) so empty and null
# inner lists are handled correctly without crashing.
inner = elem_python_type
while typing.get_origin(inner) in (list, set):
inner_args = typing.get_args(inner)
if not inner_args:
break
inner = inner_args[0]
hashable = isinstance(
inner, type
) and self._python_hasher.type_handler_registry.has_handler(inner)
if not hashable:
# non-container type and check whether it is a logical type with a
# semantic handler. Decision is made once per column (not per row)
# so empty and null inner lists are handled correctly without
# crashing.
if not self._hashes_container_elements(elem_python_type):
# Innermost element type has no semantic handler — whole-column
# passthrough, identical to main-branch behaviour.
return extension_type, storage_value
Expand Down
8 changes: 6 additions & 2 deletions src/orcapod/logical_types/identifier.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,9 +102,13 @@ def for_column(self, column: str) -> "LogicalTypeIdentifierError":
column: The name of the column the identifier was stored with.

Returns:
A new ``LogicalTypeIdentifierError`` with ``column`` set.
A new ``LogicalTypeIdentifierError`` with ``column`` set and this
error's notes.
"""
return LogicalTypeIdentifierError(self.stored, self.readable, column)
error = LogicalTypeIdentifierError(self.stored, self.readable, column)
for note in getattr(self, "__notes__", ()):
error.add_note(note)
return error


def version_metadata(version: int) -> bytes:
Expand Down
Loading
Loading