Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
c3c0cd1
docs(spec): value-based array format for numpy.ndarray in v0.3 (ITL-888)
Nov 11, 2026
506c13e
feat(arrays): shared array storage struct and value digest v1 (ITL-888)
Nov 11, 2026
eca0432
test(arrays): public-interface array scaffolding across operators, st…
Nov 11, 2026
6d42164
docs(spec): nullable struct children (F1), MergeJoin array ordering (…
Nov 11, 2026
be14416
feat(arrays)!: value-based array format for numpy.ndarray (ITL-888, I…
Nov 11, 2026
e9fe738
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
8111672
test(hashing): update golden array entries for the v0.3 array format …
Nov 11, 2026
44bc8c9
docs(spec): record the developer's decisions for the array format (IT…
Nov 11, 2026
0c63948
docs(hashing): state the <type>_v<N> value hash label convention (ITL…
Nov 11, 2026
ac68bcd
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
50b4183
fix(arrays): address CodeRabbit review; Pick raises InputValidationEr…
Nov 11, 2026
9802a9a
docs(spec,plan): golden regeneration gates; astype copy=False (ITL-888)
Nov 11, 2026
a3bc9fe
docs(plan): resume notes after restart (ITL-888)
Nov 11, 2026
17aa217
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
25b240a
test(arrays): SemiJoin with an array column passes after ITL-847 (ITL…
Nov 11, 2026
ad7a3de
fix(arrays): address CodeRabbit round 4 docs and fixture findings (IT…
Nov 11, 2026
ac8d0f4
docs(spec): torch.Tensor value type for v0.3, draft for approval (ITL…
Nov 11, 2026
3483bcd
docs(plan): torch tensor plan; record provisional picks in spec (ITL-…
Nov 11, 2026
aa8fbf2
feat(logical_types): deferred registration of optional value types (I…
Nov 11, 2026
9803670
refactor(array_format): share the stored-row ordering key (ITL-871)
Nov 11, 2026
ad09471
fix(arrays): order lists of arrays in MergeJoin; reject malformed str…
Nov 11, 2026
1e7c729
test(torch): failing tests for the torch.Tensor value type (ITL-871)
Nov 11, 2026
ecb2675
feat(logical_types): torch.Tensor value type orcapod.torch_tensor (IT…
Nov 11, 2026
4e26edb
test(torch): xfail the delta-async store case pending ITL-864 / PR #2…
Nov 11, 2026
47cad93
test(golden): pin torch tensor hashes in an optional torch group (ITL…
Nov 11, 2026
7f301c6
docs(arrays): PyTorch tensors section, hashing label, CHANGELOG and d…
Nov 11, 2026
e19df24
chore(types): targeted pyright ignores for generated extension classe…
Nov 11, 2026
121dd1f
feat(arrays)!: structured field order is part of an array's identity …
Nov 11, 2026
e118c40
fix(review): address CodeRabbit round 1 on PR #300 (ITL-871)
Nov 11, 2026
619cffb
docs(arrays): structured tag example in declared order; review nits (…
Nov 11, 2026
2c1ec63
docs(plan): resume notes after PR #300 review rounds (ITL-871)
Nov 11, 2026
2406da8
docs(plan): phase 2 plan for the developer's decisions (ITL-871)
Nov 11, 2026
f52bc63
chore(merge): merge origin/main after ITL-878 (#287) into ITL-888
Nov 11, 2026
7eec8c0
build(torch): orcapod[torch] extra, dev group installs it, CPU wheels…
Nov 11, 2026
72b7e5a
test(torch): CI guard and no-torch subprocess checks with an import b…
Nov 11, 2026
66c2594
test(torch): failing tests for complex32 and float8 tensors and a dty…
Nov 11, 2026
8e8fdce
feat(torch): store complex32 and the float8 dtypes as raw bits; look …
Nov 11, 2026
9754bee
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 11, 2026
68b5d16
docs(arrays): state the array-format decisions as decided (ITL-888)
Nov 11, 2026
ce01cac
test(torch): failing tests for the GPU/gradient conversion warning; V…
Nov 11, 2026
0dc0f1d
feat(torch): warn once when a GPU or gradient-tracking tensor is stor…
Nov 11, 2026
3e687a1
chore(types): drop torch reportMissingImports ignores now that the de…
Nov 11, 2026
a0c7d02
docs(torch): extra, raw-bit dtypes, conversion warning; CHANGELOG and…
Nov 11, 2026
fcee344
Merge branch 'eywalker/itl-888-implement-the-value-based-array-format…
Nov 11, 2026
4577540
test(torch): MergeJoin of colliding list[torch.Tensor] columns, sync …
Nov 11, 2026
5bc17ea
docs(spec): record the developer's decisions for the torch tensor typ…
Nov 11, 2026
8e9f772
docs(plan): phase 2 resume notes (ITL-871)
Nov 11, 2026
2105a91
docs(spec): decisions are final; drop provisional wording (ITL-871)
Nov 11, 2026
7384e3e
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 11, 2026
9456457
fix(review): keep torch tests and golden group runnable on torch 2.3-…
Nov 11, 2026
4660efc
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 11, 2026
61f78fc
docs(plan): resume notes after the final review rounds (ITL-871)
Nov 11, 2026
5f98283
fix(review): address CodeRabbit on 61f78fc3 (ITL-871)
Nov 11, 2026
22d4b71
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 12, 2026
073618e
test(torch): narrow the pod output to a tensor before comparing (ITL-…
Nov 12, 2026
43e1ded
docs(plan): hold notes for the refresh (ITL-871)
Nov 12, 2026
431d179
test(arrays): failing tests, design note and plan for the strict arra…
Nov 12, 2026
3900ca6
docs(specs): extension name numpy.ndarray and the v0.2 name collision…
Nov 12, 2026
962baa6
docs(specs): scope the Arrow schema hash question (ITL-981)
Nov 12, 2026
ab2684f
docs(plans): record coordinator decisions while waiting for sign-off …
Nov 12, 2026
592274a
docs(specs): final extension name orcapod.numpy.ndarray; drop the for…
Nov 12, 2026
1e5e860
Merge remote-tracking branch 'origin/main' into eywalker/itl-981-libr…
Nov 12, 2026
64c63bf
feat(logical_types): ignore-list mechanism for library value types (I…
Nov 12, 2026
5d6a3aa
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 12, 2026
c515bd4
docs(plan): native storage and strict hash for torch.Tensor (ITL-871)
Nov 12, 2026
ef7965c
fix(output_conformance): a declared class with a logical type require…
Nov 12, 2026
79807a6
Merge branch 'eywalker/itl-981-library-value-types-native-storage-str…
Nov 12, 2026
0f0071c
feat(torch)!: store tensors with torch.save and hash them strictly (I…
Nov 12, 2026
f5d82ae
test(golden): pin the torch value hash per optional property (ITL-871)
Nov 12, 2026
ed734eb
test(torch): load-back of a saved job, list outputs on stores; torch>…
Nov 12, 2026
65e9995
docs(torch): native storage, strict hash and the ignore list (ITL-871)
Nov 12, 2026
9391ad4
test(torch): type-clean reference helpers (ITL-871)
Nov 12, 2026
83d5ad9
docs(torch): state why the extra needs torch>=2.6 (ITL-871)
Nov 12, 2026
e5b5458
docs(agents): what is stored for a subclass is decided by the logical…
Nov 12, 2026
7c4ce4f
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 13, 2026
65d7ce9
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 13, 2026
f3efea8
test(torch): list[torch.Tensor] outputs pass on Delta async now that …
Nov 13, 2026
caddd0c
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 13, 2026
c5c3eef
docs(plan): resume notes after the final merge (ITL-871)
Nov 13, 2026
d6aef9b
fix(registry): deferred handlers resolve in MRO order; recheck after …
Nov 13, 2026
fbd0146
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 13, 2026
3292a90
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 13, 2026
fc03a18
fix(torch): read gradient-bearing complex32/float8 tensors, store inf…
Nov 13, 2026
e169707
fix(tensor_conversions): honour module- and line-restricted error fil…
Nov 13, 2026
36efc60
Merge remote-tracking branch 'origin/main' into eywalker/itl-871-add-…
Nov 13, 2026
cef6829
fix(torch): hashing no longer pins the caller's storage; exact-type g…
Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,11 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
- The identifier is declared in one place, the extension type returned by
`get_arrow_extension_type()`, and read through `logical_types/identifier.py`
(`logical_type_identifier`, `LogicalTypeIdentifier.text`). Never put a logical type's bare
name into a hash input: use the identifier text.
name into a hash input: use the identifier text. Exception (developer, November 12, 2026):
the leading type text inside a library value hash (`numpy.ndarray`, `torch.Tensor`, pandas,
polars) is the extension name alone, because the metadata holds the ignore list, which must
not enter the value's preimage; the full identifier (name plus metadata) still enters every
type, schema and cell hash.
- Change a type's identifier when, and only when, an existing value would be stored or hashed
differently. It is not a package version. Built-in types use `version_metadata(N)`
(`{"version":N}`); a new built-in type must write one, never empty metadata.
Expand Down Expand Up @@ -683,10 +687,14 @@ data. The rules:
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
every execution path (sync and async).
Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base.
Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base.
What is stored for a subclass instance is decided by the logical type of the declared
class (developer, November 12, 2026, ITL-871): a plain user class is stored as the base
(the declared class's fields only; subclass-only fields are omitted and do not enter the
stored row hash), while the tensor type keeps the class (an `nn.Parameter` returned for
`-> torch.Tensor` is stored, read back and hashed as an `nn.Parameter`).
Structural classes additionally accept equivalent structural identities. Numeric
narrowing and unrelated container types require explicit conversion. Base storage
omits subclass-only fields, and those fields do not enter the stored row hash.
narrowing and unrelated container types require explicit conversion.

- Every output column takes the large-normalized form of its input type
(arrow_utils.normalize_to_large_types: string to large_string, binary to large_binary, list
Expand Down
16 changes: 12 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,11 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
- The identifier is declared in one place, the extension type returned by
`get_arrow_extension_type()`, and read through `logical_types/identifier.py`
(`logical_type_identifier`, `LogicalTypeIdentifier.text`). Never put a logical type's bare
name into a hash input: use the identifier text.
name into a hash input: use the identifier text. Exception (developer, November 12, 2026):
the leading type text inside a library value hash (`numpy.ndarray`, `torch.Tensor`, pandas,
polars) is the extension name alone, because the metadata holds the ignore list, which must
not enter the value's preimage; the full identifier (name plus metadata) still enters every
type, schema and cell hash.
- Change a type's identifier when, and only when, an existing value would be stored or hashed
differently. It is not a package version. Built-in types use `version_metadata(N)`
(`{"version":N}`); a new built-in type must write one, never empty metadata.
Expand Down Expand Up @@ -725,10 +729,14 @@ data. The rules:
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
every execution path (sync and async).
Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base.
Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base.
What is stored for a subclass instance is decided by the logical type of the declared
class (developer, November 12, 2026, ITL-871): a plain user class is stored as the base
(the declared class's fields only; subclass-only fields are omitted and do not enter the
stored row hash), while the tensor type keeps the class (an `nn.Parameter` returned for
`-> torch.Tensor` is stored, read back and hashed as an `nn.Parameter`).
Structural classes additionally accept equivalent structural identities. Numeric
narrowing and unrelated container types require explicit conversion. Base storage
omits subclass-only fields, and those fields do not enter the stored row hash.
narrowing and unrelated container types require explicit conversion.

- Every output column takes the large-normalized form of its input type
(`arrow_utils.normalize_to_large_types`: `string` to `large_string`, `binary` to `large_binary`, `list`
Expand Down
77 changes: 77 additions & 0 deletions changelog.d/ITL-871.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
### Added

#### PyTorch tensors as data values (ITL-871)

- **`torch.Tensor` is a value type**: a source column, a function pod parameter or return value,
stored, cached and re-run, sync and async, in memory and in Delta stores, and inside
`list[...]` (`Batch`, `GroupBy`, `MergeJoin`).
PyTorch is optional: support works whenever `torch` can be imported, however it was
installed; the new extra `orcapod[torch]` (`torch>=2.6`) installs it from PyPI. PyTorch 2.6
is the first version whose restricted loader reads `uint16`, `uint32` and `uint64`
tensors. Tested with PyTorch 2.15. The PyTorch build must support the installed NumPy, because a tensor's
elements are hashed through NumPy.
`import orcapod` never imports PyTorch: the type is set up on first use, and reading a
stored tensor column without PyTorch raises `orcapod.errors.MissingOptionalDependencyError`.
- **Storage is PyTorch's own serialization.** Extension type `orcapod.torch.tensor` over
`large_binary`: one cell holds what `torch.save` writes for one tensor. A cell is read with
`torch.load(map_location="cpu", weights_only=True)`, PyTorch's restricted loader, always
with both arguments explicit and with no unrestricted fallback. Stored and read back:
dtype, shape, elements, `requires_grad`, strides (transposed and `channels_last` layouts),
the lazy conjugate and negative bits, and the class (`torch.Tensor` or
`torch.nn.Parameter`). Not stored: the device (a tensor reads back on the CPU) and the
autograd history (`grad_fn`, `.grad`; a tensor reads back as a leaf).
- **Only the passed elements are stored.** A tensor that does not cover its underlying buffer
exactly (a slice, a step slice, an expanded view) is stored as
`value.clone(memory_format=torch.preserve_format)`, so the rest of the buffer is never
written; such a view has the clone's strides and its lazy bits are resolved. The caller's
tensor is never modified.
- **The hash is strict and defined by Orcapod**, never on the stored bytes.
`TorchTensorHandler` returns `ContentHash("torch_tensor_v1", D)`, where
`D = SHA-256(lp("orcapod.torch.tensor") || E || requires_grad || conj_bit || neg_bit || ndim
|| strides || lp(class name))` and `E` is the array value digest of the elements
(`docs/reference/hashing.md` §8). Dtype, shape and every element bit always count.
`requires_grad`, the memory layout, the lazy conjugate and negative bits and the class count
by default. Device and autograd history never count, because storage does not keep them. A
tensor and an ndarray with equal elements never hash the same, also inside an untyped
Python structure.
- **What the hash ignores is an option of the type.** The five optional properties
(`requires_grad`, `memory_layout`, `conj_bit`, `neg_bit`, `subclass`) can be named in the
type's ignore list: `hash_ignore` in `orcapod/contexts/logical_types.json`, empty by
default. The list is in the extension metadata (`{"ignore":[],"version":1}`), so it is part
of the logical type identifier, and a data context refuses to read a tensor column stored
under another list. An unknown option name raises when a tensor is first used. There is no
user API for the list in v0.3.
- **Dtypes.** The NumPy-shared dtypes, plus `bfloat16`, `complex32` (real then imaginary
`float16`) and `float8_e4m3fn`, `float8_e5m2`, `float8_e4m3fnuz`, `float8_e5m2fnuz`,
`float8_e8m0fnu`, hashed as raw bits under the PyTorch dtype name.
- **`orcapod.errors.ValueConversionWarning`** (a `UserWarning`) reports a tensor on a non-CPU
device or with autograd history (a `grad_fn`), which is stored as a CPU leaf tensor. It
names the function and output (or the source and column), what was converted and how to
avoid it (`value.detach().cpu()`). A function warns once per output and conversion, however
many rows it processes; `DictSource` and `ListSource` warn once per column with the count.
A leaf tensor with `requires_grad=True` and an `nn.Parameter` do not warn. After
`warnings.simplefilter("error", ValueConversionWarning)` every conversion raises (Python's
`-W` option cannot name the class: it is read before installed packages are importable).
- **Refused at write time**, naming the column and reason: sparse, mkldnn, quantized, nested
and meta tensors, named tensors, other dtypes (sub-byte and packed types, `bcomplex32`),
and subclasses other than `torch.nn.Parameter` (the restricted loader cannot read them
back). Such a subclass is not hashed as a tensor either: a subclass with its own logical
type needs its own value handler.
- **Declared outputs.** A pod declared `-> torch.Tensor` must return a tensor; an ndarray or a
number raises `OutputTypeError`. A `torch.nn.Parameter` is accepted and is stored, read
back and hashed as an `nn.Parameter`. A pod declared with a subclass must return that
subclass.
- **Registration on first use.** `LogicalTypeRegistry(deferred_logical_types=...)` and
`PythonTypeHandlerRegistry.register_deferred(...)` name a value type by its qualified class
name and extension name and load it on first use, with the configuration (the ignore list)
given in the entry. The tensor entries are in `orcapod/contexts/logical_types.json`
(`deferred_logical_types`, `deferred_value_handlers`); the `std:v1:default` data context
file is unchanged. No existing hash, schema or table path changes.

### Fixed

- **A declared output class with a logical type requires an instance of it (ITL-895).** The
output check required `isinstance` only for `np.ndarray`; for any other class with a
registered logical type a wrong value was caught only if its conversion failed, so a plain
tensor returned for `-> torch.nn.Parameter` was accepted. Such a value now raises
`OutputTypeError` naming the function, the output, the declared type and the returned type.
38 changes: 38 additions & 0 deletions design_issues/ITL-871-TT1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
## `src/orcapod/logical_types/torch_type.py` — PyTorch tensors

### TT1 — `torch.Tensor` could not be a data value
**Status:** resolved
**Severity:** medium
**Issue:** ITL-871 (design: the module docstring of `logical_types/torch_type.py`; history: `superpowers/specs/2026-11-11-torch-tensor-v0.3.md`)

A pod annotated with `torch.Tensor` could not be defined, and the semantic hasher refused
tensors. Users converted at every pod boundary by hand.

**Fix:** `LogicalTorchTensor` (`orcapod.torch.tensor`, `large_binary`) stores a tensor with
PyTorch's own serialization (`torch.save`; read with
`torch.load(map_location="cpu", weights_only=True)`, never unrestricted), and
`TorchTensorHandler` hashes it under `torch_tensor_v1` with a strict, Orcapod-defined hash:
the array digest of the elements, then `requires_grad`, the lazy conjugate and negative bits,
the strides and the class. Those five can be ignored through the type's ignore list (the
shared mechanism of ITL-981, `logical_types/ignore_list.py`). Both classes register on first
use through deferred entries in `contexts/logical_types.json`, so `import orcapod` never
imports PyTorch, and support depends only on `torch` being importable.

Developer decisions (November 12, 2026): storage and hash are decided separately; a property
storage cannot restore (device, autograd history) never counts in the hash; a tensor that does
not cover its buffer is stored as `clone(memory_format=torch.preserve_format)` and its
optional properties are hashed after that clone; extension name `orcapod.torch.tensor`, which
is also the leading text of the hash; label `torch_tensor_v1`; minimum `torch>=2.6`
(`uint16`/`uint32`/`uint64` cannot be loaded with `weights_only=True` before it). Earlier
decisions that stand (November 11, 2026): the dtype list (the NumPy-shared ones plus
`bfloat16`, `complex32` and the float8 family as raw bits); `ValueConversionWarning`, issued
once per function output or source column and conversion; the convenience extra
`orcapod[torch]`. The dev dependency group pulls the extra, with CPU-only wheels on Linux, so
the required CI always runs the torch tests. A guard test fails if torch is missing under CI.

Known limits: within one in-memory run a downstream pod receives the very tensor the upstream
pod returned (device and autograd history included); only stored or read-back values are in
the stored form. NumPy arrays behave the same. A lazy negative view made by public PyTorch
code (`x.conj().imag`) never covers its buffer, so its `neg_bit` is always stored and hashed
as 0. A non-default ignore list works at the type converter and hasher level only: datagrams
find their data context by key (ITL-981).
Loading
Loading