Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
c3c0cd1
docs(spec): value-based array format for numpy.ndarray in v0.3 (ITL-888)
Nov 11, 2026
506c13e
feat(arrays): shared array storage struct and value digest v1 (ITL-888)
Nov 11, 2026
eca0432
test(arrays): public-interface array scaffolding across operators, st…
Nov 11, 2026
6d42164
docs(spec): nullable struct children (F1), MergeJoin array ordering (…
Nov 11, 2026
be14416
feat(arrays)!: value-based array format for numpy.ndarray (ITL-888, I…
Nov 11, 2026
e9fe738
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
8111672
test(hashing): update golden array entries for the v0.3 array format …
Nov 11, 2026
44bc8c9
docs(spec): record the developer's decisions for the array format (IT…
Nov 11, 2026
0c63948
docs(hashing): state the <type>_v<N> value hash label convention (ITL…
Nov 11, 2026
ac68bcd
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
50b4183
fix(arrays): address CodeRabbit review; Pick raises InputValidationEr…
Nov 11, 2026
9802a9a
docs(spec,plan): golden regeneration gates; astype copy=False (ITL-888)
Nov 11, 2026
a3bc9fe
docs(plan): resume notes after restart (ITL-888)
Nov 11, 2026
17aa217
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
25b240a
test(arrays): SemiJoin with an array column passes after ITL-847 (ITL…
Nov 11, 2026
ad7a3de
fix(arrays): address CodeRabbit round 4 docs and fixture findings (IT…
Nov 11, 2026
ad09471
fix(arrays): order lists of arrays in MergeJoin; reject malformed str…
Nov 11, 2026
121dd1f
feat(arrays)!: structured field order is part of an array's identity …
Nov 11, 2026
619cffb
docs(arrays): structured tag example in declared order; review nits (…
Nov 11, 2026
f52bc63
chore(merge): merge origin/main after ITL-878 (#287) into ITL-888
Nov 11, 2026
68b5d16
docs(arrays): state the array-format decisions as decided (ITL-888)
Nov 11, 2026
75f8959
chore(merge): merge origin/main after ITL-821 (#296) into ITL-888
Nov 11, 2026
32872cd
fix(arrays): reject malformed structured tag entries; ignore subclass…
Nov 11, 2026
d7464f8
docs(arrays): docstrings for MergeJoin's list ordering key and the PR…
Nov 11, 2026
1cf5eed
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
6ce706c
docs(arrays): plan's storage-type test matches the nullable-child for…
Nov 11, 2026
1f3ac3a
fix(arrays): state the path-move scope precisely; pin it on the v0.2 …
Nov 11, 2026
1079d98
test(arrays): detect result tables by path component, not by '/' (ITL…
Nov 11, 2026
5ed66e4
test(arrays): merged member lists must match distinct expected lists …
Nov 11, 2026
f55b858
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
89c3b81
Merge remote-tracking branch 'origin/main' into eywalker/itl-888-impl…
Nov 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,49 @@

### Breaking changes and migration

#### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883)

**The array format and every array hash changed in v0.3.** A store written by v0.2 that holds
`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays
computes again into new tables, and its v0.2 results are no longer returned. Nothing written by
v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for
v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889).

- **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata
`{"version":1}`) over `struct<dtype: large_string, shape: large_list<int64>,
data: large_binary>`: an Orcapod dtype tag, the shape, and the elements in C order,
little-endian, packed. It replaces the `.npy` file under `numpy.ndarray`; that name is no
longer registered or written.
- **Identity.** An array is its dtype, shape and element bits. A structured dtype's field
names, types and order are part of the dtype, at every nesting depth. Memory layout, byte
order, structured offsets and padding, and subclass are not part of it. Arrays read back
canonical: C order, little-endian (`>f4` reads back `<f4`), structured fields packed in
their declared order, as plain `np.ndarray`.
- **Hash.** `NumpyArrayHandler` returns `ContentHash("ndarray_v1", digest)`. The digest is
SHA-256 over a length-prefixed encoding of the stored row (`docs/reference/hashing.md`,
section 8), and the label replaces the bare `sha256`. Because the logical type name is now
`orcapod.ndarray`, the type, schema and pipeline hashes of anything with an array column
change too, and so do the table paths of those nodes. Pipeline hashes are chained, so the
pipeline and operator tables of every node downstream of an array column move as well,
even when it has no array column itself. A pod whose inputs and outputs hold no array
keeps its function result table, so it reuses its v0.2 results without computing again.
Nodes with no array column upstream do not move.
- **Refused at write time** with `orcapod.errors.UnsupportedValueError` (a `ValueError`), naming
the column and dtype: masked arrays (the mask used to be dropped silently), object dtypes,
NumPy 2 `StringDType` (used to fail with a misleading pickle error), `longdouble` and
`clongdouble`, datetimes and timedeltas without a unit, and structured dtypes with titles
(whether to accept these last three kinds later is tracked in ITL-911).
- **Function pods.** A pod declared `-> np.ndarray` that returns something other than an array,
such as a NumPy scalar, now raises `TypeError` naming the output.
- **MergeJoin** orders merged arrays by their stored `(dtype tag, shape, data)`, and lists of
arrays (an array column after `GroupBy` or `Batch`) by their elements' keys, in order.
- **Pick** on a column whose logical type has no keyed access (an array, a dataclass, a
pydantic model) raises `InputValidationError` naming the column, instead of
`NotImplementedError` (ITL-883).
- `LogicalTypeRegistry.get_by_python_type` now picks the most specific registered class along
the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its
subclasses, whatever the registration order.

#### A changed column type is refused when appending to a Delta table (ITL-884)

Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows'
Expand Down
32 changes: 32 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2380,6 +2380,8 @@ newcomer. Re-checked on `b197544e` (v0.2.0 + #291):
column ... because it's a binary field` lines on every Delta write still appear.
**Open:** awaiting the developer's choice of how to silence them.

---

## `src/orcapod/core/operators/filters.py` — `PolarsFilter` identity and nullability (ITL-828)

### PF1 — An expression predicate could not be hashed, and a filter broke a join inside a job
Expand Down Expand Up @@ -2410,3 +2412,33 @@ source row.
2026). A filtered row's data hash now equals its source row's. The pinned
`pipeline:polars_filter` row data hashes moved (approved); they are the cache keys of any
function pod downstream of a filter, so a 0.2 store recomputes those results once.

---

## `src/orcapod/logical_types/numpy_type.py`, `src/orcapod/logical_types/array_format.py` — array format

### AR1 — `numpy.ndarray` stored as a `.npy` file and hashed by file bytes
**Status:** in progress
**Severity:** high
**Issue:** ITL-888 (absorbs ITL-883; design ITL-874)

Arrays were stored as the exact `np.save` file in a `large_binary` column under extension name
`numpy.ndarray`, and hashed as SHA-256 of that file under the bare label `sha256`. The hash
followed the file, not the value: equal arrays in C and Fortran order, in little- and big-endian
byte order, or with structured fields at other offsets hashed differently. Masked arrays lost
their mask silently. `StringDType` failed with a misleading pickle error. A pod declared
`-> np.ndarray` could return a NumPy scalar unchecked.

v0.3 moves to the value-based format of `superpowers/specs/2026-11-10-array-format-v0.3.md`:
extension `orcapod.ndarray` over `struct<dtype, shape, data>` (little-endian, C order), hashed
by value under the label `ndarray_v1` (developer's decisions, November 10 and 11, 2026). Old
data is neither detected nor migrated in v0.3; migration is deferred to ITL-889. Accepting
`longdouble`/`clongdouble`, unit-less datetimes and titled structured dtypes, refused in v0.3,
is reconsidered in ITL-911. The new logical type name moves the table paths of nodes with an
array column and the pipeline and operator tables of every node downstream of one (pipeline
hashes are Merkle-chained); an array-free pod downstream keeps its result table and reuses its
results.

Known compromise: the struct's children are nullable at the Arrow level, because Polars drops
nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null
instead. Tighten under a new storage version once nested nullability survives Polars.
162 changes: 162 additions & 0 deletions docs/concepts/arrays.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# NumPy Arrays

A `numpy.ndarray` can be a data value like any other: a source column, a function pod
parameter or a return value. Orcapod stores each array by its **value**, its dtype, its shape
and its elements. How the array happens to sit in memory is not stored. The hash is defined on
that value, so two arrays that hold the same values hash the same, however each was laid out.

```python
import numpy as np

from orcapod import function_pod
from orcapod.sources import DictSource

@function_pod(output_keys="scaled")
def scale(signal: np.ndarray) -> np.ndarray:
return signal * 2

source = DictSource(
data=[{"trial": "t1", "signal": np.arange(6, dtype=np.float32).reshape(2, 3)}],
tag_columns=["trial"],
source_id="trials",
)
for tag, data in scale.pod(source).iter_data():
print(tag["trial"], data["scaled"].dtype, data["scaled"].shape)
# t1 float32 (2, 3)
```

!!! warning "v0.3 changed the array format"
Orcapod v0.2 stored arrays as `.npy` files under the extension name `numpy.ndarray` and
hashed the file. v0.3 replaces both, as described below, and does not migrate older
stores. Under v0.3, every node that handles arrays computes again into new tables, and its
v0.2 results are no longer returned. They stay on disk, readable with v0.2. Nodes
downstream of an array column get new pipeline and operator tables too; a function pod
among them that takes and returns no array reuses its stored results. Start v0.3
pipelines with a fresh store, or keep using v0.2 for an existing one. A migration is
planned after v0.3 (ITL-889).

## What counts as the same array

Two arrays are the same value when they have the same dtype, the same shape and the same
element bits.

- **Memory layout does not count.** C order, Fortran order, strided views and transposed views
of equal values are the same array. An array always reads back in C order.
- **Byte order does not count.** Arrays are stored little-endian; a big-endian `>f4` array
reads back as `<f4` with equal values.
- **Structured field order counts**, as do field names and field types, at every nesting
depth: `[("a", "i4"), ("b", "f8")]` and `[("b", "f8"), ("a", "i4")]` are different arrays,
as they are to NumPy's own dtype equality. Offsets, padding and alignment do not count:
fields are stored packed in declared order, so an aligned `[("b", "f8"), ("a", "i2")]`
reads back as the packed `[("b", "<f8"), ("a", "<i2")]`.
- **Dtype counts**, including width: `int32` and `int64` arrays with equal values differ.
- **Shape counts**, including empty shapes: `(0, 3)` and `(3, 0)` differ, and so do a 0-d
array and a one-element 1-d array.
- **Every bit counts**: `0.0` and `-0.0` differ, and so do NaNs with different payloads.
- **Subclasses are plain arrays by default.** `np.matrix`, `np.recarray`, `np.memmap` and
other subclasses use the array format: they are stored as their values and read back as
plain `np.ndarray`. The exception is a subclass with its own registered logical type, which
is used for that subclass and its own subclasses instead.

```python
from orcapod.contexts import get_default_context

hasher = get_default_context().semantic_hasher
a = np.arange(6, dtype=np.float32).reshape(2, 3)

print(hasher.hash_object(a) == hasher.hash_object(np.asfortranarray(a)))
# True
print(hasher.hash_object(a) == hasher.hash_object(a.astype(">f4")))
# True
print(hasher.hash_object(a) == hasher.hash_object(a.astype(np.float64)))
# False
ab = np.zeros(1, dtype=[("a", "i4"), ("b", "f8")])
ba = np.zeros(1, dtype=[("b", "f8"), ("a", "i4")])
print(hasher.hash_object(ab) == hasher.hash_object(ba))
# False
```

## What cannot be stored

Some arrays are refused when they are written, with a `ValueError` (an
`orcapod.errors.UnsupportedValueError`) that names the column and the dtype:

| Array | Why | What to do |
|---|---|---|
| Masked arrays (`np.ma.MaskedArray`) | extended data: the mask would be lost | store the data and the mask as two arrays |
| `dtype=object`, or a structured dtype with an object field | extended data: would need pickling | convert to a fixed dtype |
| NumPy 2 `StringDType` | extended data: variable width | use a fixed-width string dtype such as `U10` |
| `longdouble`, `clongdouble` | width and format differ by platform | `.astype(np.float64)` or `.astype(np.complex128)` |
| `datetime64` or `timedelta64` without a unit | the integer has no defined meaning | give a unit, e.g. `datetime64[ns]` |
| Structured dtypes with field titles | titles are aliases of fields | drop the titles |

Extended data may be supported later, stored as several parts rather than as one array.
Whether `longdouble` and `clongdouble`, unit-less datetimes and timedeltas, and structured
dtypes with titles should be accepted in a later release is tracked in ITL-911; v0.3 refuses
them.

```python
try:
DictSource(
data=[{"trial": "t1", "signal": np.ma.array([1.0, 2.0], mask=[False, True])}],
tag_columns=["trial"],
)
except ValueError as exc:
print(type(exc).__name__)
# UnsupportedValueError
```

A function pod that declares `-> np.ndarray` must return an array. A NumPy scalar, for
example `x * 2` on a 0-d array, raises `TypeError` that names the output. Return
`np.asarray(...)` instead.

## How an array is stored

An array column has the Arrow extension type `orcapod.ndarray`, with extension metadata
`{"version":1}`. Its storage is one struct per row:

| Field | Arrow type | Holds |
|---|---|---|
| `dtype` | `large_string` | the Orcapod dtype tag (below) |
| `shape` | `large_list<int64>` | the dimensions, outermost first; empty for a 0-d array |
| `data` | `large_binary` | the elements in C order, little-endian, packed |

The struct's children are nullable in the Arrow schema only because Polars drops nested
non-null flags. A stored array never has a null child. A null array is a null struct, allowed
when the column is annotated `np.ndarray | None`.

```python
table = source.as_table()
print(table.schema.field("signal").type.extension_name)
# orcapod.ndarray
print(table.column("signal").chunk(0).storage[0].as_py()["dtype"])
# float32
```

### Dtype tags

The tag names the element type without a byte order. For a structured dtype, the tag is a JSON
list with one `[name, tag]` entry per field, or `[name, tag, [dims]]` for a subarray field,
in declared field order. It is written with no whitespace and with non-ASCII characters escaped
(`json.dumps(..., separators=(",", ":"), ensure_ascii=True)`).

| NumPy | Tag | Element bytes |
|---|---|---|
| `bool` | `bool` | one byte, 0 or 1 |
| `int8` ... `int64`, `uint8` ... `uint64` | `int8` ... `uint64` | two's complement or unsigned, little-endian |
| `float16`, `float32`, `float64` | same names | IEEE 754, little-endian |
| `complex64`, `complex128` | same names | real part, then imaginary part |
| `datetime64[ns]`, `datetime64[10ms]`, ... | `datetime64[ns]`, `datetime64[10ms]`, ... | `int64`, little-endian (NaT is the minimum) |
| `timedelta64[s]`, ... | `timedelta64[s]`, ... | `int64`, little-endian |
| `U5` | `ucs4[5]` | five UCS-4 code units, each `uint32` little-endian, NUL-padded |
| `S3` | `bytes[3]` | three raw bytes, NUL-padded |
| `V4` | `void[4]` | four raw bytes |
| `[("b", "f8"), ("a", "i4", (2,))]` | `[["b","float64"],["a","int32",[2]]]` | fields packed in tag order |

## How an array is hashed

The hash of an array is the version-1 array digest of its stored row, labelled `ndarray_v1`.
The byte-level definition, and a reference implementation that does not use Orcapod, are in
the [Hashing Reference](../reference/hashing.md#8-array-value-digest). A future PyTorch tensor
type will share the digest under its own label, so an array and a tensor with equal values
have different hashes.
73 changes: 73 additions & 0 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -545,3 +545,76 @@ in any data-level preimage; they serve logging, observability, and result inspec
| **Known exclusions** | **Not a content hash.** Two datagrams with identical content have different UUIDs. Not used in any hash preimage. Serves as an object identity token, not a content fingerprint. |

---

## 8. Array Value Digest

**Value hash labels.** A type whose values are hashed by a dedicated handler labels its hash
`<type>_v<N>`, for example `ndarray_v1` (and later `tensor_v1`). The label is the
`ContentHash.method` and appears in every token built from the hash. The bytes produced under
an existing label never change. A new definition gets a new label, so a hash always says
which definition produced it.

A `numpy.ndarray` value is hashed by value (since v0.3, ITL-888). The semantic hasher's
`NumpyArrayHandler` returns `ContentHash(method="ndarray_v1", digest=D)`, where `D` is the
version-1 array digest of the array's stored row. Inside a table hash, an array cell becomes
the token `b"orcapod:ndarray::ndarray_v1:" + D`: the extension name, then the label and the
digest. The stored row, the dtype tags and the canonical form are described in
[NumPy Arrays](../concepts/arrays.md).

For a stored row with dtype tag `T`, shape `(d0, ..., dn-1)` and data bytes `B`:

```
P = u64le(len(utf8(T))) || utf8(T)
|| u64le(n) || i64le(d0) || ... || i64le(dn-1)
|| u64le(len(B)) || B
D = SHA-256(P)
```

`u64le` and `i64le` are 8-byte little-endian unsigned and signed integers. Every field is
length-prefixed, so two different rows never share a preimage. `D` depends only on the stored
row, so it can be recomputed from a stored table by any SHA-256 implementation. The label
`ndarray_v1` names this definition; a different definition gets a different label. A
`torch.Tensor` type is to share the digest under its own label.

| Variation | Same `D`? |
|---|---|
| C or Fortran order, strides, views | yes |
| Little- or big-endian byte order | yes |
| Structured offsets, alignment padding | yes |
| `np.matrix`, `np.recarray`, `np.memmap` against the plain array | yes |
| Dtype (including width), shape, any element bit, structured field names or field order | no |

The reference implementation below uses only the standard library, and the vectors are pinned:

```python
import hashlib
import struct

def array_digest_v1(tag: str, shape: tuple[int, ...], data: bytes) -> str:
t = tag.encode("utf-8")
p = struct.pack("<Q", len(t)) + t + struct.pack("<Q", len(shape))
p += b"".join(struct.pack("<q", d) for d in shape)
p += struct.pack("<Q", len(data)) + data
return hashlib.sha256(p).hexdigest()

print(array_digest_v1("float32", (2, 3), struct.pack("<6f", 0, 1, 2, 3, 4, 5)))
# 318f6e503e8064e89ce6039453fce709f8e01f60c501af2b63bbbe9cf8fbf90e
print(array_digest_v1("float64", (), struct.pack("<d", 2.5)))
# 47f182d862f5dd597d88836deddaed609ed94490827302e75e8364e877633cae
print(array_digest_v1("int8", (0, 3), b""))
# 54165f92526b48bf9bae265168aa7020b135a9973d372d9ceaa9ae92f611068a
```

The same values through Orcapod:

```python
import numpy as np

from orcapod.contexts import get_default_context

hasher = get_default_context().semantic_hasher
print(hasher.hash_object(np.arange(6, dtype=np.float32).reshape(2, 3)).to_string())
# ndarray_v1:318f6e503e8064e89ce6039453fce709f8e01f60c501af2b63bbbe9cf8fbf90e
print(hasher.hash_object(np.array(2.5)).to_string())
# ndarray_v1:47f182d862f5dd597d88836deddaed609ed94490827302e75e8364e877633cae
```
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ nav:
- Streams: concepts/streams.md
- Operators: concepts/operators.md
- Function Pods: concepts/function-pods.md
- NumPy Arrays: concepts/arrays.md
- Identity & Hashing: concepts/identity.md
- File Hash Caching: concepts/file-hash-caching.md
- API Reference:
Expand Down
3 changes: 2 additions & 1 deletion src/orcapod/contexts/data/v0.1.json
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,8 @@
"Wired file_hasher reference into BasicDirectoryHasher so op.Directory hashing uses the same FileContentHasherProtocol as op.File hashing; required file_hasher constructor arg enforces explicit injection",
"Moved SpikeInterface types (LogicalSIRecording, LogicalSISorting, LogicalSIMotion, LogicalSISortingAnalyzer) and their handlers to the standalone orcapod-extension-spikeinterface package; removed all SI _optional entries from this spec and SI deps from pyproject.toml; introduced OrcapodExtension protocol and op.register_extension() for normalized extension registration (ITL-473)",
"Registered SchemaHandler for orcapod.types.Schema in the context registry; moved handler dispatch before _is_structure check in hash_object so SchemaHandler takes priority over _expand_mapping for Schema objects (ITL-639)",
"Registered PolarsExprHandler for polars.Expr: hashes an expression as ContentHash('polars_expr_v1', sha256 of the canonical JSON of an Orcapod-owned expression tree parsed strictly from Expr.meta.serialize(format='json')); expressions outside the supported subset (Python functions, multi-column selectors, aggregations, alias, when/then, non-finite float literals, pl.Series literals) raise ValueError naming the expression (ITL-828)"
"Registered PolarsExprHandler for polars.Expr: hashes an expression as ContentHash('polars_expr_v1', sha256 of the canonical JSON of an Orcapod-owned expression tree parsed strictly from Expr.meta.serialize(format='json')); expressions outside the supported subset (Python functions, multi-column selectors, aggregations, alias, when/then, non-finite float literals, pl.Series literals) raise ValueError naming the expression (ITL-828)",
"Hard cut (v0.3): numpy.ndarray moves to the value-based array format. LogicalNumpyArray stores extension orcapod.ndarray over struct<dtype, shape, data> (Orcapod dtype tag, shape, little-endian C-order packed bytes; extension metadata {\"version\":1}) and NumpyArrayHandler hashes by value with label ndarray_v1 (array_digest_v1 over the stored row). The old extension name numpy.ndarray (.npy bytes, label sha256) is no longer registered or written; v0.2 stores holding arrays are not migrated (ITL-888)"
]
}
}
Loading
Loading