Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
431d179
test(arrays): failing tests, design note and plan for the strict arra…
Nov 12, 2026
3900ca6
docs(specs): extension name numpy.ndarray and the v0.2 name collision…
Nov 12, 2026
962baa6
docs(specs): scope the Arrow schema hash question (ITL-981)
Nov 12, 2026
ab2684f
docs(plans): record coordinator decisions while waiting for sign-off …
Nov 12, 2026
592274a
docs(specs): final extension name orcapod.numpy.ndarray; drop the for…
Nov 12, 2026
1e5e860
Merge remote-tracking branch 'origin/main' into eywalker/itl-981-libr…
Nov 12, 2026
64c63bf
feat(logical_types): ignore-list mechanism for library value types (I…
Nov 12, 2026
e299455
feat(arrays)!: store numpy arrays as .npy and hash them strictly (ITL…
Nov 12, 2026
2e0109a
test(arrays): rewrite the value-format tests for .npy storage and the…
Nov 12, 2026
36507ef
test(logical_types): update array type unit tests for orcapod.numpy.n…
Nov 12, 2026
dc911de
feat(hashing)!: array golden values move once; hash_schema uses the f…
Nov 12, 2026
9ab64d7
docs: arrays are stored as .npy and hashed strictly; hashing referenc…
Nov 12, 2026
261fce2
docs: changelog fragment, design-issue entry and agent rules for libr…
Nov 12, 2026
ce7a408
fix(hashing): type-clean hash_schema field rule and tests (ITL-981)
Nov 12, 2026
9b0a60b
docs(plans): record sign-off and progress (ITL-981)
Nov 12, 2026
197b35a
fix(logical_types): lock the per-list extension type caches; wrap des…
Nov 12, 2026
598acd9
docs: a non-default ignore list is not usable end to end in v0.3 (ITL…
Nov 12, 2026
2d34857
test(arrays): list[np.ndarray] pod outputs on every store and runner;…
Nov 12, 2026
960d6fd
test(arrays): compare structured arrays field by field, not their pad…
Nov 12, 2026
d19fd8b
fix(arrays): address review: zero struct padding at write, check the …
Nov 12, 2026
b6de1a4
fix(arrays): padding of a structured array read back from a store is …
Nov 12, 2026
456f667
Merge remote-tracking branch 'origin/main' into eywalker/itl-981-libr…
Nov 13, 2026
5c04d22
Merge remote-tracking branch 'origin/main' into eywalker/itl-981-libr…
Nov 13, 2026
21dbc84
test(arrays): list[np.ndarray] outputs pass on Delta async now that I…
Nov 13, 2026
fa5cd56
test(arrays): Batch and GroupBy over an array column on every store a…
Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,37 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
never fall back to the storage type. List an older value in `also_reads` only when the
current code decodes that layout.

## Library value types: storage, strict hash, ignore list

Developer, November 12, 2026 (ITL-981); see `docs/concepts/arrays.md` and
`docs/reference/hashing.md` section 8.

- Storage and hashing are decided separately. A value of a library type (`numpy.ndarray`,
later tensors and data frames) is stored with the library's own serialization and reads
back as given. Its hash is defined by Orcapod on the content: never hash the stored bytes.
- The hash is strict by default: every property of a value that storage keeps is part of
its identity. A property storage does not restore never counts and has no option name,
because `hash(read(write(v)))` must equal `hash(v)`.
- What the hash may ignore is a closed set of property names per type (`OPTIONS`). The names
a data context ignores are declared once, in `hash_ignore` of
`contexts/logical_types.json`, and referred to as `hash_ignore:<extension name>` by the
logical type entry and the value handler entry.
- The list is in the extension metadata, `{"ignore":[...],"version":N}` (sorted names,
sorted keys, `ignore` always present), written only by `ignore_list.ignore_metadata`. Two
lists are two logical types. Never put the list into a hash preimage: an ignored property's
field is always written, with its canonical value.
- Build such a type from `logical_types/ignore_list.py` (`IgnoreListLogicalType`,
`IgnoreListHandler`). A bare value is hashed with the handler's configured list; a cell
with the list of its own column (`SemanticHashingVisitor` calls `hash_ignoring`). A data
context reads only columns stored under its own list; anything else raises
`LogicalTypeIdentifierError`.
- Extension names of library value types are lower case `orcapod.<library>.<type>`
(`orcapod.numpy.ndarray`). The leading type text of a value hash is the extension name.
- `numpy.ndarray`: one `.npy` file per cell (`numpy.lib.format`, format 3.0,
`allow_pickle=False` on write and read; never raise `max_header_size`). Options:
`memory_order`, `byte_order`, `struct_layout`. Where stored arrays must be ordered, order
them by value hash (`storage_sort_key`), never by stored bytes.

## User classes: exact and structural

A class a user defines is one of two kinds (developer, November 12, 2026, ITL-972; see
Expand Down
31 changes: 31 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,37 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
never fall back to the storage type. List an older value in `also_reads` only when the
current code decodes that layout.

## Library value types: storage, strict hash, ignore list

Developer, November 12, 2026 (ITL-981); see `docs/concepts/arrays.md` and
`docs/reference/hashing.md` section 8.

- Storage and hashing are decided separately. A value of a library type (`numpy.ndarray`,
later tensors and data frames) is stored with the library's own serialization and reads
back as given. Its hash is defined by Orcapod on the content: never hash the stored bytes.
- The hash is strict by default: every property of a value that storage keeps is part of
its identity. A property storage does not restore never counts and has no option name,
because `hash(read(write(v)))` must equal `hash(v)`.
- What the hash may ignore is a closed set of property names per type (`OPTIONS`). The names
a data context ignores are declared once, in `hash_ignore` of
`contexts/logical_types.json`, and referred to as `hash_ignore:<extension name>` by the
logical type entry and the value handler entry.
- The list is in the extension metadata, `{"ignore":[...],"version":N}` (sorted names,
sorted keys, `ignore` always present), written only by `ignore_list.ignore_metadata`. Two
lists are two logical types. Never put the list into a hash preimage: an ignored property's
field is always written, with its canonical value.
- Build such a type from `logical_types/ignore_list.py` (`IgnoreListLogicalType`,
`IgnoreListHandler`). A bare value is hashed with the handler's configured list; a cell
with the list of its own column (`SemanticHashingVisitor` calls `hash_ignoring`). A data
context reads only columns stored under its own list; anything else raises
`LogicalTypeIdentifierError`.
- Extension names of library value types are lower case `orcapod.<library>.<type>`
(`orcapod.numpy.ndarray`). The leading type text of a value hash is the extension name.
- `numpy.ndarray`: one `.npy` file per cell (`numpy.lib.format`, format 3.0,
`allow_pickle=False` on write and read; never raise `max_header_size`). Options:
`memory_order`, `byte_order`, `struct_layout`. Where stored arrays must be ordered, order
them by value hash (`storage_sort_key`), never by stored bytes.

## User classes: exact and structural

A class a user defines is one of two kinds (developer, November 12, 2026, ITL-972; see
Expand Down
74 changes: 74 additions & 0 deletions changelog.d/ITL-981.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
### Breaking changes and migration

#### NumPy arrays: stored as `.npy`, hashed strictly, with a per-type ignore list (ITL-981)

**This replaces the array storage and the array hash described under "NumPy arrays: new
storage format and new hashes" (ITL-888).** That format (the extension type
`orcapod.ndarray` over a `{dtype, shape, data}` struct, with a hash that ignored memory
layout and byte order) was never released; v0.3 ships the one described here. What that
entry says about v0.2 stores still holds: they are not migrated and not read.

The developer decided on November 12, 2026 that storage and hashing are separate, that the
hash is strict by default, and that what the hash ignores is configured per type.

- **Storage.** An array column is the Arrow extension type `orcapod.numpy.ndarray` over
`large_binary`. Each cell is one `.npy` file written by NumPy
(`numpy.lib.format.write_array`, format version 3.0, `allow_pickle=False`) and read with
`allow_pickle=False`. An array reads back as it was given: dtype including byte order,
shape, element bits, C or Fortran memory order and structured field offsets and item size
are kept. Not kept: strides other than C or Fortran order (a step slice is stored
C-contiguous), the subclass (read back as plain `np.ndarray`), the `writeable` flag, dtype
`metadata` and the `align` flag of a structured dtype. Padding bytes between structured
fields are not part of the value and are written as zero, so residual process memory
never reaches a store. A cell that does not hold exactly the element bytes its `.npy`
header declares is refused on read.
- **Hash.** The hash is defined on the value, never on the stored bytes, and keeps the label
`ndarray_v1`. It is `SHA-256(lp("orcapod.numpy.ndarray") || E || memory order || byte
order || structured layout)`, where `E` is the element digest of dtype, shape and element
bits (`docs/reference/hashing.md`, section 8, with a reference implementation and pinned
vectors). By default a C-ordered and a Fortran-ordered array, a little- and a big-endian
array, and a packed and an aligned structured array with equal values now hash
**differently**. What storage does not keep never changes the hash, so a value read back
from a store hashes like the value written.
- **Ignore list.** The three properties `memory_order`, `byte_order` and `struct_layout`
can be ignored. The list of ignored names is set per type in the data context
(`hash_ignore` in `contexts/logical_types.json`; default: empty) and is written into the
extension metadata, `{"ignore":[...],"version":1}`. Two lists are two logical types:
different schema hashes, no shared type in a join, and a data context refuses to read a
column stored under another list (`LogicalTypeIdentifierError`, naming the column and both
lists). A bare value is hashed with the data context's list, a cell with the list of its
own column. v0.3 has no user API to change the list.
- **Newly refused at write** (`UnsupportedValueError`, naming the column and the dtype): a
structured dtype whose fields overlap or are not in offset order (`.npy` has no form for
it; the previous format accepted it), and a dtype whose `.npy` header exceeds 10,000 bytes
(several hundred fields). Masked arrays, object dtypes, `StringDType`, `longdouble`,
unit-less datetimes and field titles are refused as before.
- **`MergeJoin`** orders the arrays it merges by their value hash (it used the stored dtype
tag, shape and bytes).
- **What moves.** Every array value hash, and with the new extension name and metadata the
type, schema and pipeline hashes of anything with an array column and the system tag
column names built from those schema hashes. Nothing released is affected: v0.3 is not
tagged, and v0.2 stores are not read by v0.3.

### Added

- **The ignore-list mechanism for library value types** (`orcapod.logical_types.ignore_list`):
`IgnoreListLogicalType` and `IgnoreListHandler`, the extension metadata shape
`{"ignore":[...],"version":N}` (`ignore_metadata`, `parse_ignore_metadata`), and the
`hash_ignore` object of the logical type registration file. `numpy.ndarray` is the first
type built on it. (ITL-981)

### Changed

- **`StarfixArrowHasher.hash_schema` hashes a field under the same rule as `hash_table`.** A
logical type with a value handler is hashed as a `large_binary` field carrying the type's
identifier, so its storage type is no longer part of that hash. No library code calls this
method, and no stored or pinned value moves. (ITL-981)

### Known limitations

- **A non-default ignore list is not usable end to end in a pipeline in v0.3.** A data
context built from a custom logical type registration file hashes bare values and Arrow
tables with its list, but streams and datagrams find their data context by key, so their
rows are read by the default context, which refuses a column of another list. (ITL-981,
ITL-965)
6 changes: 6 additions & 0 deletions design_issues/ITL-888-AR1.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,9 @@ results.
Known compromise: the struct's children are nullable at the Arrow level, because Polars drops
nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null
instead. Tighten under a new storage version once nested nullability survives Polars.

**Superseded (ITL-981, November 12, 2026):** the struct storage and the layout-independent hash
described above were replaced before any release by `.npy` storage under
`orcapod.numpy.ndarray` and a strict value hash with a per-type ignore list; see entry LV1
(`design_issues/ITL-981-LV1.md`). The refusals, the declared-output check and the v0.2
statements of this entry still hold.
33 changes: 33 additions & 0 deletions design_issues/ITL-981-LV1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## `src/orcapod/logical_types/ignore_list.py`, `src/orcapod/logical_types/numpy_type.py` — library value types

### LV1 — Library value types: storage and hash decided separately, strict hash, per-type ignore list
**Status:** resolved
**Severity:** high
**Issue:** ITL-981 (supersedes the storage and hash of ITL-888, entry AR1)

The first v0.3 array format (ITL-888, never released) tied storage to the hash: an array was
stored in Orcapod's own canonical struct (C order, little-endian, packed fields) and hashed
as that row. So an array did not read back as it was given, and the hash silently ignored
memory order and byte order, with no way to say otherwise. The same question was open for
torch tensors, Polars and pandas data frames.

Developer's decisions (November 12, 2026): storage is the library's own serialization and a
value reads back as given; the hash is defined by Orcapod on the content, never on the stored
bytes; the hash is strict by default (anything about a value that can be changed and that
storage keeps is part of its identity); what the hash ignores is a per-type list of property
names saved in the extension metadata, so two lists are two logical types. A property storage
cannot restore never counts, because a value read back must hash like the value written.

**Fix:** `logical_types/ignore_list.py` holds the shared mechanism: the metadata shape
`{"ignore":[...],"version":N}`, `IgnoreListLogicalType`, `IgnoreListHandler`, one Arrow
extension class per list. The data context declares each type's list once, in `hash_ignore`
of `contexts/logical_types.json`. The Arrow hasher hashes a cell under the list of its own
column. `numpy.ndarray` is stored as a `.npy` file (format 3.0, no pickle) under
`orcapod.numpy.ndarray` and hashed in two steps: the element digest of dtype, shape and bits,
then the type name with memory order, byte order and structured layout, each ignorable.
Design note: `superpowers/specs/2026-11-12-itl-981-library-value-types-design.md`.

Known limits, by decision: a data context has one list per type and refuses a column stored
under another (the Python schema carries only `np.ndarray`, so a per-column list needs a user
API, which v0.3 does not have); a strided view is stored and hashed as its C-contiguous copy;
unit-less datetimes and field titles stay refused (ITL-911) although `.npy` could store them.
Loading
Loading