From c3c0cd19b198440004d12a27869dca01823e8228 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 20:42:51 -0800 Subject: [PATCH 01/23] docs(spec): value-based array format for numpy.ndarray in v0.3 (ITL-888) Co-Authored-By: Claude Opus 5.5 --- .../specs/2026-11-10-array-format-v0.3.md | 331 ++++++++++++++++++ 1 file changed, 331 insertions(+) create mode 100644 superpowers/specs/2026-11-10-array-format-v0.3.md diff --git a/superpowers/specs/2026-11-10-array-format-v0.3.md b/superpowers/specs/2026-11-10-array-format-v0.3.md new file mode 100644 index 000000000..8f8e4726a --- /dev/null +++ b/superpowers/specs/2026-11-10-array-format-v0.3.md @@ -0,0 +1,331 @@ +# Value-based array format for `numpy.ndarray` in v0.3 (ITL-888) + +**Date:** 2026-11-10 +**Issue:** ITL-888 (fixes ITL-883; refs ITL-874, ITL-871, ITL-465, ITL-882, ITL-884) +**Design basis:** ITL-874 review, PR #285 (`superpowers/specs/2026-11-10-array-tensor-storage.md`, +candidate 4c, sections C4 and C5; experiments `b0`, `c1`, `c2`, `c3`, `b7`). +**Status:** Plan for coordinator and developer review. Nothing implemented yet. + +Each decision is marked **CONFIRMED** (fixed by the developer's decision of November 10, the +coordinator brief, or the ITL-874 evidence) or **PENDING** (needs the developer's yes). + +v0.3 is a public release. The extension name, the storage struct, the dtype tags, the digest +definition and the hash label below are public commitments once v0.3 is tagged. + +--- + +## Summary + +- A `numpy.ndarray` value is stored as one Arrow struct per row: `dtype` (an Orcapod dtype tag + string), `shape` (`large_list`), `data` (`large_binary`, little-endian, C order, + packed). The column's extension name is **`orcapod.ndarray`**. The old name `numpy.ndarray` is + no longer registered and is never written again. +- The hash is defined on the value: SHA-256 over a length-prefixed encoding of exactly those three + stored fields. Its label is **`ndarray_v1`**, replacing the bare `sha256`. Memory order, strides, + byte order and structured field order do not change it. +- The canonical form is applied at **storage** time, so a stored row *is* its canonical value: a + Fortran array reads back C-ordered, `>f4` reads back ` over +struct< + dtype: large_string not null, + shape: large_list not null, + data: large_binary not null +> +``` + +- Field order is `dtype, shape, data`. The field names are part of the format. +- The children are non-null: every array has a dtype, a shape (possibly empty) and a buffer + (possibly empty). Nullability of the **column** follows the annotation, as for every other + logical type: `np.ndarray` gives a non-null column, `np.ndarray | None` a nullable one, and a + `None` value is a null struct. Justification: nullability is never changed (CLAUDE.md "Arrow + types"), and a null child would have no meaning. +- `large_*` variants throughout, so `normalize_to_large_types` is the identity on this type and + operators never rebuild it. +- `shape` holds one `int64` per dimension, outermost first, every value `>= 0`. +- `data` holds the array's elements in C order, little-endian, packed (no padding): + exactly `itemsize(dtype) * prod(shape)` bytes. + +### A2. The dtype tag (PENDING: names below) + +The tag is an Orcapod string, not NumPy's `descr`. It names the element type without byte order +(storage is always little-endian), so it can be produced by any implementation, including torch +for types NumPy lacks (`bfloat16`, `float8_*` in ITL-871). It maps one to one onto the NumPy +dtypes that are supported: + +| NumPy (`dtype.str`, any byte order) | Tag | Element bytes in `data` | +|---|---|---| +| `\|b1` | `bool` | 1 byte, as NumPy holds it (0x00 or 0x01 for values NumPy creates) | +| `i1 i2 i4 i8` | `int8 int16 int32 int64` | two's complement, LE | +| `u1 u2 u4 u8` | `uint8 uint16 uint32 uint64` | LE | +| `f2 f4 f8` | `float16 float32 float64` | IEEE 754 binary16/32/64, LE | +| `c8 c16` | `complex64 complex128` | real then imaginary, each IEEE 754 LE | +| `M8[]` | `datetime64[]`, e.g. `datetime64[ns]`, `datetime64[10ms]` | int64 LE (NaT = int64 min) | +| `m8[]` | `timedelta64[]` | int64 LE | +| `U` | `ucs4[]` | n UCS-4 code units, each uint32 LE (NumPy's own representation, NUL-padded) | +| `S` | `bytes[]` | n raw bytes, NUL-padded, no byte swap | +| `V` (unstructured) | `void[]` | n raw bytes, no byte swap | +| structured | JSON array, see below | packed fields in tag order | + +The unit and multiplier are written as NumPy writes them inside the brackets (`ns`, `10ms`, `D`, +`W`, ...); the multiplier is omitted when it is 1. + +**Structured dtypes.** The tag is a JSON array with one entry per field, sorted by field name +(Unicode code point order): `[name, tag]`, or `[name, tag, [d0, d1, ...]]` for a subarray field. +A nested structured field has a nested array as its tag. The text is the output of +`json.dumps(value, separators=(",", ":"), ensure_ascii=True)`: no whitespace, non-ASCII and +control characters escaped as lowercase `\uXXXX` (surrogate pairs above U+FFFF), `"` and `\` +escaped, and the short escapes `\b \f \n \r \t`. That rule is fully specified by RFC 8259 plus +those choices, so another implementation can reproduce it. A tag beginning with `[` is +structured; no scalar tag begins with `[`. + +Example: `np.dtype([("b", ">f8"), ("a", ")`. `ndarray_v1` replaces the bare + `sha256`, so an old and a new array hash can be told apart from the label alone (C5, c2). The + label contains no `:` (`ContentHash.to_prefixed_digest` uses `:` as its separator). +- In a table hash, the Arrow hasher already folds the extension name in front of the handler's + result, so a stored array cell hashes as the token + `b"orcapod:ndarray::ndarray_v1:" + digest`. Before: `b"numpy:ndarray::sha256:" + sha256(.npy)`. +- Binding: the digest body is shared and type-neutral; the type is bound twice, by the label + (`ndarray_v1`, later `tensor_v1` for torch) and by the extension name in the token. An ndarray + and a tensor with equal values therefore have different hashes but the same digest body + (ITL-874 D6 pick). + +### C3. Invariances (CONFIRMED for layout by c1; PENDING for byte and field order, see D) + +Same hash: C vs Fortran order, any strides or views, a transposed-twice view, big- vs +little-endian copies, structured field order, aligned vs packed structured dtypes, subclass vs +base class. +Different hash: dtype (including width: `int32` vs `int64`), shape (including `(0,3)` vs `(3,0)` +and `()` vs `(1,)`), any element bit (including `-0.0` vs `0.0` and different NaN payloads), +structured field names. + +--- + +## D. Identity decisions (all PENDING the developer's confirmation, relayed by the coordinator) + +| # | Decision | Plan | Consequence | +|---|---|---|---| +| D1 | What identity is | The value: dtype tag, shape, C-order little-endian packed buffer. Not the `.npy` file. | Layout-free hash; the digest is reproducible outside NumPy. | +| D2 | Byte order | Not identity. Canonicalised to little-endian on write; `>f4` reads back as ` double(x)->ndarray -> total(y)->float`, +`direct_sum(x)->float` and an array-free `tag_only` node; the type name changed in a fresh +process): + +- Every node with an array column gets a new pipeline hash (`double`, `total`, `direct_sum`); + `tag_only` keeps its hash and its table. +- Tables that hold array **values** move: the pipeline tables of all three nodes, and the result + table of `double` (its function identity includes the `-> np.ndarray` output type). The v0.2 + tables are left untouched and stay readable by v0.2. +- Result tables of array-**consuming** pods with no array output (`total`, `direct_sum`) keep + their path, because the function identity does not change. New rows are appended under the new + input hashes. Those tables hold no array column, so the append cannot make them unreadable. +- So under v0.3 the array nodes **recompute every row once**, and their v0.2 results are no longer + returned. Nothing is damaged and no table becomes unreadable. + +**Without D-PATH** (keeping the name `numpy.ndarray` for the type hash, so paths stay): +new-format rows are appended into the v0.2 tables. Delta accepts the append, the next read fails, +and Orcapod reports no rows (b7, ITL-884). The array-free downstream tables recompute and hide +their old rows (c3). This damages v0.2 stores, so I do not recommend it. + +For the CHANGELOG and docs: "v0.3 changes how arrays are stored and hashed. A store written by +v0.2 that holds `numpy.ndarray` columns is not migrated: under v0.3 every node that handles +arrays recomputes into new tables, and its v0.2 results are no longer returned (they remain on +disk, readable with v0.2). Use a fresh store, or keep using v0.2 for that store. A migration is +planned after v0.3." The tests will pin the D-PATH behaviour on a v0.2-format Delta store fixture +(new tables, old tables byte-identical, no unreadable table). + +--- + +## F. Room for `torch.Tensor` (ITL-871) + +New module `src/orcapod/logical_types/array_format.py` with no NumPy import and no NumPy names: + +- `ARRAY_STORAGE_TYPE`: the struct of A1. +- `ARRAY_FORMAT_VERSION = 1` and the extension metadata bytes. +- `array_digest_v1(tag: str, shape: Sequence[int], data: bytes | memoryview) -> bytes`: C1. +- `make_array_storage_value(tag, shape, data)` / field accessors for a stored row. + +`numpy_type.py` holds only the NumPy side: dtype to tag, tag to dtype, canonicalisation, +`LogicalNumpyArray` (name `orcapod.ndarray`). `NumpyArrayHandler` calls `array_digest_v1`. The +torch type will add its own tag mapping (including `bfloat16`) and its own name and label +(`orcapod.tensor`, `tensor_v1`), reusing the struct and the digest. No torch code in this PR. + +--- + +## G. Golden hash tests (PR #284) + +**PR #284 is not on main yet** (open, checked November 10). I will rebase onto it if it merges +before my PR; otherwise I will list the moves in my PR and the second PR to land updates them. +Pinned entries that move, all array entries, nothing else: + +| File | Entries | Why | +|---|---|---| +| `semantic.json` | `value.ndarray.{empty,float64,int32,int64}` | new digest and label (`sha256:` becomes `ndarray_v1:`) | +| `semantic.json` | `type.ndarray` | type hash uses the logical type name (`type:orcapod.ndarray`) | +| `schema.json` | `single_field.ndarray.arrow_type`, `single_field.ndarray.schema_hash` | new storage type; type name in the schema hash | +| `datagram.json` | `ndarray.content_hash`, `ndarray.pipeline_hash`, `ndarray.row[...].data_hash` (2), the `ndarray.row[...]._tag_*::` keys (4) and `ndarray.system_tag_columns` | value hash; pipeline hash in system tag column names | + +`arrow.json` and `pipeline.json` have no array cases and must not move; the test run will show +that every non-array entry is unchanged. I will add golden pins for a few arrays of my own +(0-d, empty, Fortran, big-endian, structured) with the digest computed by an independent +reference implementation of C1 in the test (plain `hashlib` + `struct`, no Orcapod code). + +--- + +## H. Risks and cuts + +Risks: + +1. **Polars and struct-backed extension types.** Operators round-trip through Polars and restore + column types. b2 showed this works for the 4c prototype, but the real polars extension type + over a struct is new here. Test early, on every operator. +2. **MergeJoin sorting a struct column.** MergeJoin sorts colliding values; how it orders struct + values (dtype, shape, data) must be deterministic and type-strict (two different dtypes in one + column should still be accepted, as the column type is one logical type). Test early. +3. **Delta with non-null children and nested non-null list items.** Verify write, read, schema + equality on reopen and a cached re-run in a fresh process. +4. **Golden churn if #284 lands late.** Mechanical, but must be listed in the PR. +5. **D-PATH is a table path change for array nodes.** Needs the developer's explicit yes. + +Cuts if time runs short, in order: the extra golden array pins beyond four values; docs worked +example (keep the format and digest definition); refusal of `longdouble` (fall back to accepting +it with a platform-dependent note). Not cuttable: storage layout, digest and label, operator and +database coverage, the scalar-return check, the CHANGELOG entry. + +--- + +## Implementation outline (for the plan after approval) + +1. `array_format.py` (struct, version, digest) with unit tests against an independent reference. +2. `numpy_type.py`: tag mapping, canonicalisation, refusals; `LogicalNumpyArray` renamed to the + new extension; column name added to conversion errors. +3. `NumpyArrayHandler` on the new digest and label. +4. Function pod output check for declared `np.ndarray` (D8). +5. Context JSON and changelog line; CHANGELOG entry; docs (`docs/reference/hashing.md` plus a + value-types section; there is none on main, so a new `docs/concepts/` page or a section in + an existing one, to be decided in the plan). +6. Tests first, through the public interface: every dtype class, round trip, invariances and + differences, every operator including Batch, GroupBy and MergeJoin with a colliding array + column, sync and async, in-memory and Delta, cached re-run in a fresh process, cross-process + and cross-`PYTHONHASHSEED` stability, golden pins, and the v0.2 store fixture for E. From 506c13ece6c3b021442dad4a516c10fb84388169 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 20:45:39 -0800 Subject: [PATCH 02/23] feat(arrays): shared array storage struct and value digest v1 (ITL-888) Co-Authored-By: Claude Opus 5.5 --- src/orcapod/logical_types/array_format.py | 97 +++++++++ .../plans/2026-11-10-itl-888-array-format.md | 191 ++++++++++++++++++ tests/test_logical_types/test_array_format.py | 87 ++++++++ 3 files changed, 375 insertions(+) create mode 100644 src/orcapod/logical_types/array_format.py create mode 100644 superpowers/plans/2026-11-10-itl-888-array-format.md create mode 100644 tests/test_logical_types/test_array_format.py diff --git a/src/orcapod/logical_types/array_format.py b/src/orcapod/logical_types/array_format.py new file mode 100644 index 000000000..08c5b0e13 --- /dev/null +++ b/src/orcapod/logical_types/array_format.py @@ -0,0 +1,97 @@ +"""Array storage struct and value digest shared by array-like logical types. + +This module defines the parts of Orcapod's array format that do not depend on +any array library: the Arrow struct an array value is stored as, and the +version-1 value digest computed from a stored row. ``numpy.ndarray`` uses them +today (``numpy_type.LogicalNumpyArray``); other array types (such as +``torch.Tensor``) reuse them under their own extension name and hash label. + +Storage struct (one value per row):: + + struct< + dtype: large_string not null, # Orcapod dtype tag + shape: large_list not null, + data: large_binary not null, # element bytes, C order + > + +Value digest, version 1. For a row with dtype tag ``T``, shape +``(d0, ..., d{n-1})`` and data bytes ``D``:: + + P = u64le(len(utf8(T))) || utf8(T) + || u64le(n) || i64le(d0) || ... || i64le(d{n-1}) + || u64le(len(D)) || D + digest = SHA-256(P) + +``u64le`` and ``i64le`` are 8-byte little-endian unsigned and signed integers. +Every field is length-prefixed, so distinct rows never share a preimage. The +digest depends only on the three stored fields, so it can be recomputed from a +stored table with any SHA-256 implementation. + +This module must not import NumPy. +""" + +from __future__ import annotations + +import hashlib +import struct +from collections.abc import Sequence + +import pyarrow as pa + +ARRAY_FORMAT_VERSION = 1 +"""Version of the array storage struct; recorded in the extension metadata.""" + + +ARRAY_STORAGE_TYPE: pa.StructType = pa.struct( + [ + pa.field("dtype", pa.large_string(), nullable=False), + pa.field( + "shape", + pa.large_list(pa.field("item", pa.int64(), nullable=False)), + nullable=False, + ), + pa.field("data", pa.large_binary(), nullable=False), + ] +) +"""The Arrow struct one array value is stored as (see the module docstring).""" + +_U64 = struct.Struct(" bytes: + """Compute the version-1 value digest of a stored array row. + + The buffer is fed to SHA-256 directly, so a large array is not copied. + + Args: + tag: The Orcapod dtype tag, as stored in the ``dtype`` field. + shape: The dimensions, outermost first, as stored in the ``shape`` field. + data: The element bytes, as stored in the ``data`` field. Any object + exposing a C-contiguous buffer is accepted. + + Returns: + The 32-byte SHA-256 digest defined in the module docstring. + + Raises: + ValueError: If a dimension is negative or not an integer. + """ + tag_bytes = tag.encode("utf-8") + h = hashlib.sha256() + h.update(_U64.pack(len(tag_bytes))) + h.update(tag_bytes) + h.update(_U64.pack(len(shape))) + for dim in shape: + if isinstance(dim, bool) or not isinstance(dim, int) or dim < 0: + raise ValueError( + f"array_digest_v1: every dimension must be a non-negative int, got {dim!r} " + f"in shape {tuple(shape)!r}" + ) + h.update(_I64.pack(dim)) + view = memoryview(data) + h.update(_U64.pack(view.nbytes)) + if view.nbytes: + h.update(view) + return h.digest() diff --git a/superpowers/plans/2026-11-10-itl-888-array-format.md b/superpowers/plans/2026-11-10-itl-888-array-format.md new file mode 100644 index 000000000..13d8432ae --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-888-array-format.md @@ -0,0 +1,191 @@ +# Value-based array format (ITL-888) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Store and hash `numpy.ndarray` values in the v0.3 value-based format of +`superpowers/specs/2026-11-10-array-format-v0.3.md`. + +**Architecture:** A NumPy-free module `logical_types/array_format.py` owns the storage struct and +the value digest. `logical_types/numpy_type.py` maps NumPy dtypes to tags and canonicalises +arrays. `NumpyArrayHandler` hashes through the shared digest. Public-interface tests pin +behaviour across operators, orchestrators, databases and processes. + +**Tech Stack:** numpy 2.2.6, pyarrow 23.0.1, polars 1.41.2, deltalake 1.0.2, starfix, pytest. + +**Gating:** Phase 1 is approved by the coordinator now (November 10). Phase 2 depends on PENDING +decisions (identity rules, names, label, D-PATH) and starts only when the coordinator relays +them. This plan is updated with the decided values before Phase 2 starts. + +--- + +## File structure + +| File | Responsibility | +|---|---| +| `src/orcapod/logical_types/array_format.py` (new) | Storage struct, format version, `array_digest_v1`. No NumPy import. Shared with torch (ITL-871). | +| `src/orcapod/logical_types/numpy_type.py` | dtype to tag and back, canonicalisation, refusals, `LogicalNumpyArray` (Phase 2) | +| `src/orcapod/hashing/semantic_hashing/builtin_handlers.py` | `NumpyArrayHandler` on the new digest and label (Phase 2) | +| `src/orcapod/core/function_pod.py` (or the converter it calls) | declared `np.ndarray` return check (Phase 2) | +| `src/orcapod/contexts/data/v0.1.json` | changelog line (Phase 2) | +| `tests/test_arrays/` (new package) | public-interface array tests (Phase 1 scaffolding, Phase 2 format tests) | +| `tests/test_logical_types/test_array_format.py` (new) | digest against an independent reference | + +--- + +## Phase 1 (approved) + +### Task 1: storage struct and value digest + +**Files:** +- Create: `src/orcapod/logical_types/array_format.py` +- Test: `tests/test_logical_types/test_array_format.py` + +- [ ] **Step 1: Write the failing test.** The reference is written independently with `struct` + and `hashlib` only, framing per spec C1: + +```python +import hashlib +import struct + +import pyarrow as pa +import pytest + +from orcapod.logical_types.array_format import ( + ARRAY_STORAGE_TYPE, + array_digest_v1, +) + + +def reference_digest(tag: str, shape: tuple[int, ...], data: bytes) -> bytes: + t = tag.encode("utf-8") + p = struct.pack(" + np.ndarray`; `np.ndarray | None` with a `None` row. +- Operators (called `op(stream)`, read through `iter_data`/`as_table`): `Join`, `SemiJoin`, + `MapData`, `SelectDataColumns`, `DropDataColumns`, `PolarsFilter` on a tag, `Batch`, + `GroupBy`, `MergeJoin` with a colliding array column (values preserved as a set, column is + `list[np.ndarray]`). +- Pipeline stores: `PipelineJob` with `double` and `total`, in-memory and Delta, `job.run()` and + `AsyncPipelineOrchestrator().run(job.dag)`; every node's arrays exact. +- Cached re-run: Delta store, two fresh subprocesses; the second calls no pod (call counter + file) and returns identical arrays and identical node content hashes. +- Hash stability: semantic hash of each array, datagram content hash, and a source's + `content_hash()` in three subprocesses with `PYTHONHASHSEED` 0, 1 and random: identical. + +Steps: write each file, run it (`uv run pytest tests/test_arrays -v`), expect PASS on today's +code; fix the test, not the library, if a test is wrong; report any library failure as a +finding. Commit `test(arrays): public-interface array scaffolding (ITL-888)`. + +### Task 3: draft PR + +Push the branch, open a draft PR against `main` (body: `Fixes ITL-888`, `Fixes ITL-883`, +`Refs ITL-874`, the spec link, the "draft: decisions pending" note, the trailer line), then +`orca linear attach` and `orca worktree set --worktree current --pr `. + +--- + +## Phase 2 (after decisions; expanded with the decided values before starting) + +Each task is test-first: write the failing tests, show the failure, implement, pass, commit. + +4. **Tag mapping and canonicalisation** (`numpy_type.py`): `dtype_to_tag`, `tag_to_dtype`, + `canonicalise(arr)`. Tests: every row of spec A2, structured with unsorted fields, nested and + subarray fields, refusals (object, `StringDType`, masked, plus the decided ones) naming the + dtype. Structured copies go field by field (NumPy assigns structured dtypes by position). + Byte order handled by one function, so preserved vs canonicalised is a local change. +5. **`LogicalNumpyArray` on the struct**: new extension name and metadata, polars extension, + `python_to_storage`/`storage_to_python` (writeable copy). Tests: stored Arrow type, round + trip per dtype, extension name `numpy.ndarray` no longer registered. +6. **Column name in conversion errors**: refusal raised through `DictSource` and a function pod + names the column and the dtype. +7. **`NumpyArrayHandler`**: `ContentHash(