Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2171,3 +2171,23 @@ equals the materialized one. Hashes of pipelines that already worked are unchang
Regressions: `tests/test_core/operators/test_chained_aggregation.py`. Open: on the async
barrier path, per-row source info is still row 0's (ITL-845); that assertion is a strict
xfail.

---

## `src/orcapod/logical_types/numpy_type.py` — array and tensor storage

### AT1 — NumPy array hash commits to the `.npy` file, not the array value; no tensor format yet
**Status:** open
**Severity:** high
**Issue:** ITL-874

`LogicalNumpyArray` stores each array as its `np.save` bytes, and `NumpyArrayHandler` hashes
SHA-256 of those bytes. Storage is lossless and stable across processes and NumPy 1.26/2.x.
The hash, however, changes with memory layout (C vs Fortran), byte order and structured
field order, contrary to ITL-465. The column type carries no dtype or shape, so
`NDArray[np.float32]` is rejected and MergeJoin merges float32 with float64 arrays. Masked
arrays silently lose their mask. A pod declared `-> np.ndarray` may return an unchecked NumPy
scalar. Any change of format or hash must ship as an explicit migration to new tables: in
Delta, new-format rows under an unchanged pipeline path hide the old rows. Review,
experiments and options for v0.3 and PyTorch (ITL-871):
`superpowers/specs/2026-11-10-array-tensor-storage.md`. Awaiting the developer's decision.
34 changes: 34 additions & 0 deletions superpowers/plans/2026-11-10-itl-874-array-tensor-storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# ITL-874 plan: array and tensor storage review and design

Issue: ITL-874. Branch: `eywalker/itl-874-design-array-and-tensor-storage-numpy`.
Review and design only. No change under `src/` or `tests/`.

## Deliverables

1. `superpowers/specs/2026-11-10-array-tensor-storage.md`: the design document.
2. `superpowers/specs/array-tensor-storage-experiments/*.py`: experiment scripts. Each one runs
with `uv run python <script>` and prints its result. Every claim in the document cites one,
or says "not verified".
3. A `DESIGN_ISSUES.md` entry for ITL-874, appended at the end.
4. A PR against `main` with only those files. The body says "Refs ITL-874".

## Steps

1. Part A1, hashing and storage (`a1_*.py`). Find the stored bytes and the hashed bytes.
Check hash stability across processes, memory layout, byte order, dtype, `.npy` header
version and NumPy 1.26 against 2.x. Cover 0-d, empty and large arrays.
2. Part A2, dtype coverage (`a2_*.py`). Cover every dtype kind, structured field order,
object, masked, `np.generic` and subclasses.
3. Part A3, pipeline behaviour through the public interface (`a3_*.py`). Cover function pods,
every operator, the sync and async orchestrators, the in-memory, Delta and SQLite
databases, and a cached re-run in a fresh process.
4. Part A4, annotations and what foreign tools see (`a4_*.py`).
5. Part B, candidates (`b*_*.py`). Compare `.npy`, `fixed_shape_tensor`,
`variable_shape_tensor`, a flattened logical type and out-of-table storage on the brief's
criteria. Run torch through `uv run --with torch`.
6. Write the document. Part A findings and defects with repros, the Part B table, migration
paths, and Part C options with my pick.
7. Verify that every script runs. Run ruff on the scripts. Merge `origin/main`, run the full
suite, push, open the PR and link it.

Steps 3–4 and step 5 run as parallel forks. Steps 1–2 and 6–7 run in the main session.
519 changes: 519 additions & 0 deletions superpowers/specs/2026-11-10-array-tensor-storage.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,243 @@
"""A1: what today's NumPy support stores and what it hashes.

Run: ``uv run python superpowers/specs/array-tensor-storage-experiments/a1_stored_and_hashed_bytes.py``

Claims tested:

* The stored value is exactly ``np.save(arr, allow_pickle=False)`` bytes, header included,
in a ``large_binary`` column with extension name ``numpy.ndarray``.
* A datagram's content hash is starfix over a token column
``b"numpy:ndarray::sha256:" + sha256(np.save(np.load(stored)))``, so the hash commits
to the whole ``.npy`` file, header included.
* Hash stability across processes, memory layout (C, Fortran, non-contiguous views),
byte order, dtype width, ``.npy`` header version, 0-d, empty and large arrays.
"""

from __future__ import annotations

from typing import Any

import hashlib
import io
import json
import subprocess
import sys
import time
import tracemalloc

import numpy as np
import numpy.typing as npt
import pyarrow as pa
from starfix import ArrowDigester

from orcapod.contexts import get_default_context
from orcapod.core.datagrams.datagram import Datagram


def check(name, fn):
try:
status, observed = fn()
print(f"[{status}] {name}: {observed}")
except Exception as exc: # noqa: BLE001 (experiment: report every failure)
print(f"[ERROR] {name}: {type(exc).__name__}: {exc}")


def npy_bytes(arr: npt.NDArray[Any]) -> bytes:
buf = io.BytesIO()
np.save(buf, arr, allow_pickle=False)
return buf.getvalue()


def datagram_hash(arr) -> str:
return Datagram({"x": arr}).content_hash().to_hex()


def stored(arr) -> tuple[pa.Field[Any], bytes]:
table = Datagram({"x": arr}).as_table()
field = table.schema.field("x")
col = table.column("x").combine_chunks()
storage = col.storage if isinstance(col, pa.ExtensionArray) else col
return field, storage[0].as_py()


ARR = np.arange(6, dtype=np.float64).reshape(2, 3)


def stored_is_npy():
field, raw = stored(ARR)
same = raw == npy_bytes(ARR)
header_len = int.from_bytes(raw[8:10], "little")
header = raw[10 : 10 + header_len].decode("latin1").strip()
return (
"OK" if same else "DEFECT",
f"field type={field.type!r}; storage==np.save bytes: {same}; "
f"magic={raw[:8]!r}; header={header!r}; total={len(raw)} bytes "
f"for {ARR.nbytes} data bytes",
)


def reproduce_hash():
ctx = get_default_context()
digest = hashlib.sha256(npy_bytes(ARR)).digest()
token = b"numpy:ndarray::sha256:" + digest
# The datagram field is non-nullable; nullability is part of the starfix hash.
schema = pa.schema([pa.field("x", pa.large_binary(), nullable=False)])
table = pa.table({"x": pa.array([token], type=pa.large_binary())}, schema=schema)
manual = ArrowDigester.hash_table(table, include_metadata=False)
via_hasher = ctx.arrow_hasher.hash_table(Datagram({"x": ARR}).as_table()).digest
dg = Datagram({"x": ARR}).content_hash()
ok = manual == via_hasher
return (
"OK" if ok else "DEFECT",
"starfix over a non-null large_binary column holding "
"b'numpy:ndarray::sha256:' + sha256(np.save(arr)) == arrow_hasher(datagram "
f"table): {ok}; "
f"datagram.content_hash method={dg.method!r}, "
f"equals arrow_hasher digest: {dg.digest == via_hasher}",
)


CHILD = """
import numpy as np
from orcapod.core.datagrams.datagram import Datagram
print(Datagram({"x": np.arange(6, dtype=np.float64).reshape(2, 3)}).content_hash().to_hex())
"""


def cross_process():
outs = {
subprocess.run(
[sys.executable, "-c", CHILD], capture_output=True, text=True, check=True
).stdout.strip()
for _ in range(2)
}
outs.add(datagram_hash(ARR))
return (
"OK" if len(outs) == 1 else "DEFECT",
f"distinct hashes over 3 processes: {len(outs)}",
)


def layout_c_vs_f():
c = np.ascontiguousarray(ARR)
f = np.asfortranarray(ARR)
same = datagram_hash(c) == datagram_hash(f)
back = stored(f)[1]
rt = np.load(io.BytesIO(back))
return (
"INFO",
f"equal values, C vs Fortran order -> same hash: {same}; "
f"Fortran input stored with fortran_order=True: {b'True' in back[:128]}; "
f"read-back flags F_CONTIGUOUS={rt.flags.f_contiguous}",
)


def noncontiguous():
big = np.arange(24, dtype=np.float64).reshape(4, 6)
view = big[::2, 1:4] # neither C nor F contiguous
copy = view.copy()
t_view = big.T # F-contiguous view of a C array
t_copy = np.ascontiguousarray(big.T)
return (
"INFO",
f"strided view vs C copy same hash: {datagram_hash(view) == datagram_hash(copy)}; "
f"transposed view (F-contig) vs C copy same hash: "
f"{datagram_hash(t_view) == datagram_hash(t_copy)}",
)


def byte_order():
le = np.array([1.0, 2.0], dtype="<f4")
be = np.array([1.0, 2.0], dtype=">f4")
rt = np.load(io.BytesIO(stored(be)[1]))
return (
"INFO",
f"<f4 vs >f4 equal values same hash: {datagram_hash(le) == datagram_hash(be)}; "
f"big-endian read back as dtype {rt.dtype.str!r}",
)


def dtype_width():
return (
"INFO",
"int32 vs int64 [1,2,3] same hash: "
f"{datagram_hash(np.array([1, 2, 3], np.int32)) == datagram_hash(np.array([1, 2, 3], np.int64))}",
)


def header_version():
# An array that arrives from a v2.0 .npy file is re-encoded by np.save on store,
# so the stored and hashed header is v1.0 regardless of the source file.
buf = io.BytesIO()
np.lib.format.write_array(buf, ARR, version=(2, 0))
from_v2 = np.load(io.BytesIO(buf.getvalue()))
same = datagram_hash(from_v2) == datagram_hash(ARR)
# np.save itself picks v3.0 for non-latin1 field names.
s = np.zeros(2, dtype=[("é中", "<i4")])
raw = npy_bytes(s)
return (
"INFO",
f"array loaded from a v2.0 file hashes as from v1.0: {same}; "
f"np.save version for non-latin1 field name: {raw[6]}.{raw[7]}; "
f"for ARR: {npy_bytes(ARR)[6]}.{npy_bytes(ARR)[7]}",
)


def zero_dim_and_empty():
out = []
for arr in (
np.array(3.5),
np.zeros((0,), np.float64),
np.zeros((0, 3), np.float64),
np.zeros((3, 0), np.float64),
):
rt = np.load(io.BytesIO(stored(arr)[1]))
out.append(
f"shape {arr.shape}: rt shape {rt.shape}, hash {datagram_hash(arr)[:12]}"
)
distinct = len({datagram_hash(np.zeros(s)) for s in [(0,), (0, 3), (3, 0)]})
return (
"INFO",
"; ".join(out) + f"; distinct hashes for empty shapes: {distinct}/3",
)


def large():
arr = np.ones((32 * 1024 * 1024,), dtype=np.float32) # 128 MiB
tracemalloc.start()
t0 = time.perf_counter()
h = datagram_hash(arr)
dt = time.perf_counter() - t0
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
return (
"INFO",
f"128 MiB float32: datagram content hash in {dt:.2f}s, traced peak "
f"{peak / arr.nbytes:.1f}x the array size (hash {h[:12]})",
)


def handler_direct():
ctx = get_default_context()
h = ctx.semantic_hasher.hash_object(ARR)
return (
"INFO",
f"semantic_hasher.hash_object(ndarray) method={h.method!r}, digest == "
f"sha256(.npy): {h.digest == hashlib.sha256(npy_bytes(ARR)).digest()}",
)


if __name__ == "__main__":
print(f"numpy {np.__version__}, pyarrow {pa.__version__}")
print(json.dumps({"datagram_hash_ARR": datagram_hash(ARR)}))
check("stored bytes", stored_is_npy)
check("hashed bytes", reproduce_hash)
check("semantic hasher on a bare ndarray", handler_direct)
check("cross-process", cross_process)
check("C vs Fortran", layout_c_vs_f)
check("non-contiguous", noncontiguous)
check("byte order", byte_order)
check("dtype width", dtype_width)
check(".npy header version", header_version)
check("0-d and empty", zero_dim_and_empty)
check("large array", large)
Loading
Loading