diff --git a/CHANGELOG.md b/CHANGELOG.md index b9688f8a4..dfdc26d3b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,49 @@ ### Breaking changes and migration +#### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883) + +**The array format and every array hash changed in v0.3.** A store written by v0.2 that holds +`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays +computes again into new tables, and its v0.2 results are no longer returned. Nothing written by +v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for +v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889). + +- **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata + `{"version":1}`) over `struct, + data: large_binary>`: an Orcapod dtype tag, the shape, and the elements in C order, + little-endian, packed. It replaces the `.npy` file under `numpy.ndarray`; that name is no + longer registered or written. +- **Identity.** An array is its dtype, shape and element bits. A structured dtype's field + names, types and order are part of the dtype, at every nesting depth. Memory layout, byte + order, structured offsets and padding, and subclass are not part of it. Arrays read back + canonical: C order, little-endian (`>f4` reads back ` np.ndarray` that returns something other than an array, + such as a NumPy scalar, now raises `TypeError` naming the output. +- **MergeJoin** orders merged arrays by their stored `(dtype tag, shape, data)`, and lists of + arrays (an array column after `GroupBy` or `Batch`) by their elements' keys, in order. +- **Pick** on a column whose logical type has no keyed access (an array, a dataclass, a + pydantic model) raises `InputValidationError` naming the column, instead of + `NotImplementedError` (ITL-883). +- `LogicalTypeRegistry.get_by_python_type` now picks the most specific registered class along + the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its + subclasses, whatever the registration order. + #### A changed column type is refused when appending to a Delta table (ITL-884) Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows' diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 616fc31fe..695708a20 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2380,6 +2380,8 @@ newcomer. Re-checked on `b197544e` (v0.2.0 + #291): column ... because it's a binary field` lines on every Delta write still appear. **Open:** awaiting the developer's choice of how to silence them. +--- + ## `src/orcapod/core/operators/filters.py` — `PolarsFilter` identity and nullability (ITL-828) ### PF1 — An expression predicate could not be hashed, and a filter broke a join inside a job @@ -2410,3 +2412,33 @@ source row. 2026). A filtered row's data hash now equals its source row's. The pinned `pipeline:polars_filter` row data hashes moved (approved); they are the cache keys of any function pod downstream of a filter, so a 0.2 store recomputes those results once. + +--- + +## `src/orcapod/logical_types/numpy_type.py`, `src/orcapod/logical_types/array_format.py` — array format + +### AR1 — `numpy.ndarray` stored as a `.npy` file and hashed by file bytes +**Status:** in progress +**Severity:** high +**Issue:** ITL-888 (absorbs ITL-883; design ITL-874) + +Arrays were stored as the exact `np.save` file in a `large_binary` column under extension name +`numpy.ndarray`, and hashed as SHA-256 of that file under the bare label `sha256`. The hash +followed the file, not the value: equal arrays in C and Fortran order, in little- and big-endian +byte order, or with structured fields at other offsets hashed differently. Masked arrays lost +their mask silently. `StringDType` failed with a misleading pickle error. A pod declared +`-> np.ndarray` could return a NumPy scalar unchecked. + +v0.3 moves to the value-based format of `superpowers/specs/2026-11-10-array-format-v0.3.md`: +extension `orcapod.ndarray` over `struct` (little-endian, C order), hashed +by value under the label `ndarray_v1` (developer's decisions, November 10 and 11, 2026). Old +data is neither detected nor migrated in v0.3; migration is deferred to ITL-889. Accepting +`longdouble`/`clongdouble`, unit-less datetimes and titled structured dtypes, refused in v0.3, +is reconsidered in ITL-911. The new logical type name moves the table paths of nodes with an +array column and the pipeline and operator tables of every node downstream of one (pipeline +hashes are Merkle-chained); an array-free pod downstream keeps its result table and reuses its +results. + +Known compromise: the struct's children are nullable at the Arrow level, because Polars drops +nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null +instead. Tighten under a new storage version once nested nullability survives Polars. diff --git a/docs/concepts/arrays.md b/docs/concepts/arrays.md new file mode 100644 index 000000000..987d03582 --- /dev/null +++ b/docs/concepts/arrays.md @@ -0,0 +1,162 @@ +# NumPy Arrays + +A `numpy.ndarray` can be a data value like any other: a source column, a function pod +parameter or a return value. Orcapod stores each array by its **value**, its dtype, its shape +and its elements. How the array happens to sit in memory is not stored. The hash is defined on +that value, so two arrays that hold the same values hash the same, however each was laid out. + +```python +import numpy as np + +from orcapod import function_pod +from orcapod.sources import DictSource + +@function_pod(output_keys="scaled") +def scale(signal: np.ndarray) -> np.ndarray: + return signal * 2 + +source = DictSource( + data=[{"trial": "t1", "signal": np.arange(6, dtype=np.float32).reshape(2, 3)}], + tag_columns=["trial"], + source_id="trials", +) +for tag, data in scale.pod(source).iter_data(): + print(tag["trial"], data["scaled"].dtype, data["scaled"].shape) +# t1 float32 (2, 3) +``` + +!!! warning "v0.3 changed the array format" + Orcapod v0.2 stored arrays as `.npy` files under the extension name `numpy.ndarray` and + hashed the file. v0.3 replaces both, as described below, and does not migrate older + stores. Under v0.3, every node that handles arrays computes again into new tables, and its + v0.2 results are no longer returned. They stay on disk, readable with v0.2. Nodes + downstream of an array column get new pipeline and operator tables too; a function pod + among them that takes and returns no array reuses its stored results. Start v0.3 + pipelines with a fresh store, or keep using v0.2 for an existing one. A migration is + planned after v0.3 (ITL-889). + +## What counts as the same array + +Two arrays are the same value when they have the same dtype, the same shape and the same +element bits. + +- **Memory layout does not count.** C order, Fortran order, strided views and transposed views + of equal values are the same array. An array always reads back in C order. +- **Byte order does not count.** Arrays are stored little-endian; a big-endian `>f4` array + reads back as `f4"))) +# True +print(hasher.hash_object(a) == hasher.hash_object(a.astype(np.float64))) +# False +ab = np.zeros(1, dtype=[("a", "i4"), ("b", "f8")]) +ba = np.zeros(1, dtype=[("b", "f8"), ("a", "i4")]) +print(hasher.hash_object(ab) == hasher.hash_object(ba)) +# False +``` + +## What cannot be stored + +Some arrays are refused when they are written, with a `ValueError` (an +`orcapod.errors.UnsupportedValueError`) that names the column and the dtype: + +| Array | Why | What to do | +|---|---|---| +| Masked arrays (`np.ma.MaskedArray`) | extended data: the mask would be lost | store the data and the mask as two arrays | +| `dtype=object`, or a structured dtype with an object field | extended data: would need pickling | convert to a fixed dtype | +| NumPy 2 `StringDType` | extended data: variable width | use a fixed-width string dtype such as `U10` | +| `longdouble`, `clongdouble` | width and format differ by platform | `.astype(np.float64)` or `.astype(np.complex128)` | +| `datetime64` or `timedelta64` without a unit | the integer has no defined meaning | give a unit, e.g. `datetime64[ns]` | +| Structured dtypes with field titles | titles are aliases of fields | drop the titles | + +Extended data may be supported later, stored as several parts rather than as one array. +Whether `longdouble` and `clongdouble`, unit-less datetimes and timedeltas, and structured +dtypes with titles should be accepted in a later release is tracked in ITL-911; v0.3 refuses +them. + +```python +try: + DictSource( + data=[{"trial": "t1", "signal": np.ma.array([1.0, 2.0], mask=[False, True])}], + tag_columns=["trial"], + ) +except ValueError as exc: + print(type(exc).__name__) +# UnsupportedValueError +``` + +A function pod that declares `-> np.ndarray` must return an array. A NumPy scalar, for +example `x * 2` on a 0-d array, raises `TypeError` that names the output. Return +`np.asarray(...)` instead. + +## How an array is stored + +An array column has the Arrow extension type `orcapod.ndarray`, with extension metadata +`{"version":1}`. Its storage is one struct per row: + +| Field | Arrow type | Holds | +|---|---|---| +| `dtype` | `large_string` | the Orcapod dtype tag (below) | +| `shape` | `large_list` | the dimensions, outermost first; empty for a 0-d array | +| `data` | `large_binary` | the elements in C order, little-endian, packed | + +The struct's children are nullable in the Arrow schema only because Polars drops nested +non-null flags. A stored array never has a null child. A null array is a null struct, allowed +when the column is annotated `np.ndarray | None`. + +```python +table = source.as_table() +print(table.schema.field("signal").type.extension_name) +# orcapod.ndarray +print(table.column("signal").chunk(0).storage[0].as_py()["dtype"]) +# float32 +``` + +### Dtype tags + +The tag names the element type without a byte order. For a structured dtype, the tag is a JSON +list with one `[name, tag]` entry per field, or `[name, tag, [dims]]` for a subarray field, +in declared field order. It is written with no whitespace and with non-ASCII characters escaped +(`json.dumps(..., separators=(",", ":"), ensure_ascii=True)`). + +| NumPy | Tag | Element bytes | +|---|---|---| +| `bool` | `bool` | one byte, 0 or 1 | +| `int8` ... `int64`, `uint8` ... `uint64` | `int8` ... `uint64` | two's complement or unsigned, little-endian | +| `float16`, `float32`, `float64` | same names | IEEE 754, little-endian | +| `complex64`, `complex128` | same names | real part, then imaginary part | +| `datetime64[ns]`, `datetime64[10ms]`, ... | `datetime64[ns]`, `datetime64[10ms]`, ... | `int64`, little-endian (NaT is the minimum) | +| `timedelta64[s]`, ... | `timedelta64[s]`, ... | `int64`, little-endian | +| `U5` | `ucs4[5]` | five UCS-4 code units, each `uint32` little-endian, NUL-padded | +| `S3` | `bytes[3]` | three raw bytes, NUL-padded | +| `V4` | `void[4]` | four raw bytes | +| `[("b", "f8"), ("a", "i4", (2,))]` | `[["b","float64"],["a","int32",[2]]]` | fields packed in tag order | + +## How an array is hashed + +The hash of an array is the version-1 array digest of its stored row, labelled `ndarray_v1`. +The byte-level definition, and a reference implementation that does not use Orcapod, are in +the [Hashing Reference](../reference/hashing.md#8-array-value-digest). A future PyTorch tensor +type will share the digest under its own label, so an array and a tensor with equal values +have different hashes. diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 1616667ad..13ddcf73a 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -545,3 +545,76 @@ in any data-level preimage; they serve logging, observability, and result inspec | **Known exclusions** | **Not a content hash.** Two datagrams with identical content have different UUIDs. Not used in any hash preimage. Serves as an object identity token, not a content fingerprint. | --- + +## 8. Array Value Digest + +**Value hash labels.** A type whose values are hashed by a dedicated handler labels its hash +`_v`, for example `ndarray_v1` (and later `tensor_v1`). The label is the +`ContentHash.method` and appears in every token built from the hash. The bytes produced under +an existing label never change. A new definition gets a new label, so a hash always says +which definition produced it. + +A `numpy.ndarray` value is hashed by value (since v0.3, ITL-888). The semantic hasher's +`NumpyArrayHandler` returns `ContentHash(method="ndarray_v1", digest=D)`, where `D` is the +version-1 array digest of the array's stored row. Inside a table hash, an array cell becomes +the token `b"orcapod:ndarray::ndarray_v1:" + D`: the extension name, then the label and the +digest. The stored row, the dtype tags and the canonical form are described in +[NumPy Arrays](../concepts/arrays.md). + +For a stored row with dtype tag `T`, shape `(d0, ..., dn-1)` and data bytes `B`: + +``` +P = u64le(len(utf8(T))) || utf8(T) + || u64le(n) || i64le(d0) || ... || i64le(dn-1) + || u64le(len(B)) || B +D = SHA-256(P) +``` + +`u64le` and `i64le` are 8-byte little-endian unsigned and signed integers. Every field is +length-prefixed, so two different rows never share a preimage. `D` depends only on the stored +row, so it can be recomputed from a stored table by any SHA-256 implementation. The label +`ndarray_v1` names this definition; a different definition gets a different label. A +`torch.Tensor` type is to share the digest under its own label. + +| Variation | Same `D`? | +|---|---| +| C or Fortran order, strides, views | yes | +| Little- or big-endian byte order | yes | +| Structured offsets, alignment padding | yes | +| `np.matrix`, `np.recarray`, `np.memmap` against the plain array | yes | +| Dtype (including width), shape, any element bit, structured field names or field order | no | + +The reference implementation below uses only the standard library, and the vectors are pinned: + +```python +import hashlib +import struct + +def array_digest_v1(tag: str, shape: tuple[int, ...], data: bytes) -> str: + t = tag.encode("utf-8") + p = struct.pack(" (Orcapod dtype tag, shape, little-endian C-order packed bytes; extension metadata {\"version\":1}) and NumpyArrayHandler hashes by value with label ndarray_v1 (array_digest_v1 over the stored row). The old extension name numpy.ndarray (.npy bytes, label sha256) is no longer registered or written; v0.2 stores holding arrays are not migrated (ITL-888)" ] } } diff --git a/src/orcapod/core/data_function.py b/src/orcapod/core/data_function.py index 864784eec..c84986d66 100644 --- a/src/orcapod/core/data_function.py +++ b/src/orcapod/core/data_function.py @@ -7,6 +7,7 @@ import uuid from abc import abstractmethod from collections.abc import Callable, Iterable, Sequence +import types import typing from typing import TYPE_CHECKING, Any, ClassVar, Generic, Literal, TypeVar @@ -111,6 +112,52 @@ def parse_function_outputs( E = TypeVar("E", bound=DataFunctionExecutorProtocol) +def _declared_ndarray_type(declared: Any) -> type | None: + """Return the ndarray class a declared output type requires, if any. + + Handles ``np.ndarray`` (or a subclass), parameterized aliases such as + ``npt.NDArray[np.float32]``, and ``Optional``/``| None`` of either. + NumPy is never imported here: a class counts as an ndarray when its MRO + contains ``numpy.ndarray``. + """ + candidates = [declared] + if typing.get_origin(declared) in (typing.Union, types.UnionType): + candidates = [a for a in typing.get_args(declared) if a is not type(None)] + for candidate in candidates: + # A parameterized alias such as npt.NDArray[np.float32] has origin np.ndarray. + candidate = typing.get_origin(candidate) or candidate + if isinstance(candidate, type) and any( + base.__module__ == "numpy" and base.__qualname__ == "ndarray" + for base in candidate.__mro__ + ): + return candidate + return None + + +def _check_declared_ndarray_outputs( + output_data: dict[str, Any], output_schema: Schema +) -> None: + """Refuse a non-array returned for an output declared as a NumPy array (ITL-883). + + A NumPy scalar (for example ``x * 2`` on a 0-d array) is not an array, and + Orcapod never wraps one implicitly. + + Raises: + TypeError: Naming the output key, the declared type and the returned type. + """ + for key, value in output_data.items(): + if value is None: + continue + required = _declared_ndarray_type(output_schema.get(key)) + if required is not None and not isinstance(value, required): + raise TypeError( + f"Output {key!r} is declared as {required.__module__}.{required.__qualname__}" + f" but the function returned {type(value).__module__}." + f"{type(value).__qualname__} ({value!r}); return an array, for example" + " with np.asarray(...)" + ) + + class DataFunctionBase(TraceableBase, Generic[E]): """Abstract base class for DataFunctionProtocol. @@ -576,6 +623,7 @@ def _build_output_data(self, values: Any) -> DataProtocol: A Data containing the parsed outputs with source info. """ output_data = parse_function_outputs(self._output_keys, values) + _check_declared_ndarray_outputs(output_data, self.output_data_schema) def combine(*components: tuple[str, ...]) -> str: inner_parsed = [":".join(component) for component in components] diff --git a/src/orcapod/core/operators/merge_join.py b/src/orcapod/core/operators/merge_join.py index 65bf70846..7c5d6b53b 100644 --- a/src/orcapod/core/operators/merge_join.py +++ b/src/orcapod/core/operators/merge_join.py @@ -1,3 +1,4 @@ +from collections.abc import Callable from typing import TYPE_CHECKING, Any from orcapod.core.operators.base import BinaryOperator, require_shared_arrow_type @@ -49,6 +50,71 @@ def _output_types( return resolved +def _identity_key(value: Any) -> Any: + """Order stored values as themselves (the default MergeJoin ordering).""" + return value + + +def _storage_sort_key( + type_converter: Any, arrow_type: "pa.DataType" +) -> Callable[[Any], Any]: + """Return the ordering key MergeJoin uses for a column's stored values. + + The array logical type defines ``storage_sort_key`` (its stored ``(dtype tag, + shape, data)``). A list of such values, at any depth (an array column after + ``GroupBy`` or ``Batch``), is ordered by the tuple of its elements' keys, with + ``None`` first. Every other type is ordered by its stored value as before. + General struct ordering is ITL-839. + """ + if isinstance(arrow_type, pa.ExtensionType): + lt = type_converter.get_logical_type_by_arrow_extension_name( + arrow_type.extension_name + ) + key = _logical_sort_key(lt) + if key is not None: + return key + return _identity_key + + +def _logical_sort_key(logical_type: Any) -> Callable[[Any], Any] | None: + """Return a logical type's own ordering key for stored values, if it has one. + + Args: + logical_type: A logical type, or ``None``. + + Returns: + The type's ``storage_sort_key``; for a list logical type whose elements + have a key, a key ordering lists by their elements' keys; otherwise + ``None``. + """ + key = getattr(logical_type, "storage_sort_key", None) + if callable(key): + return key + element = getattr(logical_type, "element_logical_type", None) + element_key = _logical_sort_key(element) if element is not None else None + if element_key is None: + return None + + def list_key(storage_value: Any) -> tuple[Any, ...]: + """Return the ordering key of one stored list value. + + A null list sorts before every non-null list. Non-null lists compare by + the tuple of their elements' keys, element by element in list order, so + a list that is a prefix of another sorts first. + + Args: + storage_value: The stored list, or ``None`` for a null list. + + Returns: + ``(0,)`` for a null list, otherwise ``(1, element keys...)``. + """ + if storage_value is None: + return (0,) + return (1, tuple(element_key(v) for v in storage_value)) + + return list_key + + class MergeJoin(BinaryOperator): """ Binary operator that joins two streams, merging colliding data columns @@ -357,6 +423,10 @@ def binary_static_process( left_sources = joined.column(left_source_col).to_pylist() right_sources = joined.column(right_source_col).to_pylist() + # A logical type whose stored values are not orderable as Python values + # (struct storage comes back as dicts) supplies its own ordering key. + sort_key = _storage_sort_key(tc, output_types[col]) + merged_vals = [] merged_sources = [] if has_source else None for i in range(len(left_vals)): @@ -364,11 +434,13 @@ def binary_static_process( if has_source: ls, rs = left_sources[i], right_sources[i] # Sort by data value, carry source along - pairs = sorted(zip([lv, rv], [ls, rs]), key=lambda p: p[0]) + pairs = sorted( + zip([lv, rv], [ls, rs]), key=lambda p: sort_key(p[0]) + ) merged_vals.append([p[0] for p in pairs]) merged_sources.append([p[1] for p in pairs]) else: - merged_vals.append(sorted([lv, rv])) + merged_vals.append(sorted([lv, rv], key=sort_key)) # Replace the left column with merged list, drop right column col_idx = joined.column_names.index(left_col_name) diff --git a/src/orcapod/core/operators/pick.py b/src/orcapod/core/operators/pick.py index 084317445..ef65e4d4c 100644 --- a/src/orcapod/core/operators/pick.py +++ b/src/orcapod/core/operators/pick.py @@ -137,9 +137,8 @@ def validate_unary_input(self, stream: StreamProtocol) -> None: Raises: InputValidationError: If ``column`` is missing, ``out`` collides with an existing column, or the column type does not support - pick. - NotImplementedError: If the column holds an extension type whose - ``pick_field`` implementation is not yet available. + pick (including a logical type without ``pick_field``, such as + ``numpy.ndarray``). """ _, data_schema = stream.output_schema() data_columns = list(data_schema.keys()) @@ -177,8 +176,16 @@ def validate_unary_input(self, stream: StreamProtocol) -> None: f"not a supported pick target (not dict[K,V] and no registered " f"logical type)." ) - # May raise NotImplementedError for types not yet implemented - self._output_type = lt.pick_field(self.key) + try: + self._output_type = lt.pick_field(self.key) + except NotImplementedError as exc: + # A logical type without keyed access is an invalid pick target, + # reported like every other invalid operator input (ITL-883). + raise InputValidationError( + f"Pick: column {self.column!r} has type {col_type!r}, whose " + f"logical type {lt.logical_type_name!r} does not support pick " + f"(keyed field access): {exc}" + ) from exc self._mode = "struct" # Resolved now because the streaming path has no stream. A failure is diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 50fe2aaf4..0e471a15d 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -30,6 +30,14 @@ class DuplicateTagError(ValueError): pass +class UnsupportedValueError(ValueError): + """A value cannot be stored by the logical type of its column. + + Raised at write time, for example for a masked or object-dtype NumPy array. + The type converter prefixes the message with the column name. + """ + + class DataFunctionUnavailableError(RuntimeError): """Raised when a data function proxy is invoked without a bound function. diff --git a/src/orcapod/hashing/semantic_hashing/builtin_handlers.py b/src/orcapod/hashing/semantic_hashing/builtin_handlers.py index c590cde8f..5ecf4122c 100644 --- a/src/orcapod/hashing/semantic_hashing/builtin_handlers.py +++ b/src/orcapod/hashing/semantic_hashing/builtin_handlers.py @@ -12,7 +12,7 @@ SchemaHandler -- Schema objects FileHandler -- orcapod.File: file content hash DirectoryHandler -- orcapod.Directory: recursive Merkle tree hash - NumpyArrayHandler -- numpy.ndarray: SHA-256 ContentHash of .npy bytes + NumpyArrayHandler -- numpy.ndarray: ndarray_v1 value digest (array_digest_v1) PandasDataFrameHandler -- pd.DataFrame: Arrow Table via Arrow hasher (lazy default-context fallback) PandasSeriesHandler -- pd.Series: Arrow Table via Arrow hasher (lazy default-context fallback) PolarsExprHandler -- pl.Expr: polars_expr_v1 canonical expression tree (ITL-828) @@ -313,56 +313,45 @@ def handle(self, obj: Any, hasher: "SemanticHasherProtocol") -> ContentHash: class NumpyArrayHandler: - """Hasher for ``numpy.ndarray`` — content hash via SHA-256 of numpy's ``.npy`` bytes. - - Serialises the array to numpy's ``.npy`` binary format (which encodes dtype, - including structured/record field names, shape, byte order, and data), then - returns a ``ContentHash`` produced by SHA-256 of those bytes. This is identical - to what ``LogicalNumpyArray`` stores in Arrow, so the hash input and storage - representation are always consistent. - - Returning ``ContentHash`` directly (rather than the raw ``.npy`` bytes) avoids - the hex-expansion and JSON-serialisation overhead that the semantic hasher would - apply to ``bytes`` returns — important for large arrays. - - Object-dtype arrays are rejected with ``ValueError`` immediately (before - ``np.save`` is called). Structured dtypes containing object-typed fields - are caught by ``allow_pickle=False`` in ``np.save``. + """Hasher for ``numpy.ndarray``: the version-1 value digest under ``ndarray_v1``. + + The array is put in its canonical form (C order, little-endian, structured + fields packed in declared order; see ``logical_types.numpy_type``), which is + exactly what ``LogicalNumpyArray`` stores. The digest is + ``array_format.array_digest_v1(tag, shape, data)`` of that stored row, so + memory layout, byte order, structured padding and subclass do not change the + hash, while dtype (including structured field names and order), shape and + every element bit do. + + The label ``ndarray_v1`` names the definition (and binds the digest to the + ndarray type): another array type sharing the digest body uses its own label. """ + HASH_LABEL = "ndarray_v1" + """``ContentHash.method`` of an ndarray hash.""" + def handle(self, obj: Any, hasher: "SemanticHasherProtocol") -> ContentHash: - """Return a SHA-256 ``ContentHash`` of the ``.npy`` bytes for ``obj``. + """Return the ``ndarray_v1`` value hash of ``obj``. Args: - obj: A ``numpy.ndarray`` instance. + obj: A ``numpy.ndarray`` instance (subclasses are hashed as plain arrays). hasher: Ignored. Present for protocol conformance. Returns: - A ``ContentHash`` with ``method="sha256"`` and digest equal to the - SHA-256 of the array's ``.npy`` binary representation. + ``ContentHash("ndarray_v1", array_digest_v1(tag, shape, data))``. Raises: TypeError: If ``obj`` is not a ``numpy.ndarray``. - ValueError: If ``obj`` has ``dtype.kind == "O"`` (object dtype). + ValueError: If the array cannot be stored (masked, object, + ``StringDType``, ``longdouble``, ...). """ - import hashlib - import io - import numpy as np + from orcapod.logical_types.array_format import array_digest_v1 + from orcapod.logical_types.numpy_type import array_bytes, canonical_array - if not isinstance(obj, np.ndarray): - raise TypeError( - f"NumpyArrayHandler: expected numpy.ndarray, got {type(obj)!r}" - ) - if obj.dtype.kind == "O": - raise ValueError( - f"NumpyArrayHandler does not support object-dtype arrays " - f"(dtype={obj.dtype!r}). Object arrays require pickling, " - "which is disabled for security." - ) - buf = io.BytesIO() - np.save(buf, obj, allow_pickle=False) + tag, array = canonical_array(obj) return ContentHash( - method="sha256", digest=hashlib.sha256(buf.getvalue()).digest() + method=self.HASH_LABEL, + digest=array_digest_v1(tag, array.shape, array_bytes(array)), ) diff --git a/src/orcapod/logical_types/array_format.py b/src/orcapod/logical_types/array_format.py new file mode 100644 index 000000000..bc8a04b5f --- /dev/null +++ b/src/orcapod/logical_types/array_format.py @@ -0,0 +1,105 @@ +"""Array storage struct and value digest shared by array-like logical types. + +This module defines the parts of Orcapod's array format that do not depend on +any array library: the Arrow struct an array value is stored as, and the +version-1 value digest computed from a stored row. ``numpy.ndarray`` uses them +today (``numpy_type.LogicalNumpyArray``); other array types (such as +``torch.Tensor``) reuse them under their own extension name and hash label. + +Storage struct (one value per row): + + struct< + dtype: large_string, # Orcapod dtype tag + shape: large_list, # dimensions, outermost first + data: large_binary, # element bytes, C order + > + +The children are nullable at the Arrow level only because Polars drops nested +non-null flags when it converts back to Arrow (ITL-842), and the extension type +would then no longer match. A stored array value never has a null child: the +logical type refuses one on write and on read. A null array is a null struct. + +Value digest, version 1. For a row with dtype tag ``T``, shape +``(d0, ..., d{n-1})`` and data bytes ``D``: + + P = u64le(len(utf8(T))) || utf8(T) + || u64le(n) || i64le(d0) || ... || i64le(d{n-1}) + || u64le(len(D)) || D + digest = SHA-256(P) + +``u64le`` and ``i64le`` are 8-byte little-endian unsigned and signed integers. +Every field is length-prefixed, so distinct rows never share a preimage. The +digest depends only on the three stored fields, so it can be recomputed from a +stored table with any SHA-256 implementation. + +This module must not import NumPy. +""" + +from __future__ import annotations + +import hashlib +import struct +from collections.abc import Sequence + +import pyarrow as pa + +ARRAY_FORMAT_VERSION = 1 +"""Version of the array storage struct; recorded in the extension metadata.""" + +ARRAY_FORMAT_METADATA = b'{"version":1}' +"""``ARROW:extension:metadata`` of every array-format extension type (canonical JSON). + +A reader rejects a stored type whose metadata differs, so a later storage version +is never decoded as this one. +""" + + +ARRAY_STORAGE_TYPE: pa.StructType = pa.struct( + [ + pa.field("dtype", pa.large_string()), + pa.field("shape", pa.large_list(pa.int64())), + pa.field("data", pa.large_binary()), + ] +) +"""The Arrow struct one array value is stored as (see the module docstring).""" + +_U64 = struct.Struct(" bytes: + """Compute the version-1 value digest of a stored array row. + + The buffer is fed to SHA-256 directly, so a large array is not copied. + + Args: + tag: The Orcapod dtype tag, as stored in the ``dtype`` field. + shape: The dimensions, outermost first, as stored in the ``shape`` field. + data: The element bytes, as stored in the ``data`` field. Any object + exposing a C-contiguous buffer is accepted. + + Returns: + The 32-byte SHA-256 digest defined in the module docstring. + + Raises: + ValueError: If a dimension is negative or not an integer. + """ + tag_bytes = tag.encode("utf-8") + h = hashlib.sha256() + h.update(_U64.pack(len(tag_bytes))) + h.update(tag_bytes) + h.update(_U64.pack(len(shape))) + for dim in shape: + if isinstance(dim, bool) or not isinstance(dim, int) or dim < 0: + raise ValueError( + f"array_digest_v1: every dimension must be a non-negative int, got {dim!r} " + f"in shape {tuple(shape)!r}" + ) + h.update(_I64.pack(dim)) + view = memoryview(data) + h.update(_U64.pack(view.nbytes)) + if view.nbytes: + h.update(view) + return h.digest() diff --git a/src/orcapod/logical_types/list_logical_type_factory.py b/src/orcapod/logical_types/list_logical_type_factory.py index 06c3f4639..9a173c69d 100644 --- a/src/orcapod/logical_types/list_logical_type_factory.py +++ b/src/orcapod/logical_types/list_logical_type_factory.py @@ -115,6 +115,15 @@ def logical_type_name(self) -> str: """ return self._logical_type_name + @property + def element_logical_type(self) -> LogicalTypeProtocol: + """The logical type of the list's elements. + + Returns: + The element logical type this list type was built from. + """ + return self._element_logical_type + @property def python_type(self) -> type: """The Python generic alias this logical type represents. diff --git a/src/orcapod/logical_types/numpy_type.py b/src/orcapod/logical_types/numpy_type.py index 83e24de1c..5844db693 100644 --- a/src/orcapod/logical_types/numpy_type.py +++ b/src/orcapod/logical_types/numpy_type.py @@ -1,24 +1,43 @@ -"""``numpy.ndarray`` logical type for orcapod. +"""``numpy.ndarray`` logical type: the v0.3 value-based array format. -``LogicalNumpyArray`` maps ``numpy.ndarray`` ↔ Arrow ``large_binary`` using -numpy's own ``.npy`` binary format as the storage envelope. This preserves -dtype (including structured/record field names and types), shape, byte order, -and the raw data buffer for all array kinds: 1-D, 2-D, N-D, and structured. +An array is stored as one row of the shared array struct +(``array_format.ARRAY_STORAGE_TYPE``): an Orcapod dtype tag, the shape, and the +element bytes in C order, little-endian and packed. The extension name is +``orcapod.ndarray``. The format and the dtype tags are specified in +``superpowers/specs/2026-11-10-array-format-v0.3.md`` and documented in +``docs/concepts/arrays.md``. -Object-dtype arrays (``dtype=object`` or structured dtypes containing object -fields) are explicitly rejected — they require pickling, which is disabled for -security. See the ``Limitations`` section in the ITL-460 design spec. +Writing canonicalises what is not part of an array's identity: memory order +and strides (stored C order), byte order (stored little-endian) and structured +offsets, padding and alignment (fields packed). Structured field order is part +of the identity, as NumPy's own dtype equality has it: fields are stored in +declared order, at every nesting depth. So a stored row is the array's +canonical value, and its hash (``NumpyArrayHandler``) is a function of the +stored row alone. + +Refused at write time, with a ``ValueError`` naming the dtype (the type +converter adds the column): masked arrays, object dtypes, NumPy 2 +``StringDType`` and other "extended data" arrays, ``longdouble`` and +``clongdouble`` (platform-dependent), datetimes and timedeltas without a unit, +and structured dtypes with field titles. """ from __future__ import annotations -import io +import json +import re from typing import TYPE_CHECKING, Any import numpy as np +import numpy.typing as npt import polars as pl import pyarrow as pa +from orcapod.errors import UnsupportedValueError +from orcapod.logical_types.array_format import ( + ARRAY_FORMAT_METADATA, + ARRAY_STORAGE_TYPE, +) from orcapod.logical_types.base_logical_type import BaseLogicalType from orcapod.logical_types.registry import ( make_arrow_extension_type, @@ -28,25 +47,272 @@ if TYPE_CHECKING: from orcapod.logical_types.protocols import TypeConverterProtocol +NDARRAY_EXTENSION_NAME = "orcapod.ndarray" +"""Arrow extension name (and logical type name) of the array format.""" -class LogicalNumpyArray(BaseLogicalType): - """Logical type for ``numpy.ndarray``. +_INT_TAGS = {1: "int8", 2: "int16", 4: "int32", 8: "int64"} +_UINT_TAGS = {1: "uint8", 2: "uint16", 4: "uint32", 8: "uint64"} +_FLOAT_TAGS = {2: "float16", 4: "float32", 8: "float64"} +_COMPLEX_TAGS = {8: "complex64", 16: "complex128"} + +_TAG_TO_CODE: dict[str, str] = { + "bool": "|b1", + **{tag: f" UnsupportedValueError: + """Build the refusal for an array that cannot be stored. + + Args: + dtype: The offending dtype, shown with ``repr``. + reason: Why it cannot be stored and, where possible, what to do. + + Returns: + The error to raise; the type converter prefixes the column name. + """ + return UnsupportedValueError( + f"numpy arrays with dtype {dtype!r} cannot be stored: {reason}" + ) + + +def _tag_value(dtype: np.dtype[Any]) -> str | list[Any]: + """Return the tag of ``dtype`` as a string or, for structured dtypes, a JSON list.""" + if dtype.fields is not None: + entries: list[Any] = [] + fields = {str(key): value for key, value in dtype.fields.items()} + for name in dtype.names or (): + field = fields[name] + if len(field) > 2: + raise _refuse(dtype, "structured fields with titles are not supported") + sub = field[0] + if sub.subdtype is not None: + base, sub_shape = sub.subdtype + entries.append([name, _tag_value(base), list(sub_shape)]) + else: + entries.append([name, _tag_value(sub)]) + return entries + if dtype.subdtype is not None: + raise _refuse(dtype, "a subarray dtype is only supported as a structured field") + kind, size = dtype.kind, dtype.itemsize + if dtype.char in "gG": + raise _refuse( + dtype, + "longdouble and clongdouble differ in width and format between platforms " + "(convert with .astype(np.float64) or .astype(np.complex128))", + ) + if kind == "b": + return "bool" + if kind == "i" and size in _INT_TAGS: + return _INT_TAGS[size] + if kind == "u" and size in _UINT_TAGS: + return _UINT_TAGS[size] + if kind == "f" and size in _FLOAT_TAGS: + return _FLOAT_TAGS[size] + if kind == "c" and size in _COMPLEX_TAGS: + return _COMPLEX_TAGS[size] + if kind in "Mm": + unit, count = np.datetime_data(dtype) + if unit == "generic": + raise _refuse( + dtype, "datetimes and timedeltas need a unit, e.g. datetime64[ns]" + ) + name = "datetime64" if kind == "M" else "timedelta64" + return f"{name}[{'' if count == 1 else count}{unit}]" + if kind == "U": + return f"ucs4[{size // 4}]" + if kind == "S": + return f"bytes[{size}]" + if kind == "V": + return f"void[{size}]" + if kind == "O": + raise _refuse( + dtype, + "object arrays are extended data and are not supported " + "(they would need pickling)", + ) + if kind == "T": + raise _refuse( + dtype, + "StringDType arrays are extended data and are not supported " + "(use a fixed-width string dtype such as 'U10')", + ) + raise _refuse(dtype, "this dtype has no array-format tag") + + +def dtype_to_tag(dtype: np.dtype[Any]) -> str: + """Return the Orcapod dtype tag of a NumPy dtype. + + Args: + dtype: Any NumPy dtype, in any byte order. + + Returns: + A scalar tag such as ``"float32"`` or ``"datetime64[ns]"``, or for a + structured dtype a JSON list of ``[name, tag]`` / ``[name, tag, shape]`` + entries in declared field order (``separators=(",", ":")``, + ``ensure_ascii=True``). + + Raises: + UnsupportedValueError: If the dtype cannot be stored. + """ + value = _tag_value(dtype) + if isinstance(value, str): + return value + return json.dumps(value, separators=(",", ":"), ensure_ascii=True) + + +def _is_subarray_shape(shape: Any) -> bool: + """Return whether a parsed tag value is a valid subarray shape. + + Args: + shape: The third part of a structured tag entry. + + Returns: + ``True`` for a list of non-negative ``int`` values (``bool`` excluded). + """ + return isinstance(shape, list) and all(type(d) is int and d >= 0 for d in shape) + + +def _dtype_from_value(value: Any) -> np.dtype[Any]: + """Return the canonical NumPy dtype of a parsed tag value. + + Args: + value: A scalar tag string, or the parsed JSON list of a structured tag. - Stores arrays as Arrow ``large_binary`` using numpy's ``.npy`` on-disk - format as the storage envelope. The ``.npy`` format is self-describing: - it encodes dtype (including structured/record field names), shape, byte - order, and raw data — so no custom parsing is required and all array - kinds round-trip exactly. + Returns: + The little-endian, packed NumPy dtype. - The extension name ``"numpy.ndarray"`` is chosen over the ``orcapod.*`` - convention because it is a highly stable, well-known name that makes - Arrow schemas immediately self-documenting and may enable interoperability - with other ecosystem tools that independently serialise numpy arrays. + Raises: + ValueError: If the value is not a known tag. + """ + if isinstance(value, list): + fields: list[Any] = [] + for entry in value: + if ( + not isinstance(entry, list) + or len(entry) not in (2, 3) + or not isinstance(entry[0], str) + or (len(entry) == 3 and not _is_subarray_shape(entry[2])) + ): + expected = ( + "[name, tag] or [name, tag, shape], with a string name and a " + "shape that is a list of non-negative integers" + ) + raise ValueError( + f"malformed array dtype tag entry {entry!r}: expected {expected}" + ) + if len(entry) == 3: + fields.append((entry[0], _dtype_from_value(entry[1]), tuple(entry[2]))) + else: + fields.append((entry[0], _dtype_from_value(entry[1]))) + return np.dtype(fields) + if not isinstance(value, str): + raise ValueError(f"malformed array dtype tag element {value!r}") + if value in _TAG_TO_CODE: + return np.dtype(_TAG_TO_CODE[value]) + if m := _SIZED_TAG.match(value): + return np.dtype(f"{_SIZED_CODES[m.group(1)]}{m.group(2)}") + if m := _TIME_TAG.match(value): + code = " np.dtype[Any]: + """Return the canonical NumPy dtype of an Orcapod dtype tag. + + The canonical dtype is little-endian and, for structured tags, packed with + fields in tag order (the declared order of the written array). + + Args: + tag: A tag produced by ``dtype_to_tag``. + + Returns: + The canonical NumPy dtype. + + Raises: + ValueError: If the tag is malformed or unknown. + """ + if tag.startswith("["): + return _dtype_from_value(json.loads(tag)) + return _dtype_from_value(tag) - Object-dtype arrays are not supported. ``python_to_storage`` raises - ``ValueError`` immediately (before calling ``np.save``) when - ``array.dtype.kind == "O"``. ``np.save(..., allow_pickle=False)`` acts as - a secondary guard for structured dtypes that contain object-typed fields. + +def _copy_by_field_name(dst: npt.NDArray[Any], src: npt.NDArray[Any]) -> None: + """Copy ``src`` into ``dst`` field by field, by name, recursively. + + ``dst`` has the same fields in the same declared order as ``src`` but a + packed, little-endian layout; copying by name keeps the pairing explicit + whatever the source offsets are. + """ + names = dst.dtype.names + if names is None: + dst[...] = src + return + for name in names: + _copy_by_field_name(dst[name], src[name]) + + +def canonical_array(value: npt.NDArray[Any]) -> tuple[str, npt.NDArray[Any]]: + """Return the dtype tag and the canonical form of an array. + + Args: + value: A ``numpy.ndarray`` (subclasses are treated as plain arrays). + + Returns: + ``(tag, array)`` where ``array`` is a C-contiguous plain ``np.ndarray`` of + ``tag_to_dtype(tag)`` holding the same values. It may share memory with + ``value`` when ``value`` is already canonical; do not modify it. + + Raises: + UnsupportedValueError: For masked arrays and dtypes that cannot be stored. + TypeError: If ``value`` is not a ``numpy.ndarray``. + """ + if isinstance(value, np.ma.MaskedArray): + raise UnsupportedValueError( + f"numpy masked arrays (dtype {value.dtype!r}) are extended data and " + "cannot be stored: store the data and the mask as two arrays" + ) + if not isinstance(value, np.ndarray): + raise TypeError(f"expected a numpy.ndarray, got {type(value).__name__}") + # The base implementation, so a subclass's own ``view`` cannot change the values. + plain = np.ndarray.view(value, np.ndarray) + tag = dtype_to_tag(plain.dtype) + dtype = tag_to_dtype(tag) + if dtype.names is None: + return tag, plain.astype(dtype, order="C", copy=False) + out = np.empty(plain.shape, dtype=dtype) + _copy_by_field_name(out, plain) + return tag, out + + +def array_bytes(array: npt.NDArray[Any]) -> memoryview: + """Return the bytes of a C-contiguous array without copying. + + Args: + array: A C-contiguous array, as returned by ``canonical_array``. + + Returns: + A byte view of the array's buffer (empty when it holds no bytes). + """ + if array.size == 0 or array.dtype.itemsize == 0: + return memoryview(b"") + # A uint8 view sidesteps dtypes NumPy will not export as a buffer (datetime64). + return array.reshape(-1).view(np.uint8).data + + +class LogicalNumpyArray(BaseLogicalType): + """Logical type for ``numpy.ndarray`` in the v0.3 value-based array format. + + Stores each array as ``{"dtype": tag, "shape": [...], "data": bytes}`` in + ``ARRAY_STORAGE_TYPE`` under the extension name ``orcapod.ndarray``. A value + reads back as a writeable, C-ordered, little-endian plain ``np.ndarray``. Example: >>> lt = LogicalNumpyArray() @@ -56,31 +322,33 @@ class LogicalNumpyArray(BaseLogicalType): True """ - _arrow_ext_class = make_arrow_extension_type("numpy.ndarray", pa.large_binary()) + _arrow_ext_class = make_arrow_extension_type( + NDARRAY_EXTENSION_NAME, ARRAY_STORAGE_TYPE, ARRAY_FORMAT_METADATA + ) _arrow_ext: pa.ExtensionType | None = None - _polars_ext_class = make_polars_extension_type("numpy.ndarray", pa.large_binary()) + _polars_ext_class = make_polars_extension_type( + NDARRAY_EXTENSION_NAME, ARRAY_STORAGE_TYPE, ARRAY_FORMAT_METADATA.decode() + ) _polars_ext: pl.BaseExtension | None = None - logical_type_name: str = "numpy.ndarray" + logical_type_name: str = NDARRAY_EXTENSION_NAME python_type: type = np.ndarray def get_arrow_extension_type(self) -> pa.ExtensionType: - """Return the cached Arrow extension type for ``numpy.ndarray``. + """Return the cached Arrow extension type ``orcapod.ndarray``. Returns: - A ``pa.ExtensionType`` instance with extension name - ``"numpy.ndarray"`` and storage type ``pa.large_binary()``. + The ``pa.ExtensionType`` over ``ARRAY_STORAGE_TYPE``. """ if LogicalNumpyArray._arrow_ext is None: LogicalNumpyArray._arrow_ext = LogicalNumpyArray._arrow_ext_class() return LogicalNumpyArray._arrow_ext def get_polars_extension_type(self) -> pl.BaseExtension: - """Return the cached Polars extension type for ``numpy.ndarray``. + """Return the cached Polars extension type ``orcapod.ndarray``. Returns: - A ``pl.BaseExtension`` instance registered under - ``"numpy.ndarray"``. + The ``pl.BaseExtension`` registered under ``orcapod.ndarray``. """ if LogicalNumpyArray._polars_ext is None: LogicalNumpyArray._polars_ext = LogicalNumpyArray._polars_ext_class() @@ -88,45 +356,79 @@ def get_polars_extension_type(self) -> pl.BaseExtension: def python_to_storage( self, value: Any, converter: TypeConverterProtocol | None = None - ) -> bytes: - """Serialise a ``numpy.ndarray`` to its ``.npy`` byte representation. + ) -> dict[str, Any]: + """Encode an array as its canonical stored row. Args: - value: A ``numpy.ndarray`` instance. Object-dtype arrays - (``dtype.kind == "O"``) are rejected immediately. + value: A ``numpy.ndarray``. converter: Ignored. Present for protocol conformance. Returns: - Raw bytes in numpy's ``.npy`` format encoding dtype, shape, - byte order, and data. + ``{"dtype": tag, "shape": list_of_ints, "data": bytes}``. Raises: - ValueError: If ``value.dtype.kind == "O"`` (object-dtype array), - or if the array contains object-typed fields in a structured - dtype (caught by ``allow_pickle=False`` in ``np.save``). + UnsupportedValueError: For masked arrays and dtypes that cannot be stored. + TypeError: If ``value`` is not a ``numpy.ndarray``. """ - if value.dtype.kind == "O": - raise ValueError( - f"LogicalNumpyArray does not support object-dtype arrays " - f"(dtype={value.dtype!r}). Object arrays require pickling, " - "which is disabled for security. Serialise the array contents " - "manually (e.g. to bytes or a File) before passing to orcapod." - ) - buf = io.BytesIO() - np.save(buf, value, allow_pickle=False) - return buf.getvalue() + tag, array = canonical_array(value) + return { + "dtype": tag, + "shape": [int(d) for d in array.shape], + "data": array_bytes(array).tobytes(), + } def storage_to_python( self, storage_value: Any, converter: TypeConverterProtocol | None = None - ) -> np.ndarray: - """Reconstruct a ``numpy.ndarray`` from its ``.npy`` byte representation. + ) -> npt.NDArray[Any]: + """Decode a stored row into a writeable ``numpy.ndarray``. Args: - storage_value: Raw bytes as stored in Arrow ``large_binary``. + storage_value: The stored ``{"dtype", "shape", "data"}`` mapping. converter: Ignored. Present for protocol conformance. Returns: - A ``numpy.ndarray`` with dtype and shape as originally stored. + A C-ordered plain ``np.ndarray`` of the canonical dtype. + + Raises: + ValueError: If a field is missing or null, or the data length does + not match the dtype and shape. + """ + tag = storage_value.get("dtype") + shape = storage_value.get("shape") + data = storage_value.get("data") + if tag is None or shape is None or data is None: + raise ValueError( + "stored numpy array value has a null field " + f"(dtype={tag!r}, shape={shape!r}, data is None: {data is None})" + ) + dtype = tag_to_dtype(tag) + dims = tuple(int(d) for d in shape) + count = int(np.prod(dims, dtype=np.int64)) if dims else 1 + if len(data) != count * dtype.itemsize: + raise ValueError( + f"stored numpy array value has {len(data)} data bytes; dtype {tag!r} " + f"and shape {list(dims)} need {count * dtype.itemsize}" + ) + if dtype.itemsize == 0: + return np.zeros(dims, dtype=dtype) + return np.frombuffer(data, dtype=dtype, count=count).reshape(dims).copy() + + def storage_sort_key(self, storage_value: Any) -> tuple[Any, ...]: + """Ordering key for stored values, used where operators sort values. + + Orders by the stored ``(dtype tag, shape, data)``; ``None`` sorts first. + + Args: + storage_value: A stored row, or ``None``. + + Returns: + A tuple that orders stored rows deterministically. """ - buf = io.BytesIO(bytes(storage_value)) - return np.load(buf, allow_pickle=False) + if storage_value is None: + return (0,) + return ( + 1, + storage_value["dtype"], + list(storage_value["shape"]), + bytes(storage_value["data"]), + ) diff --git a/src/orcapod/logical_types/pandas_type.py b/src/orcapod/logical_types/pandas_type.py index 2bb962155..8fd735d8f 100644 --- a/src/orcapod/logical_types/pandas_type.py +++ b/src/orcapod/logical_types/pandas_type.py @@ -73,8 +73,7 @@ class LogicalPandasDataFrame(BaseLogicalType): nullable integer, boolean, datetime, categorical) is supported. Columns with truly mixed Python types that Arrow cannot infer raise ``ValueError``. - The extension name ``"pandas.dataframe"`` is library-qualified, following the - ``"numpy.ndarray"`` precedent. + The extension name ``"pandas.dataframe"`` is library-qualified. Example: >>> lt = LogicalPandasDataFrame() diff --git a/src/orcapod/logical_types/registry.py b/src/orcapod/logical_types/registry.py index a33790422..847626f8f 100644 --- a/src/orcapod/logical_types/registry.py +++ b/src/orcapod/logical_types/registry.py @@ -351,13 +351,20 @@ def get_by_logical_name(self, name: str) -> LogicalTypeProtocol | None: def get_by_python_type(self, python_type: type) -> LogicalTypeProtocol | None: """Return the logical type for *python_type*, or ``None``. - Checks exact match first, then falls back to an ``issubclass`` scan. - When multiple registered types are superclasses of *python_type*, the - one registered first wins (insertion-order dict, Python 3.7+). + The most specific registered type wins: *python_type* itself, then each + class of its MRO in order, so a subclass registered with its own logical + type is used for itself and its subclasses whatever the registration + order (an ``np.ndarray`` subclass without one is an array). Types that are + subclasses only through ``__subclasshook__``/ABC registration fall back + to an ``issubclass`` scan, where the first registered match wins. """ result = self._by_python_type.get(python_type) if result is not None: return result + for base in getattr(python_type, "__mro__", ())[1:]: + result = self._by_python_type.get(base) + if result is not None: + return result for registered_type, registered_lt in self._by_python_type.items(): try: if issubclass(python_type, registered_type): diff --git a/src/orcapod/semantic_types/universal_converter.py b/src/orcapod/semantic_types/universal_converter.py index b68a6f39c..35bd27df0 100644 --- a/src/orcapod/semantic_types/universal_converter.py +++ b/src/orcapod/semantic_types/universal_converter.py @@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Any, TypedDict, get_args, get_origin from orcapod.contexts import DataContext, resolve_context +from orcapod.errors import UnsupportedValueError from orcapod.semantic_types.type_inference import infer_python_schema_from_pylist_data from orcapod.types import DataType, Schema, SchemaLike from orcapod.utils import arrow_utils @@ -1006,7 +1007,13 @@ def python_dicts_to_struct_dicts( converted_record = {} for field_name, converter in converters.items(): if field_name in record: - converted_record[field_name] = converter(record[field_name]) + try: + converted_record[field_name] = converter(record[field_name]) + except UnsupportedValueError as exc: + # The value's own type cannot say which column it was in. + raise UnsupportedValueError( + f"Column {field_name!r}: {exc}" + ) from exc else: converted_record[field_name] = None converted_data.append(converted_record) diff --git a/superpowers/plans/2026-11-10-itl-888-array-format.md b/superpowers/plans/2026-11-10-itl-888-array-format.md new file mode 100644 index 000000000..5f5c8ef67 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-888-array-format.md @@ -0,0 +1,275 @@ +# Value-based array format (ITL-888) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Store and hash `numpy.ndarray` values in the v0.3 value-based format of +`superpowers/specs/2026-11-10-array-format-v0.3.md`. + +**Architecture:** A NumPy-free module `logical_types/array_format.py` owns the storage struct and +the value digest. `logical_types/numpy_type.py` maps NumPy dtypes to tags and canonicalises +arrays. `NumpyArrayHandler` hashes through the shared digest. Public-interface tests pin +behaviour across operators, orchestrators, databases and processes. + +**Tech Stack:** numpy 2.2.6, pyarrow 23.0.1, polars 1.41.2, deltalake 1.0.2, starfix, pytest. + +**Gating:** Phase 1 was approved by the coordinator on November 10. Phase 2 implements the +developer's decisions on the identity rules, the name `orcapod.ndarray`, the label `ndarray_v1`, +D-PATH and the refused dtypes (November 10 and 11, 2026; listed in the spec's Decisions). + +--- + +## File structure + +| File | Responsibility | +|---|---| +| `src/orcapod/logical_types/array_format.py` (new) | Storage struct, format version, `array_digest_v1`. No NumPy import. Shared with torch (ITL-871). | +| `src/orcapod/logical_types/numpy_type.py` | dtype to tag and back, canonicalisation, refusals, `LogicalNumpyArray` (Phase 2) | +| `src/orcapod/hashing/semantic_hashing/builtin_handlers.py` | `NumpyArrayHandler` on the new digest and label (Phase 2) | +| `src/orcapod/core/function_pod.py` (or the converter it calls) | declared `np.ndarray` return check (Phase 2) | +| `src/orcapod/contexts/data/v0.1.json` | changelog line (Phase 2) | +| `tests/test_arrays/` (new package) | public-interface array tests (Phase 1 scaffolding, Phase 2 format tests) | +| `tests/test_logical_types/test_array_format.py` (new) | digest against an independent reference | + +--- + +## Phase 1 (approved) + +### Task 1: storage struct and value digest + +**Files:** +- Create: `src/orcapod/logical_types/array_format.py` +- Test: `tests/test_logical_types/test_array_format.py` + +- [ ] **Step 1: Write the failing test.** The reference is written independently with `struct` + and `hashlib` only, framing per spec C1: + +```python +import hashlib +import struct + +import pyarrow as pa +import pytest + +from orcapod.logical_types.array_format import ( + ARRAY_STORAGE_TYPE, + array_digest_v1, +) + + +def reference_digest(tag: str, shape: tuple[int, ...], data: bytes) -> bytes: + t = tag.encode("utf-8") + p = struct.pack(" + np.ndarray`; `np.ndarray | None` with a `None` row. +- Operators (called `op(stream)`, read through `iter_data`/`as_table`): `Join`, `SemiJoin`, + `MapData`, `SelectDataColumns`, `DropDataColumns`, `PolarsFilter` on a tag, `Batch`, + `GroupBy`, `MergeJoin` with a colliding array column (values preserved as a set, column is + `list[np.ndarray]`). +- Pipeline stores: `PipelineJob` with `double` and `total`, in-memory and Delta, `job.run()` and + `AsyncPipelineOrchestrator().run(job.dag)`; every node's arrays exact. +- Cached re-run: Delta store, two fresh subprocesses; the second calls no pod (call counter + file) and returns identical arrays and identical node content hashes. +- Hash stability: semantic hash of each array, datagram content hash, and a source's + `content_hash()` in three subprocesses with `PYTHONHASHSEED` 0, 1 and random: identical. + +Steps: write each file, run it (`uv run pytest tests/test_arrays -v`), expect PASS on today's +code; fix the test, not the library, if a test is wrong; report any library failure as a +finding. Commit `test(arrays): public-interface array scaffolding (ITL-888)`. + +### Task 3: draft PR + +Push the branch, open a draft PR against `main` (body: `Fixes ITL-888`, `Fixes ITL-883`, +`Refs ITL-874`, the spec link, the trailer line), then +`orca linear attach` and `orca worktree set --worktree current --pr `. + +--- + +## Phase 2 (after decisions; expanded with the decided values before starting) + +Each task is test-first: write the failing tests, show the failure, implement, pass, commit. + +4. **Tag mapping and canonicalisation** (`numpy_type.py`): `dtype_to_tag`, `tag_to_dtype`, + `canonicalise(arr)`. Tests: every row of spec A2, structured with unsorted fields, nested and + subarray fields, refusals (object, `StringDType`, masked, plus the decided ones) naming the + dtype. Structured copies go field by field (NumPy assigns structured dtypes by position). + Byte order handled by one function, so preserved vs canonicalised is a local change. +5. **`LogicalNumpyArray` on the struct**: new extension name and metadata, polars extension, + `python_to_storage`/`storage_to_python` (writeable copy). Tests: stored Arrow type, round + trip per dtype, extension name `numpy.ndarray` no longer registered. +6. **Column name in conversion errors**: refusal raised through `DictSource` and a function pod + names the column and the dtype. +7. **`NumpyArrayHandler`**: `ContentHash(