Skip to content

feat(arrays): value-based array format for numpy.ndarray in v0.3 (ITL-888) - #289

Merged
eywalker merged 31 commits into
mainfrom
eywalker/itl-888-implement-the-value-based-array-format-for
Nov 11, 2026
Merged

eywalker merged 31 commits into
mainfrom
eywalker/itl-888-implement-the-value-based-array-format-for

Conversation

@eywalker

@eywalker eywalker commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

v0.3 stores and hashes numpy.ndarray by value, under a new extension name and a versioned hash label. It does not migrate v0.2 stores (developer's decision, November 10, 2026; migration is ITL-889). Spec: superpowers/specs/2026-11-10-array-format-v0.3.md. Plan: superpowers/plans/2026-11-10-itl-888-array-format.md.

  • Storage: extension orcapod.ndarray (metadata {"version":1}) over struct<dtype: large_string, shape: large_list<int64>, data: large_binary>. Each row holds an Orcapod dtype tag, the shape, and the elements in C order, little-endian, packed. The old numpy.ndarray / .npy type is no longer registered or written. The struct's children are nullable at the Arrow level, because Polars drops nested non-null flags (ITL-842). The logical type enforces non-null instead.
  • Identity: an array is its dtype, shape and element bits. A structured dtype's field names, types and order are part of the dtype, at every nesting depth, as in NumPy's own dtype equality. Memory layout, byte order, structured offsets and padding, and subclass do not count. Arrays read back canonical: C order, < byte order, fields packed in declared order, plain np.ndarray.
  • Hash: ContentHash("ndarray_v1", array_digest_v1(tag, shape, data)), SHA-256 over a length-prefixed encoding of the stored row. The byte-level definition, a stdlib-only reference implementation and pinned vectors are in docs/reference/hashing.md §8. The digest and struct live in the NumPy-free orcapod.logical_types.array_format, ready for torch.Tensor (ITL-871), which will use its own name and label.
  • Refused at write time with orcapod.errors.UnsupportedValueError (a ValueError), naming the column and dtype: masked arrays, object dtypes, StringDType, longdouble/clongdouble, unit-less datetimes, structured dtypes with titles (ITL-883). Whether to accept the last three kinds in a later release is ITL-911.
  • Pods: a pod declared -> np.ndarray that returns a non-array (for example a NumPy scalar) raises TypeError naming the output (ITL-883).
  • MergeJoin orders merged arrays by the stored (dtype tag, shape, data), and lists of arrays (after GroupBy or Batch) by their elements' keys (structs are not orderable as Python values; general struct ordering stays in ITL-839).
  • Registry: LogicalTypeRegistry.get_by_python_type picks the most specific registered class along the MRO, so a registered ndarray subclass wins for itself and its subclasses.
  • Old stores (D-PATH): the logical type name moved to orcapod.ndarray, so nodes with array columns, and every node downstream of one, get new pipeline hashes and new pipeline tables. v0.2 tables are left byte-identical and never mixed with the new format. This is pinned by tests/test_arrays/test_array_v02_store.py on a real v0.2 Delta store. Detection and migration are deferred to ITL-889. Example below.
  • Docs: new docs/concepts/arrays.md, docs/reference/hashing.md §8. CHANGELOG entry under Breaking Changes.

Decisions (developer, November 10 and 11, 2026)

  • Identity (November 10): an array is its dtype, shape and element bits. Memory layout and byte order are not identity; storage is little-endian.
  • Structured field order is identity (November 11), with field names and types, at every nesting depth.
  • Hashing by bits (November 10): 0.0 and -0.0 hash differently, as do different NaN payloads.
  • Subclasses are treated as plain arrays (November 10).
  • Extended arrays (masked, object, StringDType) are refused (November 10).
  • NumPy array and PyTorch tensor are distinct types (November 10); the tensor type (ITL-871) gets its own name and label.
  • Names (November 11): extension and logical type name orcapod.ndarray; hash label ndarray_v1 (convention <type>_v<N>).
  • Table-path move (D-PATH) (November 11): nodes with array columns move to new tables. Because pipeline hashes are Merkle-chained, the pipeline and operator tables of every node downstream of an array column move too, even when it has no array column. An array-free pod keeps its function result table, so it reuses its results. Nodes with no array column upstream do not move.
  • Refused dtypes (November 11): longdouble/clongdouble, unit-less datetimes and timedeltas, and titled structured dtypes are refused in v0.3.
  • No migration in v0.3 (November 10).

Deferred

  • Detection and migration of v0.2 array stores: ITL-889.
  • Accepting the three refused dtype kinds above in a later release: ITL-911.

Table-path change: example

The pod double(x: np.ndarray) -> np.ndarray (tests/test_arrays/v02_store_pods.py, store prefix v02), run on a Delta store. v0.2.0 paths computed by running the v0.2.0 tag's source; this PR's by running the branch on the same store:

Table v0.2.0 This PR
result v02/_result/double/semantic_v0.1:25bb13c5…0fda0c8b/v0/python.function.v0/rdb_v1 v02/_result/double/semantic_v0.1:aa91b89c…b1d373d9db/v0/python.function.v0/rdb_v1
pipeline v02/double/semantic_v0.1:25bb13c5…0fda0c8b/v0/python.function.v0/schema:semantic_v0.1:362fa6da…be20c110/pdb_v1 v02/double/semantic_v0.1:aa91b89c…b1d373d9db/v0/python.function.v0/schema:semantic_v0.1:7371429f…728c3f2a41/pdb_v1

Full hashes: v0.2.0 function 25bb13c50733ae93f6d4d8cbcaa471f8a290c864c4c591c7cba762140fda0c8b, schema 362fa6da23f20ec976af02d23145144fc14e20fe34ff8adf6d24fe7dbe20c110; this PR function aa91b89cc1f3cccfffc174d174db814655192730c01e00b5b2be63b1d373d9db, schema 7371429f9f4ca23bb43ecbcde80e1b7466aed165577681e28ed376728c3f2a41. The function hash moves because the pod's identity includes its -> np.ndarray output type; the schema segment moves because the input has an array column. In the same store, total(y: np.ndarray) -> float keeps its result table and gets a new pipeline table; its v0.2 rows stay beside the new rows under distinct record ids. plus_one(n: int) -> int on an array-free source keeps both tables.

Descendants without an array column move too. For source(x: np.ndarray, n: int) -> DropDataColumns(["x"]) -> plus_one(n), the plus_one pipeline hash is 7c13dce1… under v0.2.0 and 9f5326e3… on this branch, while plus_one over an array-free source keeps b2d5c5d7… on both. Such a pod keeps its function result table, so it reuses its v0.2 results without calling the function; only its pipeline table is new. tests/test_arrays/test_array_v02_store.py pins this on the v0.2 store fixture (node after_drop).

Intended hash changes

Approved by the developer (November 10, 2026): new array format and versioned array hash, with no migration. The migration assessment is spec section E. Only array entries move; _tag_record_id values are unchanged.

File Entry Why
semantic.json type.ndarray type hash uses the logical type name (orcapod.ndarray)
semantic.json value.ndarray.empty, value.ndarray.float64, value.ndarray.int32, value.ndarray.int64 value digest; label sha256 becomes ndarray_v1
schema.json single_field.ndarray.arrow_type, single_field.ndarray.schema_hash new storage type; type name
datagram.json ndarray.content_hash, ndarray.pipeline_hash value hash; type name
datagram.json ndarray.row[r1].data_hash, ndarray.row[r2].data_hash value hash
datagram.json ndarray.row[r1/r2]._tag_record_id::… and ._tag_source_id::… keys (4), ndarray.system_tag_columns the column names embed the pipeline hash (7e66521d… → be37e32a…); the values are unchanged

Test plan

  • Full suite on Python 3.11, 3.12 and 3.13: 7136 passed, 15 xfailed, 96 skipped each (after merging main at c8efc09). All skips are infrastructure: 77 opt-in Postgres, 9 SpiralDB integration, 7 S3/MinIO (no Docker), 3 Graphviz dot.
  • ruff check ., ruff format --check ., basedpyright 0 errors
  • tests/test_arrays/, all through the public interface:
    • storage layout and dtype tags for every dtype class
    • canonical read-back, invariances and differences of the hash (field order and nested field order differ; padding and offsets do not), pinned digests checked against an independent reference
    • refusals naming the column, scalar return refused at the pod
    • every operator, including SemiJoin (passes since ITL-847, fix(semijoin): nested columns and string/large_string keys via a Polars semi join (ITL-847, ITL-829) #291), Batch, GroupBy and MergeJoin with a colliding array column
    • sync and async runs on in-memory and Delta stores
    • cached re-run in a fresh process (0 pod calls)
    • cross-process and cross-PYTHONHASHSEED stability
    • v0.2 store left intact; array-free pod downstream of an array column reuses its v0.2 results; shared result table keeps v0.2 rows beside new ones (fixture regenerated with the v0.2.0 tag)
  • Golden suite: only the entries above changed

Linear

Fixes ITL-888
Fixes ITL-883
Refs ITL-874
Refs ITL-889
Refs ITL-911

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • NumPy arrays now use value-based storage and versioned hashing. Memory layout and byte order do not affect identity; reads return writable, C-contiguous arrays.
  • Bug Fixes
    • Unsupported array values and invalid function outputs receive clearer validation errors. Merge operations use consistent ordering, and unsupported field selection reports a validation error.
  • Documentation
    • Added guidance on array storage, hashing, supported types, and limitations.
  • Compatibility
    • Existing v0.2 array data is not automatically migrated. Affected pipeline nodes use new tables; array-free results may continue using existing tables. v0.2 results remain readable without conversion.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant