Repository navigation
feat(arrays): value-based array format for numpy.ndarray in v0.3 (ITL-888) - #289
Merged
eywalker merged 31 commits intoNov 11, 2026
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v0.3 stores and hashes
numpy.ndarrayby value, under a new extension name and a versioned hash label. It does not migrate v0.2 stores (developer's decision, November 10, 2026; migration is ITL-889). Spec:superpowers/specs/2026-11-10-array-format-v0.3.md. Plan:superpowers/plans/2026-11-10-itl-888-array-format.md.orcapod.ndarray(metadata{"version":1}) overstruct<dtype: large_string, shape: large_list<int64>, data: large_binary>. Each row holds an Orcapod dtype tag, the shape, and the elements in C order, little-endian, packed. The oldnumpy.ndarray/.npytype is no longer registered or written. The struct's children are nullable at the Arrow level, because Polars drops nested non-null flags (ITL-842). The logical type enforces non-null instead.<byte order, fields packed in declared order, plainnp.ndarray.ContentHash("ndarray_v1", array_digest_v1(tag, shape, data)), SHA-256 over a length-prefixed encoding of the stored row. The byte-level definition, a stdlib-only reference implementation and pinned vectors are indocs/reference/hashing.md§8. The digest and struct live in the NumPy-freeorcapod.logical_types.array_format, ready fortorch.Tensor(ITL-871), which will use its own name and label.orcapod.errors.UnsupportedValueError(aValueError), naming the column and dtype: masked arrays, object dtypes,StringDType,longdouble/clongdouble, unit-less datetimes, structured dtypes with titles (ITL-883). Whether to accept the last three kinds in a later release is ITL-911.-> np.ndarraythat returns a non-array (for example a NumPy scalar) raisesTypeErrornaming the output (ITL-883).(dtype tag, shape, data), and lists of arrays (afterGroupByorBatch) by their elements' keys (structs are not orderable as Python values; general struct ordering stays in ITL-839).LogicalTypeRegistry.get_by_python_typepicks the most specific registered class along the MRO, so a registered ndarray subclass wins for itself and its subclasses.orcapod.ndarray, so nodes with array columns, and every node downstream of one, get new pipeline hashes and new pipeline tables. v0.2 tables are left byte-identical and never mixed with the new format. This is pinned bytests/test_arrays/test_array_v02_store.pyon a real v0.2 Delta store. Detection and migration are deferred to ITL-889. Example below.docs/concepts/arrays.md,docs/reference/hashing.md§8. CHANGELOG entry under Breaking Changes.Decisions (developer, November 10 and 11, 2026)
0.0and-0.0hash differently, as do different NaN payloads.StringDType) are refused (November 10).orcapod.ndarray; hash labelndarray_v1(convention<type>_v<N>).longdouble/clongdouble, unit-less datetimes and timedeltas, and titled structured dtypes are refused in v0.3.Deferred
Table-path change: example
The pod
double(x: np.ndarray) -> np.ndarray(tests/test_arrays/v02_store_pods.py, store prefixv02), run on a Delta store. v0.2.0 paths computed by running the v0.2.0 tag's source; this PR's by running the branch on the same store:v02/_result/double/semantic_v0.1:25bb13c5…0fda0c8b/v0/python.function.v0/rdb_v1v02/_result/double/semantic_v0.1:aa91b89c…b1d373d9db/v0/python.function.v0/rdb_v1v02/double/semantic_v0.1:25bb13c5…0fda0c8b/v0/python.function.v0/schema:semantic_v0.1:362fa6da…be20c110/pdb_v1v02/double/semantic_v0.1:aa91b89c…b1d373d9db/v0/python.function.v0/schema:semantic_v0.1:7371429f…728c3f2a41/pdb_v1Full hashes: v0.2.0 function
25bb13c50733ae93f6d4d8cbcaa471f8a290c864c4c591c7cba762140fda0c8b, schema362fa6da23f20ec976af02d23145144fc14e20fe34ff8adf6d24fe7dbe20c110; this PR functionaa91b89cc1f3cccfffc174d174db814655192730c01e00b5b2be63b1d373d9db, schema7371429f9f4ca23bb43ecbcde80e1b7466aed165577681e28ed376728c3f2a41. The function hash moves because the pod's identity includes its-> np.ndarrayoutput type; the schema segment moves because the input has an array column. In the same store,total(y: np.ndarray) -> floatkeeps its result table and gets a new pipeline table; its v0.2 rows stay beside the new rows under distinct record ids.plus_one(n: int) -> inton an array-free source keeps both tables.Descendants without an array column move too. For
source(x: np.ndarray, n: int) -> DropDataColumns(["x"]) -> plus_one(n), theplus_onepipeline hash is7c13dce1…under v0.2.0 and9f5326e3…on this branch, whileplus_oneover an array-free source keepsb2d5c5d7…on both. Such a pod keeps its function result table, so it reuses its v0.2 results without calling the function; only its pipeline table is new.tests/test_arrays/test_array_v02_store.pypins this on the v0.2 store fixture (nodeafter_drop).Intended hash changes
Approved by the developer (November 10, 2026): new array format and versioned array hash, with no migration. The migration assessment is spec section E. Only array entries move;
_tag_record_idvalues are unchanged.semantic.jsontype.ndarrayorcapod.ndarray)semantic.jsonvalue.ndarray.empty,value.ndarray.float64,value.ndarray.int32,value.ndarray.int64sha256becomesndarray_v1schema.jsonsingle_field.ndarray.arrow_type,single_field.ndarray.schema_hashdatagram.jsonndarray.content_hash,ndarray.pipeline_hashdatagram.jsonndarray.row[r1].data_hash,ndarray.row[r2].data_hashdatagram.jsonndarray.row[r1/r2]._tag_record_id::…and._tag_source_id::…keys (4),ndarray.system_tag_columns7e66521d…→be37e32a…); the values are unchangedTest plan
dot.ruff check .,ruff format --check ., basedpyright 0 errorstests/test_arrays/, all through the public interface:PYTHONHASHSEEDstabilityLinear
Fixes ITL-888
Fixes ITL-883
Refs ITL-874
Refs ITL-889
Refs ITL-911
🤖 Generated with Claude Code
Summary by CodeRabbit