Repository navigation
feat!: hash format 2, byte-identical to Rust; ordered_columns / ordered_struct_fields (ITL-898) - #9
Conversation
…ITL-898) Byte-level tests for spec Section 9 worked examples T1-T5, Phase 1 JSON for every flags value, invariants I1-I8, node-path rejection, streaming batch checks and keyword-only options. Adds the option table only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ons (ITL-898) Implements spec Section 9 of the starfix byte layout: options tag, flags-aware Phase 1 / hash_array type JSON, nested validity entries, node-path rejection and the per-batch schema check when streaming. With all options off every code path and output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k (ITL-898) The parity test runs every vector and every relation of the Rust golden file (to be copied in unchanged). README documents the three options. The golden-sync check also diffs ordered_v0.5.json against starfix main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th options; golden file (ITL-898) - nested_validity feeds one byte per valid Boolean value (spec 9.5.1, T6) - with any option on, Map serializes its entries field and keys_sorted as Rust does; the default (flags 0) form is unchanged - tests/golden/ordered_v0.5.json copied byte for byte from starfix (sha256 ff86795d...) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…design note (ITL-898) Default Map JSON and the default hash_array non-ASCII escaping differ from Rust in released starfix-python. They are left unchanged (moving released hashes is the developer's decision) and pinned as xfail(strict=True). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c 2.1 and 9.7 tests (ITL-898) - re-copy tests/golden/ordered_v0.5.json byte for byte (sha256 12deacf4...) - names_array_struct_f0 / _int8_f0 are the known default-mode non-ASCII hash_array divergence (strict xfail) - spec 2.1 FixedSizeList and Map examples, JSON text rules, 9.7 metadata check - 9.7 default-mode batch check (Rust checks at flags 0, Python does not): pinned as strict xfails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The Boolean digest path materializes the entire column as a Python list and byte string, causing roughly 900 MB of temporary storage for 100 million valid values instead of hashing bounded chunks.
1 open issue
What changed in this PR
Adds opt-in order-sensitive and nested-validity hashing to the pure-Python ArrowDigester, targeting the provisional Rust v0.5 layout while preserving default hashes.
Changes:
- Adds three keyword-only options, tagged hashing, ambiguous-path rejection, and streaming schema validation.
- Adds byte-level, invariant, and Rust golden-parity tests, with strict expected failures for known default-mode divergences.
- Documents the options and extends CI to check the ordered fixture against Rust upstream.
| File | Description |
|---|---|
tests/test_ordered_hashing.py |
Tests option layouts, invariants, validation, and streaming behavior. |
tests/test_known_divergences.py |
Pins known default-mode differences from Rust. |
tests/test_golden_parity_ordered.py |
Checks ordered golden vectors and hash relations. |
src/starfix/arrow_digester.py |
Implements hashing options and schema validation. |
README.md |
Explains option usage and compatibility behavior. |
docs/metamorphic/specs/2026-11-11-ordered-hashing-design.md |
Records provisional Python design decisions. |
.github/workflows/ci.yml |
Adds upstream synchronization checks for the ordered fixture. |
Review effort: Balanced
Expanding to one byte per value now works on 64Ki-element chunks via pyarrow compute (filter, cast to uint8) instead of materializing the whole column as a Python list and byte string. The digest input bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…golden at starfix 544cb1b (ITL-898) - golden file re-copied byte for byte (226 vectors, 77 relations; sha256 e21dd703...) - with any option on: Null, Interval, Utf8View, BinaryView schema forms and Rust map metadata paths (entries m/, key m//key, value m//value) - default-mode misses are strict xfails; map entries-field metadata vectors are strict xfails because pyarrow drops that metadata on IPC read - README: qualify Rust parity for default calls; document the Boolean effect of nested_validity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ess-fix scope test (ITL-898) Developer ruling 2026-11-11: the lossless layout becomes the default fix. Prepares that change and the other pending default-mode decisions as isolated module switches (all off; behaviour unchanged): _LOSSLESS_BY_DEFAULT, _FIX_DEFAULT_MAP_JSON, _FIX_DEFAULT_TYPE_JSON_UTF8, _FIX_DEFAULT_TYPE_FORMS, _FIX_DEFAULT_MAP_METADATA_PATHS, _FIX_DEFAULT_BATCH_CHECK. Tests show each flipped switch reproduces the Rust golden values, and that every golden input the 'unaffected' rule covers hashes identically under the lossless layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(ITL-898) Developer decision 2026-11-11: bump the hash format version. _FORMAT_VERSION (1 today) derives VERSION_BYTES and every pending default-mode fix, so moving to format 2 is a one-line change once the regenerated Rust spec and golden file land. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The golden suite fails on supported PyArrow 14 without decimal-schema capability guards, while option-enabled metadata traversal can miss changes due to path collisions and dictionary-wrapped types.
2 open issues
3 resolved since last review
Review effort: Balanced
…-pyarrow golden decoding (ITL-898) - with any option on, Phase 2 metadata traversal looks through dictionary types (spec 2.2.1 paths are the 9.6 node paths); format 1 unchanged - golden vectors that pyarrow < 19 (Decimal32/64) or < 16 (view types) cannot decode are strict xfails on those versions only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TL-898) src/starfix/_format2.py implements nauticalab/starfix byte-layout-spec format 2 (041b8f1, provisional): 12-byte header, canonical type JSON with the name-free Map form, Phase 1/2 keyed by node path, extension types (storage type; ARROW:extension:* re-added for registered types), the one rule (entry = validity, lengths, values; row count after the schema digest), hash_array, per-batch schema check, rejections in every mode. Data paths work on Arrow buffers (filter, flatten, bitmaps), linear in rows. tests/test_format2.py checks the 7 worked examples, every final hash the spec prints (examples 1 and 5, Map, FixedSizeList, non-ASCII hash_array), primitive forms, JSON text rules, metadata paths, invariants, rejections, streaming and extension types, byte for byte from the spec text. ArrowDigester dispatches to format 2 when _FORMAT_VERSION == 2 (still 1: nothing changes until the design is approved and the golden lands). Pending spec answers E1-E6 are marked PENDING in the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…offset tests (ITL-898) A valid dictionary key that points at a null dictionary value is a null element (pyarrow's DictionaryArray.null_count counts keys only), as Rust resolves it. Adds slicing tests at array offsets 7, 8, 9, 16, 63, 64, 65 for Boolean, Int32 and string leaves under a nullable struct and for a list of Booleans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, ITL-908) Ports the ITL-908 guards to format 2: hashing a list column never copies its shared values buffer (tracemalloc growth, isolated from caller tracing), and the number of node visits does not grow with the row count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ped literals (ITL-898) - struct child node path is always p + "/" + name (spec 3.1, R7) - absent ARROW:extension:metadata hashed as "" (spec 2.3, E1) - data hashing of unsupported types rejected before hashing, any row count, ValueError (spec 7 item 3, E4) - tests/golden/format2_v0.5.json and format1_to_format2.json copied byte for byte from starfix f7aa441 (sha256 1b6102f4..., 9d40973c...) - tests/golden/python_literals_format1_inputs.json: the inputs of the 69 hand-coded format-1 literals of this repo that format1_to_format2.json does not cover, recovered by recording the calls that produce each literal (all 69 reproduce their format-1 value from the IPC), for Rust to supply the format-2 values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BREAKING CHANGE: every hash changes; the prefix is 00 00 02 (format 2). - src/starfix/arrow_digester.py is a thin ArrowDigester over _format2 (keywords include_metadata, ordered_columns, ordered_struct_fields; no nested_validity); all format-1 code and the _FORMAT_VERSION / _FIX_* switches are deleted - removed format-1-only tests and the unreleased ordered_v0.5.json; test_golden_parity_metadata.py removed (include_metadata_v0.3.json is kept as the historical format-1 record, see tests/golden/README.md) - hand-coded literals moved to their Rust-produced format-2 values from format1_to_format2.json (39); the other 69 await Rust values for the inputs in python_literals_format1_inputs.json - tests/test_golden_parity_format2.py asserts every vector and relation of format2_v0.5.json (only version-gated skips for old pyarrow) - tests/test_api.py: keyword-only options, no third option - CI golden-sync-check diffs format2_v0.5.json and format1_to_format2.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l tests at their golden vectors (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d89) (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…printed hashes (starfix bcbe74a) (ITL-898) - metadata pairs (duplicate detection, extension keys of pyarrow-native types) are read from the PyCapsule returned by __arrow_c_schema__(), the documented Arrow C data interface route in spec 2.3, instead of the private _export_to_c; the capsule releases the exported schema - Map, FixedSizeList and non-ASCII hash_array printed hashes pinned (all five printed values of spec bcbe74a now pinned) - README extension-type wording aligned with spec 2.3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…den contract (ITL-898) Copilot review: the Python design note still said no format-1 code is kept and described version-gated golden-test skips that were removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alidity at depth and the FixedSizeList element path (ITL-898) Spec re-read at starfix 9c9a88c: - Section 13: hashes are defined over little-endian buffers; values are hashed as stored, so every format-2 entry point raises NotImplementedError on a big-endian host (README 'Platforms') - dictionary validity is that of the decoded array at any depth: tested for a column, a struct child and a list element - the FixedSizeList element node f/ claims its path: tested Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onary per batch (ITL-898) Copilot review (needs a closer look): computing dictionary validity took the dictionary's validity per batch, O(batches x dictionary size) for many small batches. When the dictionary has no nulls, validity is the indices' (identical bytes). Regression test: 50 one-row batches over a 10,000-value dictionary make no take() over the dictionary validity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Copilot's review of 98b38b1 had no inline findings. Its summary notes one scaling edge, recorded here as a known limitation and not changed in this PR:
The pure-Python package is planned to be replaced by a wrapper around the Rust crate, and the developer ruled out further pure-Python performance work. The cached-dictionary-validity optimisation for this rare input is left for later. Hashes are not affected either way. Posted by a Claude Code agent on behalf of @eywalker. |
Copy tests/golden/format2_v0.5.json byte for byte from starfix e079d56 (sha256 4d5ba202f9fa88e69c94c09947e715c51c5403803b9fa0dfc3ec7dda9582845e; 423 vectors, 114 relations, the previous 339/96 an unchanged prefix) and raise the pinned minimum counts. format1_to_format2.json is unchanged. Add golden-independent tests for the two shapes the Rust review found: a leaf null hidden under a null inner list of a nested list (hash_array, one batch and single-row batches agree with the clean form), and a dictionary of nested structs with a null hidden under a null outer value (dictionary-encoded equals plain), under every option combination. Refs ITL-898 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TL-898) pyarrow 19.0.x and 20.0.0 have no `filter` kernel for decimal32/decimal64, so dropping nulls from those arrays (directly, or nested under list/struct/dictionary of list or struct) raised ArrowNotImplementedError on the supported minimum pyarrow. Fixed-width values are now selected on a zero-copy view of their value buffer as an unsigned integer of the same width (or FixedSizeBinary(width) for wider values), so no kernel of the logical type is used. The bytes selected are unchanged: no hash changes (all 423 golden vectors and 114 relations match with no skips). tests/test_format2_type_matrix.py runs every supported leaf type, with and without nulls, sliced, chunked, batched, dictionary-encoded and nested, through hash_array, record batch and stream hashing, against the Rust golden vectors where they exist and across the equivalent forms otherwise. CI runs the suite on pyarrow 19.0.0, 20.0.0 and the latest, and the pytest run asserts the pyarrow it imports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er; README memory note (ITL-898)
_field_metadata no longer receives the node path only to build error messages: it raises
_AmbiguousMetadata (a ValueError) and phase2_json re-raises it as ValueError prefixed with
the field path ("Field 's/c': Duplicate metadata key 'a' on the field: ..."). Tests assert
the full messages, including the path.
README: auxiliary memory is not bounded (validity, lengths and list parent indices per
visited element; dictionary-of-struct/list values are materialized with take); only leaf
payload feeding is chunked.
Also the cheap ruff findings in _format2.py (sorted __slots__, unquoted annotation,
itertools.pairwise). No hash changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The breaking hash-format migration depends on the still-open metamorphic-labs/starfix#20 and requires maintainer validation of cross-repository fixture parity, Python/Arrow CI results, and coordinated merge/release sequencing.
1 resolved since last review
Review effort: Balanced
origin/main's tree is byte-identical to this branch's last #9 commit (940887c), so every add/add and content conflict (CHANGELOG.md, README.md, src/starfix/_format2.py, src/starfix/arrow_digester.py, tests/golden/format2_v0.5.json, tests/test_format2.py, tests/test_format2_type_matrix.py, tests/test_golden_parity_format2.py) resolves to main's content plus this branch's FixedSizeList changes, which is this branch's side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: failing tests for ordered and nested-validity hashing options (ITL-898) Byte-level tests for spec Section 9 worked examples T1-T5, Phase 1 JSON for every flags value, invariants I1-I8, node-path rejection, streaming batch checks and keyword-only options. Adds the option table only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: ordered_columns, ordered_struct_fields and nested_validity options (ITL-898) Implements spec Section 9 of the starfix byte layout: options tag, flags-aware Phase 1 / hash_array type JSON, nested validity entries, node-path rejection and the per-batch schema check when streaming. With all options off every code path and output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: golden parity test for ordered_v0.5.json; docs and CI sync check (ITL-898) The parity test runs every vector and every relation of the Rust golden file (to be copied in unchanged). README documents the three options. The golden-sync check also diffs ordered_v0.5.json against starfix main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: Boolean one-byte layout under nested_validity; Rust Map form with options; golden file (ITL-898) - nested_validity feeds one byte per valid Boolean value (spec 9.5.1, T6) - with any option on, Map serializes its entries field and keys_sorted as Rust does; the default (flags 0) form is unchanged - tests/golden/ordered_v0.5.json copied byte for byte from starfix (sha256 ff86795d...) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: pin known default-mode divergences from Rust as strict xfails; design note (ITL-898) Default Map JSON and the default hash_array non-ASCII escaping differ from Rust in released starfix-python. They are left unchanged (moving released hashes is the developer's decision) and pinned as xfail(strict=True). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: let the ordered fixture drift step explain a missing upstream file (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: golden file at starfix e5b49c4 (171 vectors, 73 relations); spec 2.1 and 9.7 tests (ITL-898) - re-copy tests/golden/ordered_v0.5.json byte for byte (sha256 12deacf4...) - names_array_struct_f0 / _int8_f0 are the known default-mode non-ASCII hash_array divergence (strict xfail) - spec 2.1 FixedSizeList and Map examples, JSON text rules, 9.7 metadata check - 9.7 default-mode batch check (Rust checks at flags 0, Python does not): pinned as strict xfails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: list all three default-mode divergences in the design note (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: show the escaped form in the divergence table (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: tidy ordered hashing tests (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: hash Booleans under nested_validity in bounded chunks (ITL-898) Expanding to one byte per value now works on 64Ki-element chunks via pyarrow compute (filter, cast to uint8) instead of materializing the whole column as a Python list and byte string. The digest input bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: spec 2.1 type forms and 2.2.1 map metadata paths with options; golden at starfix 544cb1b (ITL-898) - golden file re-copied byte for byte (226 vectors, 77 relations; sha256 e21dd703...) - with any option on: Null, Interval, Utf8View, BinaryView schema forms and Rust map metadata paths (entries m/, key m//key, value m//value) - default-mode misses are strict xfails; map entries-field metadata vectors are strict xfails because pyarrow drops that metadata on IPC read - README: qualify Rust parity for default calls; document the Boolean effect of nested_validity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: one-line switches for pending default-mode decisions; lossless-fix scope test (ITL-898) Developer ruling 2026-11-11: the lossless layout becomes the default fix. Prepares that change and the other pending default-mode decisions as isolated module switches (all off; behaviour unchanged): _LOSSLESS_BY_DEFAULT, _FIX_DEFAULT_MAP_JSON, _FIX_DEFAULT_TYPE_JSON_UTF8, _FIX_DEFAULT_TYPE_FORMS, _FIX_DEFAULT_MAP_METADATA_PATHS, _FIX_DEFAULT_BATCH_CHECK. Tests show each flipped switch reproduces the Rust golden values, and that every golden input the 'unaffected' rule covers hashes identically under the lossless layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: tie default-mode fixes to one hash format version constant (ITL-898) Developer decision 2026-11-11: bump the hash format version. _FORMAT_VERSION (1 today) derives VERSION_BYTES and every pending default-mode fix, so moving to format 2 is a one-line change once the regenerated Rust spec and golden file land. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: look through dictionaries in option-on metadata paths; guard old-pyarrow golden decoding (ITL-898) - with any option on, Phase 2 metadata traversal looks through dictionary types (spec 2.2.1 paths are the 9.6 node paths); format 1 unchanged - golden vectors that pyarrow < 19 (Decimal32/64) or < 16 (view types) cannot decode are strict xfails on those versions only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: hash format 2, written from the spec, behind _FORMAT_VERSION (ITL-898) src/starfix/_format2.py implements nauticalab/starfix byte-layout-spec format 2 (041b8f1, provisional): 12-byte header, canonical type JSON with the name-free Map form, Phase 1/2 keyed by node path, extension types (storage type; ARROW:extension:* re-added for registered types), the one rule (entry = validity, lengths, values; row count after the schema digest), hash_array, per-batch schema check, rejections in every mode. Data paths work on Arrow buffers (filter, flatten, bitmaps), linear in rows. tests/test_format2.py checks the 7 worked examples, every final hash the spec prints (examples 1 and 5, Map, FixedSizeList, non-ASCII hash_array), primitive forms, JSON text rules, metadata paths, invariants, rejections, streaming and extension types, byte for byte from the spec text. ArrowDigester dispatches to format 2 when _FORMAT_VERSION == 2 (still 1: nothing changes until the design is approved and the golden lands). Pending spec answers E1-E6 are marked PENDING in the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): hash_array counts nulls on resolved dictionary values; offset tests (ITL-898) A valid dictionary key that points at a null dictionary value is a null element (pyarrow's DictionaryArray.null_count counts keys only), as Rust resolves it. Adds slicing tests at array offsets 7, 8, 9, 16, 63, 64, 65 for Boolean, Int32 and string leaves under a nullable struct and for a list of Booleans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(format2): deterministic scaling guards for list hashing (ITL-898, ITL-908) Ports the ITL-908 guards to format 2: hashing a list column never copies its shared values buffer (tracemalloc growth, isolated from caller tracing), and the number of node visits does not grow with the row count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(format2): final spec deltas; final golden files; inputs of unmapped literals (ITL-898) - struct child node path is always p + "/" + name (spec 3.1, R7) - absent ARROW:extension:metadata hashed as "" (spec 2.3, E1) - data hashing of unsupported types rejected before hashing, any row count, ValueError (spec 7 item 3, E4) - tests/golden/format2_v0.5.json and format1_to_format2.json copied byte for byte from starfix f7aa441 (sha256 1b6102f4..., 9d40973c...) - tests/golden/python_literals_format1_inputs.json: the inputs of the 69 hand-coded format-1 literals of this repo that format1_to_format2.json does not cover, recovered by recording the calls that produce each literal (all 69 reproduce their format-1 value from the IPC), for Rust to supply the format-2 values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat!: hash format 2 is the only format; delete format-1 code (ITL-898) BREAKING CHANGE: every hash changes; the prefix is 00 00 02 (format 2). - src/starfix/arrow_digester.py is a thin ArrowDigester over _format2 (keywords include_metadata, ordered_columns, ordered_struct_fields; no nested_validity); all format-1 code and the _FORMAT_VERSION / _FIX_* switches are deleted - removed format-1-only tests and the unreleased ordered_v0.5.json; test_golden_parity_metadata.py removed (include_metadata_v0.3.json is kept as the historical format-1 record, see tests/golden/README.md) - hand-coded literals moved to their Rust-produced format-2 values from format1_to_format2.json (39); the other 69 await Rust values for the inputs in python_literals_format1_inputs.json - tests/test_golden_parity_format2.py asserts every vector and relation of format2_v0.5.json (only version-gated skips for old pyarrow) - tests/test_api.py: keyword-only options, no third option - CI golden-sync-check diffs format2_v0.5.json and format1_to_format2.json Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: README, CHANGELOG and design note for hash format 2 (ITL-898) README documents format 2 (prefix, guarantees, options, rejections, format history); CHANGELOG lists the breaking format change, the fixed format-1 losses and Python/Rust differences, the ITL-908 performance fix and the new options; the superseded options design note is replaced by a format-2 note of the Python-specific decisions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: format2_v0.5.json at starfix d2b26c6 (canonical tensor vector) (ITL-898) Copied byte for byte (sha256 e7e1ccf1...). Golden parity: 254/254 vectors, 89/89 relations, no skips, no xfails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: format 2 module docstring (approved) (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: move the last 69 literals to Rust format-2 values; golden at starfix a4bd86d (ITL-898) format2_v0.5.json (323 vectors, 89 relations; sha256 afd7280d...) and format1_to_format2.json (203 entries; sha256 b06a259e...) copied byte for byte. Every hand-coded literal now carries its Rust-produced format-2 value (69 more via the recovered inputs). Parity 323/323 + 89/89, no skips, no xfails; whole suite green on Python 3.10-3.12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: fetch format-2 fixtures with the raw media type (no 1 MB limit) (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: report a missing upstream fixture as missing, not as drift (ITL-898) On a 404 gh api prints the error body to stdout; keep the fetched file only when the request succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build!: require pyarrow >= 19; remove old-pyarrow skips; CI job on pyarrow 19.0.0 (ITL-898) Developer decision 2026-11-11. Older pyarrow cannot read Decimal32/64 from Arrow IPC; with the floor at 19 every golden vector is decoded and matched on every supported version, so the version-gated skips (_pyarrow_compat, list_view, arrow.uuid) are removed. tests/test_golden_parity_r2.py builds its struct columns with the schema's field types (pyarrow 19 cannot cast nullable struct children to non-nullable ones); same data, same hashes. New CI job test-min-pyarrow runs the suite on Python 3.10 with exactly pyarrow 19.0.0. CHANGELOG and README state the requirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): atomic batch updates, dictionary-of-extension metadata, byte-bounded binary hashing (ITL-898) Copilot review on PR #9: - a batch rejected while traversed (unmasked null) no longer leaves earlier columns' entries updated: entries are snapshotted in O(nodes) (bit length, byte length, last partial byte, copies of the SHA-256 states) and restored on failure - a dictionary of a registered extension type keeps its ARROW:extension:* keys in Phase 2 (dictionaries are looked through, extensions are not) - variable-length values are serialized in updates of about 1 MiB; a value of at least that size is hashed straight from the data buffer - regression tests: unmasked nulls rejected (column, struct child, dictionary index to a null value), masked child null accepted, rollback with and without partial validity bytes, dictionary-of-extension metadata, byte budget Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): bounded memory for nullable leaves and finalization (ITL-898) Copilot review on PR #9: - nullable leaves are no longer filtered as a whole column: Booleans and fixed-width values are filtered in chunks (fixed-width chunks bounded by _BYTE_BUDGET bytes); strings and binary skip nulls while reading the original offsets/data buffers, so the valid payload is never copied - entries are fed to the final SHA-256 one by one (validity bits as a memoryview) instead of being joined into one byte string Same bytes: goldens and a chunk-size sweep test (budgets 1, 7, 64, 1 MiB) over nullable string, Int32, Boolean and FixedSizeBinary leaves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: legacy 0.4 mode, starfix.legacy.v0_4, byte-identical to starfix-python 0.4.0 (ITL-898) Developer decision 2026-11-11: keep the 0.4 behaviour available under an explicitly labelled legacy name; the main API stays format 2. - src/starfix/legacy/v0_4.py: starfix-python v0.4.0 src/starfix/arrow_digester.py (tag v0.4.0, d2ed97e) VERBATIM below a frozen-module header (its sha256 equals the PyPI starfix==0.4.0 module); stdlib + pyarrow imports only; nothing shared with format 2. starfix/legacy/__init__.py re-exports nothing. - tests/test_legacy_v0_4_*.py: the five v0.4.0 test files restored with their original format-1 values (import pointed at the legacy module; the r2 file builds struct inputs so pyarrow 19 can construct them), including the include_metadata_v0.3.json golden: 193 passed - tests/test_legacy_v0_4_frozen.py: sha256 pin of the legacy file, and no imports between the legacy and format-2 modules - scripts/legacy_differential.py + CI job legacy-differential: ~2800 results over 400 seeded random inputs and the Python-0.4 quirks, against the released starfix==0.4.0 in an isolated environment: 0 differ (a one-character mutation of the legacy file is caught) - README 'Legacy 0.4 mode' (purpose, never for new data, defects, Python-vs-Rust 0.4 divergence table), CHANGELOG, tests/golden/README.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: legacy 0.4 mode hashes list slices without copying the shared buffer (ITL-908) One performance-only line in src/starfix/legacy/v0_4.py, marked LEGACY PATCH: raw = memoryview(data_buf) instead of data_buf.to_pybytes() in _hash_fixed_size_data (the ITL-908 P1 fix). List columns now hash in linear time when migration reads large stores. No hash changes: the restored 0.4.0 tests pass and the differential test against the released starfix==0.4.0 still compares 2794 results with 0 differences. The header says the file differs from 0.4.0 by exactly this line; the freeze guard's sha256 is updated in this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): zero-width FixedSizeBinary contributes no value bytes (ITL-898) Copilot review: a nullable FixedSizeBinary(0) leaf divided by zero when sizing its chunks. Zero-width values have no payload; their validity is recorded as usual. Regression test at the root and under a nullable struct. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(format2): dictionary of List and hidden child nulls under non-nullable list items (ITL-898) Inputs from the independent Rust review: a dictionary whose values are a List hashes as the plain list (column, struct child, hash_array); child nulls under a null list slot or outside the offsets are not elements and are accepted; a visible null under a valid slot is rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: legacy separation guard resolves relative and package-level imports (ITL-898) Copilot review: the guard missed from .legacy.v0_4 import X, from . import legacy and from starfix import legacy. Relative imports are now resolved against the file's package and imported names are included; a self-test covers each form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): list children are zero-copy slices with a selection mask (ITL-898) Copilot review: a list with a null row (or under a null struct value) was filtered as a whole, which copies the child buffers (a second copy of a list<binary> payload). Now a list node writes the lengths of its present rows (value_lengths filtered as integers) and visits its child as a zero-copy slice of the original values with a selection mask take(present_rows, list_parent_indices): elements outside the selection are not elements (no validity, lengths or values), as the spec requires. Same bytes (goldens, spec examples, offset and hidden-null tests). Regression test on a fresh proxy of the Arrow memory pool (tracemalloc cannot see Arrow allocations): hashing a nullable list<binary> with a 4 MB payload peaks far below the payload (the old code peaked at 4.03 MB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(format2): entry-point kind in the header; hash_array root always nullable (ITL-898) Developer decision 2026-11-11 (provisional until the Rust spec and golden): - header = 0x00 || "starfix" || 00 00 02 || kind || flags (13 bytes), kind 'S' schema digest, 'R' record batch / stream / table, 'A' hash_array - the record-batch / stream final digest starts with its own 'R' header - the hash_array root is always nullable: its validity is always written Spec-level tests migrated to the new layout; the old printed hashes are replaced by byte-structure assertions until the new spec prints its values. Pins: nested dictionaries resolve recursively; FixedSizeBinary(0) 1 row != 5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): hash leaf dictionaries through their indices (bounded memory) (ITL-898) Copilot review: dictionary_decode() materialized the whole logical payload (10,000 indices to one 1 MiB value = 10 GiB). Leaf dictionaries now stay encoded: validity is index validity AND the validity of the value it points to (take over booleans), strings/binary are fed from the dictionary buffers through the indices, fixed-width and Boolean values are gathered in byte-bounded chunks. Nested dictionaries resolve inside the dictionary only; dictionaries of struct/list values are decoded (their children are visited). Hash-neutral: 1602/1602 identical against the previous module (400 seeded random batches with string and numeric dictionaries, plus dictionaries in lists, nested, of structs, with null values, sliced). Regression test on a proxy of the Arrow pool: 80 references to a 1 MiB value peak far below the decoded size (the old code peaked at 140 MB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: independent review of PR #9, behaviour items 2, 3, 9, 12, 14 (ITL-898) 2 an Arrow-valid empty list array without an offsets buffer segfaulted: empty lists no longer touch offsets or lengths (subprocess test, List and LargeList, top level and in a struct) 3 with include_metadata, duplicate metadata keys (field, struct child, schema) are rejected with ValueError naming the node and key; metadata is read through the Arrow C data interface, which keeps duplicates. Explicit ARROW:extension:* keys conflicting with a registered extension type are rejected; agreeing ones are accepted 9 wrong container types raise TypeError naming the expected type (hash_record_batch(Table), hash_table(RecordBatch), update(Table), hash_schema / ArrowDigester with a non-schema); hash_array accepts a ChunkedArray, hashed chunk by chunk, equal to the combined array 12 legacy: the performance-only line keeps 0.4.0's AttributeError for a missing values buffer (differential now 2795 results, 4 raising the same exception, 0 differ); r2 test docstring states the input helpers 14 _Bits appends replace the partial last byte in one slice assignment, and restore truncates and resets the byte at its saved index Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs/test: independent review of PR #9, items 4, 5, 7, 8, 10, 11, 13, 15 (ITL-898) 4 README/CHANGELOG: extension types are hashed as stored; exact only where pyarrow can observe the stored form (list of non-canonical cases; a Python-defined type's serializer must reproduce the stored bytes); the 'byte-identical to Rust' claim names this one exception 5 ArrowDigester docstring: ValueError (not NotImplementedError) for unsupported data, plus every rejection; TypeError for wrong containers; not thread-safe; hash_array deliberately without include_metadata 7/8/13 README Performance section (linear; strings ~3-5M rows/s; a few us per column per update); _visit docstring states what is still allocated 10 golden test asserts minimum vector/relation counts; pytest empty_parameter_set_mark = fail_at_collect 11 test_format2.py: expectations no longer taken from the implementation (hand-written Phase 1 / type JSON); docstring says what is independent; stale notes removed 15 CI: Python 3.13 in the test matrix; golden-sync also diffs python_literals_format1_inputs.json (identical to the Rust copy) CHANGELOG: the version comes from the git tag at release Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: format2_v0.5.json at starfix b5398ce (entry-point kind, hash_array root); literals remapped (ITL-898) format2_v0.5.json (339 vectors, 96 relations; sha256 e049bc22...) and format1_to_format2.json (203 entries, unique ids; sha256 bda80b0b...) copied byte for byte from starfix b5398ce. Every hand-coded literal (108) moved to its new Rust value (joined on the format-1 hash). Parity 339/339 vectors and 96/96 relations, no skips, no xfails; minimum counts raised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: point the dictionary-of-List, FixedSizeBinary(0) and hidden-null tests at their golden vectors (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: pin the spec's printed hashes of Examples 1 and 5 (starfix d145d89) (ITL-898) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: read exported metadata through __arrow_c_schema__; pin all spec-printed hashes (starfix bcbe74a) (ITL-898) - metadata pairs (duplicate detection, extension keys of pyarrow-native types) are read from the PyCapsule returned by __arrow_c_schema__(), the documented Arrow C data interface route in spec 2.3, instead of the private _export_to_c; the capsule releases the exported schema - Map, FixedSizeList and non-ASCII hash_array printed hashes pinned (all five printed values of spec bcbe74a now pinned) - README extension-type wording aligned with spec 2.3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: design note describes the legacy 0.4 module and the no-skip golden contract (ITL-898) Copilot review: the Python design note still said no format-1 code is kept and described version-gated golden-test skips that were removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): refuse big-endian hosts; tests for decoded dictionary validity at depth and the FixedSizeList element path (ITL-898) Spec re-read at starfix 9c9a88c: - Section 13: hashes are defined over little-endian buffers; values are hashed as stored, so every format-2 entry point raises NotImplementedError on a big-endian host (README 'Platforms') - dictionary validity is that of the decoded array at any depth: tested for a column, a struct child and a list element - the FixedSizeList element node f/ claims its path: tested Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(format2): dictionary validity without scanning a null-free dictionary per batch (ITL-898) Copilot review (needs a closer look): computing dictionary validity took the dictionary's validity per batch, O(batches x dictionary size) for many small batches. When the dictionary has no nulls, validity is the indices' (identical bytes). Regression test: 50 one-row batches over a 10,000-value dictionary make no take() over the dictionary validity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(format2): sync golden to starfix e079d56 and pin hidden-null cases Copy tests/golden/format2_v0.5.json byte for byte from starfix e079d56 (sha256 4d5ba202f9fa88e69c94c09947e715c51c5403803b9fa0dfc3ec7dda9582845e; 423 vectors, 114 relations, the previous 339/96 an unchanged prefix) and raise the pinned minimum counts. format1_to_format2.json is unchanged. Add golden-independent tests for the two shapes the Rust review found: a leaf null hidden under a null inner list of a nested list (hash_array, one batch and single-row batches agree with the clean form), and a dictionary of nested structs with a null hidden under a null outer value (dictionary-encoded equals plain), under every option combination. Refs ITL-898 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(format2): hash decimal32/decimal64 with nulls on pyarrow 19/20 (ITL-898) pyarrow 19.0.x and 20.0.0 have no `filter` kernel for decimal32/decimal64, so dropping nulls from those arrays (directly, or nested under list/struct/dictionary of list or struct) raised ArrowNotImplementedError on the supported minimum pyarrow. Fixed-width values are now selected on a zero-copy view of their value buffer as an unsigned integer of the same width (or FixedSizeBinary(width) for wider values), so no kernel of the logical type is used. The bytes selected are unchanged: no hash changes (all 423 golden vectors and 114 relations match with no skips). tests/test_format2_type_matrix.py runs every supported leaf type, with and without nulls, sliced, chunked, batched, dictionary-encoded and nested, through hash_array, record batch and stream hashing, against the Rust golden vectors where they exist and across the equivalent forms otherwise. CI runs the suite on pyarrow 19.0.0, 20.0.0 and the latest, and the pytest run asserts the pyarrow it imports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(format2): field metadata errors get their path from the caller; README memory note (ITL-898) _field_metadata no longer receives the node path only to build error messages: it raises _AmbiguousMetadata (a ValueError) and phase2_json re-raises it as ValueError prefixed with the field path ("Field 's/c': Duplicate metadata key 'a' on the field: ..."). Tests assert the full messages, including the path. README: auxiliary memory is not bounded (validity, lengths and list parent indices per visited element; dictionary-of-struct/list values are materialized with take); only leaf payload feeding is chunked. Also the cheap ruff findings in _format2.py (sorted __slots__, unquoted annotation, itertools.pairwise). No hash changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(format2): sync golden to starfix e3d5780 (73 FixedSizeList vectors, 34 relations) (ITL-927) Copied byte for byte from nauticalab/starfix tests/golden/format2_v0.5.json (sha256 0cd11ea872a64b15db6ae18394b7f606aac21eba65267bb316dd0b9758e7a989). Floors raised to 496 vectors and 148 relations. The FixedSizeList data vectors fail until FixedSizeList data is hashed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(format2): hash FixedSizeList data (ITL-927) A fixed-size-list node's entry is its effective validity if nullable and empty otherwise (no lengths digest: n is in the schema). Its element node gets the n items of every effectively valid slot; items of null or struct-masked slots and child positions past len * n are never visited, and struct masking restarts below the list, as for List. Slot i covers values[(offset + i) * n, (offset + i + 1) * n): pyarrow's FixedSizeListArray.values ignores the array offset. Without null slots the items are one zero-copy slice; otherwise slots are visited in chunks of about 1M items, with the slot bits repeated n times as the selection (list_parent_indices skips null fixed-size-list slots, so it runs on a validity-free FixedSizeList of nulls instead). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(format2): FixedSizeList in the type matrix; docs for FixedSizeList data (ITL-927) The type matrix wraps every supported leaf type in fixed-size lists of size 0, 1 and 2 (a null slot, trailing child positions), nested in list / struct / dictionary and containing struct / list / fixed-size list, each compared with a reference that holds other content under the null slot and no trailing positions. README, CHANGELOG (0.5.0 section) and the arrow_digester docstring no longer list FixedSizeList as unsupported; design note added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(format2): reflow docstrings touched by FixedSizeList support (ITL-927) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): release section 0.5.0 (2026-11-11); FixedSizeList items in the rejected-data entry (ITL-927) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Edgar Y. Walker <> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>



Warning
Merge only together with the Rust PR (metamorphic-labs/starfix#20). CI's
golden-sync-checkcompares this repo's golden files with starfixmainand passes once #20 is merged. The developer merges and releases.Summary
ITL-898. starfix-python 0.5 hashes Arrow data with hash format 2, byte-identical to the Rust starfix crate 0.5 (metamorphic-labs/starfix#20). The developer approved format 2 on 2026-11-11.
Every hash changes. The 35-byte hash starts with the format number
00 00 02, so it can never be confused with a format-1 hash (00 00 01, starfix ≤ 0.4).Lossless layout. Every nullable node records its validity, every list its lengths, Booleans hash one byte per value, the row count is hashed, and the
hash_arrayroot's validity is always written. This fixes the format-1 collisions listed inCHANGELOG.md.Entry-point marker. A 13-byte header (
0x00 "starfix" 00 00 02 KIND FLAGS, withKIND=Sschema,Rrecord batch / stream / table,Aarray) separates entry points and option sets.Options.
ordered_columnsandordered_struct_fieldsare keyword-only and default off, next toinclude_metadata.Rejections, in every mode (
ValueError):include_metadata;A wrong container type raises
TypeError.Python matches Rust everywhere. The format-1 Python/Rust differences are gone (Map JSON, non-ASCII names in
hash_array, map metadata paths, Null/Interval/view type forms, per-batch check, list of numeric dictionaries hashing indices). The one documented exception is extension-type metadata that pyarrow cannot observe (see "Extension types" in the README).Legacy 0.4 mode.
starfix.legacy.v0_4reproduces starfix-python 0.4.0 byte for byte, for reading or matching data hashed by 0.4.x only.pyarrow ≥ 19 is required (was ≥ 14).
Implementation
src/starfix/_format2.pydocs/byte-layout-spec.md). Works on Arrow buffers: each node is visited once per batch, list children are zero-copy slices with a selection mask, leaf dictionaries are hashed through their indices, and payloads go out in byte-bounded chunks. Hashing time is linear in the data. A rejected batch leaves the digester unchanged.src/starfix/arrow_digester.pyArrowDigester(format 2 only).src/starfix/legacy/v0_4.pytests/golden/format2_v0.5.json(423 vectors, 114 relations; sha2564d5ba202f9fa88e69c94c09947e715c51c5403803b9fa0dfc3ec7dda9582845e) andformat1_to_format2.json(sha256bda80b0b4f83ac37152f7d41af4681c17874de308f9bca6459c8c84abc454e61), both copied byte for byte from starfixe079d56(head of metamorphic-labs/starfix#20);python_literals_format1_inputs.json;include_metadata_v0.3.json(format 1, asserted by the legacy tests); a README.test-min-pyarrowruns exactly pyarrow 19.0.0.legacy-differentialcompares against PyPIstarfix==0.4.0.golden-sync-checkchecks the three shared golden files against starfixmain.CHANGELOG.md,docs/metamorphic/specs/2026-11-11-format-2-python.md.Verification
List<List<Int32 non-null>>(hash_array, one batch and three single-row batches all equal the clean form), andDictionary<Int8, Struct<child: Struct<i: Int32 non-null> non-null>>with a null hidden under a null outer value (dictionary-encoded equals plain), under every option combination.tests/test_format2.py): the expected bytes are written out from the spec's rules and examples, independent of the implementation.starfix==0.4.0compares 2,795 results with 0 differences, including 4 cases that raise the same exceptions;tracemalloc: no copy of list payloads, dictionary payloads, or shared buffers, and node visits don't grow with rows.Version
Proposed: 0.5.0, matching the Rust crate. The version comes from the git tag at release (hatch-vcs). Every hash changes, which is a breaking change, flagged prominently in the README and CHANGELOG. Format-1 hashes stay available through
starfix.legacy.v0_4. Nothing is bumped or released here.🤖 Generated with Claude Code