Skip to content

feat!: hash format 2, byte-identical to Rust; ordered_columns / ordered_struct_fields (ITL-898) - #9

Merged
eywalker merged 49 commits into
mainfrom
eywalker/itl-898-starfix-optional-order-sensitive-hashing
Nov 11, 2026
Merged

eywalker merged 49 commits into
mainfrom
eywalker/itl-898-starfix-optional-order-sensitive-hashing

Conversation

@eywalker

@eywalker eywalker commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Merge only together with the Rust PR (metamorphic-labs/starfix#20). CI's golden-sync-check compares this repo's golden files with starfix main and passes once #20 is merged. The developer merges and releases.

Summary

ITL-898. starfix-python 0.5 hashes Arrow data with hash format 2, byte-identical to the Rust starfix crate 0.5 (metamorphic-labs/starfix#20). The developer approved format 2 on 2026-11-11.

  • Every hash changes. The 35-byte hash starts with the format number 00 00 02, so it can never be confused with a format-1 hash (00 00 01, starfix ≤ 0.4).

  • Lossless layout. Every nullable node records its validity, every list its lengths, Booleans hash one byte per value, the row count is hashed, and the hash_array root's validity is always written. This fixes the format-1 collisions listed in CHANGELOG.md.

  • Entry-point marker. A 13-byte header (0x00 "starfix" 00 00 02 KIND FLAGS, with KIND = S schema, R record batch / stream / table, A array) separates entry points and option sets.

  • Options. ordered_columns and ordered_struct_fields are keyword-only and default off, next to include_metadata.

  • Rejections, in every mode (ValueError):

    • ambiguous node paths;
    • ListView, LargeListView, Union and RunEndEncoded types;
    • data of types whose data is unsupported, whatever the row count;
    • a non-nullable field holding an unmasked null;
    • duplicate metadata keys with include_metadata;
    • a streamed batch whose schema doesn't match.

    A wrong container type raises TypeError.

  • Python matches Rust everywhere. The format-1 Python/Rust differences are gone (Map JSON, non-ASCII names in hash_array, map metadata paths, Null/Interval/view type forms, per-batch check, list of numeric dictionaries hashing indices). The one documented exception is extension-type metadata that pyarrow cannot observe (see "Extension types" in the README).

  • Legacy 0.4 mode. starfix.legacy.v0_4 reproduces starfix-python 0.4.0 byte for byte, for reading or matching data hashed by 0.4.x only.

  • pyarrow ≥ 19 is required (was ≥ 14).

Implementation

Area Change
src/starfix/_format2.py Format 2, written from the spec text (starfix docs/byte-layout-spec.md). Works on Arrow buffers: each node is visited once per batch, list children are zero-copy slices with a selection mask, leaf dictionaries are hashed through their indices, and payloads go out in byte-bounded chunks. Hashing time is linear in the data. A rejected batch leaves the digester unchanged.
src/starfix/arrow_digester.py The public ArrowDigester (format 2 only).
src/starfix/legacy/v0_4.py The starfix-python v0.4.0 hasher, verbatim except one marked performance-only line.
tests/golden/ format2_v0.5.json (423 vectors, 114 relations; sha256 4d5ba202f9fa88e69c94c09947e715c51c5403803b9fa0dfc3ec7dda9582845e) and format1_to_format2.json (sha256 bda80b0b4f83ac37152f7d41af4681c17874de308f9bca6459c8c84abc454e61), both copied byte for byte from starfix e079d56 (head of metamorphic-labs/starfix#20); python_literals_format1_inputs.json; include_metadata_v0.3.json (format 1, asserted by the legacy tests); a README.
CI Tests on Python 3.10–3.13. test-min-pyarrow runs exactly pyarrow 19.0.0. legacy-differential compares against PyPI starfix==0.4.0. golden-sync-check checks the three shared golden files against starfix main.
docs README (format 2, options, rejections, extension types, performance, hash format history, legacy mode), CHANGELOG.md, docs/metamorphic/specs/2026-11-11-format-2-python.md.

Verification

  • Golden parity: 423 of 423 vectors and 114 of 114 relations, with no skips and no xfails. The test asserts minimum counts, and an empty parametrization fails.
  • Hidden nulls: golden-independent tests pin the two shapes the Rust review found: a leaf null hidden under a null inner list of List<List<Int32 non-null>> (hash_array, one batch and three single-row batches all equal the clean form), and Dictionary<Int8, Struct<child: Struct<i: Int32 non-null> non-null>> with a null hidden under a null outer value (dictionary-encoded equals plain), under every option combination.
  • Literal tests: all 108 hand-coded literals carry Rust-produced format-2 values. For the 69 literals whose old Rust emitters were gone, the inputs were recovered by recording the calls that produced them, and Rust computed the values.
  • Spec-level tests (tests/test_format2.py): the expected bytes are written out from the spec's rules and examples, independent of the implementation.
  • Legacy:
    • the restored v0.4.0 tests (193) pass;
    • a differential test against the released starfix==0.4.0 compares 2,795 results with 0 differences, including 4 cases that raise the same exceptions;
    • a freeze guard pins the legacy file's sha256 and the separation of imports between the legacy and format-2 modules.
  • Memory and scaling guards use Arrow memory-pool proxies and tracemalloc: no copy of list payloads, dictionary payloads, or shared buffers, and node visits don't grow with rows.
  • Whole suite: 917 passed on Python 3.10, 3.11, 3.12 and 3.13, and on pyarrow 19.0.0.
  • Reviews: every Copilot finding and every finding of the independent review is fixed with tests or documented, and answered. The independent review's differential fuzzing found no Python/Rust difference on hashable inputs, except the two documented areas, both now addressed: duplicate metadata keys are rejected, and extension-type caveats are documented.

Version

Proposed: 0.5.0, matching the Rust crate. The version comes from the git tag at release (hatch-vcs). Every hash changes, which is a breaking change, flagged prominently in the README and CHANGELOG. Format-1 hashes stay available through starfix.legacy.v0_4. Nothing is bumped or released here.

🤖 Generated with Claude Code

Edgar Y. Walker and others added 5 commits November 11, 2026 02:43
…ITL-898)

Byte-level tests for spec Section 9 worked examples T1-T5, Phase 1 JSON
for every flags value, invariants I1-I8, node-path rejection, streaming
batch checks and keyword-only options. Adds the option table only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ons (ITL-898)

Implements spec Section 9 of the starfix byte layout: options tag,
flags-aware Phase 1 / hash_array type JSON, nested validity entries,
node-path rejection and the per-batch schema check when streaming.
With all options off every code path and output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k (ITL-898)

The parity test runs every vector and every relation of the Rust golden
file (to be copied in unchanged). README documents the three options.
The golden-sync check also diffs ordered_v0.5.json against starfix main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th options; golden file (ITL-898)

- nested_validity feeds one byte per valid Boolean value (spec 9.5.1, T6)
- with any option on, Map serializes its entries field and keys_sorted
  as Rust does; the default (flags 0) form is unchanged
- tests/golden/ordered_v0.5.json copied byte for byte from starfix
  (sha256 ff86795d...)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…design note (ITL-898)

Default Map JSON and the default hash_array non-ASCII escaping differ from
Rust in released starfix-python. They are left unchanged (moving released
hashes is the developer's decision) and pinned as xfail(strict=True).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Edgar Y. Walker and others added 5 commits November 11, 2026 02:52
…e (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…c 2.1 and 9.7 tests (ITL-898)

- re-copy tests/golden/ordered_v0.5.json byte for byte (sha256 12deacf4...)
- names_array_struct_f0 / _int8_f0 are the known default-mode non-ASCII
  hash_array divergence (strict xfail)
- spec 2.1 FixedSizeList and Map examples, JSON text rules, 9.7 metadata check
- 9.7 default-mode batch check (Rust checks at flags 0, Python does not):
  pinned as strict xfails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The Boolean digest path materializes the entire column as a Python list and byte string, causing roughly 900 MB of temporary storage for 100 million valid values instead of hashing bounded chunks.

1 open issue
What changed in this PR

Adds opt-in order-sensitive and nested-validity hashing to the pure-Python ArrowDigester, targeting the provisional Rust v0.5 layout while preserving default hashes.

Changes:

  • Adds three keyword-only options, tagged hashing, ambiguous-path rejection, and streaming schema validation.
  • Adds byte-level, invariant, and Rust golden-parity tests, with strict expected failures for known default-mode divergences.
  • Documents the options and extends CI to check the ordered fixture against Rust upstream.
File Description
tests/​test_ordered_hashing.py Tests option layouts, invariants, validation, and streaming behavior.
tests/​test_known_divergences.py Pins known default-mode differences from Rust.
tests/​test_golden_parity_ordered.py Checks ordered golden vectors and hash relations.
src/​starfix/​arrow_digester.py Implements hashing options and schema validation.
README.md Explains option usage and compatibility behavior.
docs/​metamorphic/​specs/​2026-11-11-ordered-hashing-design.md Records provisional Python design decisions.
.github/​workflows/​ci.yml Adds upstream synchronization checks for the ordered fixture.

Review effort: Balanced

Comment thread src/starfix/arrow_digester.py Resolved
@eywalker
eywalker marked this pull request as ready for review November 11, 2026 10:57
@eywalker
eywalker requested a balanced review from Copilot November 11, 2026 10:57
Expanding to one byte per value now works on 64Ki-element chunks via
pyarrow compute (filter, cast to uint8) instead of materializing the
whole column as a Python list and byte string. The digest input bytes
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The README overstates Rust compatibility for default calls and omits how nested_validity prevents Boolean trailing-False collisions and batch-split-dependent hashes.

3 open issues

Review effort: Balanced

Comment thread README.md Resolved
Comment thread README.md Resolved
…golden at starfix 544cb1b (ITL-898)

- golden file re-copied byte for byte (226 vectors, 77 relations; sha256 e21dd703...)
- with any option on: Null, Interval, Utf8View, BinaryView schema forms and
  Rust map metadata paths (entries m/, key m//key, value m//value)
- default-mode misses are strict xfails; map entries-field metadata vectors
  are strict xfails because pyarrow drops that metadata on IPC read
- README: qualify Rust parity for default calls; document the Boolean
  effect of nested_validity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Edgar Y. Walker and others added 2 commits November 11, 2026 03:07
…ess-fix scope test (ITL-898)

Developer ruling 2026-11-11: the lossless layout becomes the default fix.
Prepares that change and the other pending default-mode decisions as
isolated module switches (all off; behaviour unchanged):
_LOSSLESS_BY_DEFAULT, _FIX_DEFAULT_MAP_JSON, _FIX_DEFAULT_TYPE_JSON_UTF8,
_FIX_DEFAULT_TYPE_FORMS, _FIX_DEFAULT_MAP_METADATA_PATHS,
_FIX_DEFAULT_BATCH_CHECK. Tests show each flipped switch reproduces the
Rust golden values, and that every golden input the 'unaffected' rule
covers hashes identically under the lossless layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(ITL-898)

Developer decision 2026-11-11: bump the hash format version. _FORMAT_VERSION
(1 today) derives VERSION_BYTES and every pending default-mode fix, so
moving to format 2 is a one-line change once the regenerated Rust spec and
golden file land. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The golden suite fails on supported PyArrow 14 without decimal-schema capability guards, while option-enabled metadata traversal can miss changes due to path collisions and dictionary-wrapped types.

2 open issues
3 resolved since last review

Review effort: Balanced

Comment thread tests/test_golden_parity_ordered.py Resolved
Comment thread src/starfix/arrow_digester.py Resolved
Edgar Y. Walker and others added 6 commits November 11, 2026 03:15
…-pyarrow golden decoding (ITL-898)

- with any option on, Phase 2 metadata traversal looks through dictionary
  types (spec 2.2.1 paths are the 9.6 node paths); format 1 unchanged
- golden vectors that pyarrow < 19 (Decimal32/64) or < 16 (view types)
  cannot decode are strict xfails on those versions only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TL-898)

src/starfix/_format2.py implements nauticalab/starfix byte-layout-spec
format 2 (041b8f1, provisional): 12-byte header, canonical type JSON with
the name-free Map form, Phase 1/2 keyed by node path, extension types
(storage type; ARROW:extension:* re-added for registered types), the one
rule (entry = validity, lengths, values; row count after the schema
digest), hash_array, per-batch schema check, rejections in every mode.
Data paths work on Arrow buffers (filter, flatten, bitmaps), linear in rows.

tests/test_format2.py checks the 7 worked examples, every final hash the
spec prints (examples 1 and 5, Map, FixedSizeList, non-ASCII hash_array),
primitive forms, JSON text rules, metadata paths, invariants, rejections,
streaming and extension types, byte for byte from the spec text.

ArrowDigester dispatches to format 2 when _FORMAT_VERSION == 2 (still 1:
nothing changes until the design is approved and the golden lands).
Pending spec answers E1-E6 are marked PENDING in the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…offset tests (ITL-898)

A valid dictionary key that points at a null dictionary value is a null
element (pyarrow's DictionaryArray.null_count counts keys only), as Rust
resolves it. Adds slicing tests at array offsets 7, 8, 9, 16, 63, 64, 65
for Boolean, Int32 and string leaves under a nullable struct and for a
list of Booleans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, ITL-908)

Ports the ITL-908 guards to format 2: hashing a list column never copies
its shared values buffer (tracemalloc growth, isolated from caller
tracing), and the number of node visits does not grow with the row count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ped literals (ITL-898)

- struct child node path is always p + "/" + name (spec 3.1, R7)
- absent ARROW:extension:metadata hashed as "" (spec 2.3, E1)
- data hashing of unsupported types rejected before hashing, any row count,
  ValueError (spec 7 item 3, E4)
- tests/golden/format2_v0.5.json and format1_to_format2.json copied byte for
  byte from starfix f7aa441 (sha256 1b6102f4..., 9d40973c...)
- tests/golden/python_literals_format1_inputs.json: the inputs of the 69
  hand-coded format-1 literals of this repo that format1_to_format2.json does
  not cover, recovered by recording the calls that produce each literal (all
  69 reproduce their format-1 value from the IPC), for Rust to supply the
  format-2 values

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BREAKING CHANGE: every hash changes; the prefix is 00 00 02 (format 2).

- src/starfix/arrow_digester.py is a thin ArrowDigester over _format2
  (keywords include_metadata, ordered_columns, ordered_struct_fields;
  no nested_validity); all format-1 code and the _FORMAT_VERSION / _FIX_*
  switches are deleted
- removed format-1-only tests and the unreleased ordered_v0.5.json;
  test_golden_parity_metadata.py removed (include_metadata_v0.3.json is
  kept as the historical format-1 record, see tests/golden/README.md)
- hand-coded literals moved to their Rust-produced format-2 values from
  format1_to_format2.json (39); the other 69 await Rust values for the
  inputs in python_literals_format1_inputs.json
- tests/test_golden_parity_format2.py asserts every vector and relation of
  format2_v0.5.json (only version-gated skips for old pyarrow)
- tests/test_api.py: keyword-only options, no third option
- CI golden-sync-check diffs format2_v0.5.json and format1_to_format2.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l tests at their golden vectors (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Edgar Y. Walker and others added 2 commits November 11, 2026 05:13
…d89) (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…printed hashes (starfix bcbe74a) (ITL-898)

- metadata pairs (duplicate detection, extension keys of pyarrow-native
  types) are read from the PyCapsule returned by __arrow_c_schema__(), the
  documented Arrow C data interface route in spec 2.3, instead of the
  private _export_to_c; the capsule releases the exported schema
- Map, FixedSizeList and non-ASCII hash_array printed hashes pinned (all
  five printed values of spec bcbe74a now pinned)
- README extension-type wording aligned with spec 2.3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The implementation record incorrectly says no format-1 code is retained and describes version-gated golden-test skips that have been removed.

3 open issues
2 resolved since last review

Review effort: Balanced

…den contract (ITL-898)

Copilot review: the Python design note still said no format-1 code is kept
and described version-gated golden-test skips that were removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alidity at depth and the FixedSizeList element path (ITL-898)

Spec re-read at starfix 9c9a88c:
- Section 13: hashes are defined over little-endian buffers; values are
  hashed as stored, so every format-2 entry point raises
  NotImplementedError on a big-endian host (README 'Platforms')
- dictionary validity is that of the decoded array at any depth: tested for
  a column, a struct child and a list element
- the FixedSizeList element node f/ claims its path: tested

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to weigh an unpublished performance finding where dictionary-validity bitmap work can grow to O(N × D) with one-row batches.

1 open issue
2 resolved since last review

Review effort: Balanced

…onary per batch (ITL-898)

Copilot review (needs a closer look): computing dictionary validity took
the dictionary's validity per batch, O(batches x dictionary size) for many
small batches. When the dictionary has no nulls, validity is the indices'
(identical bytes). Regression test: 50 one-row batches over a 10,000-value
dictionary make no take() over the dictionary validity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because an unpublished finding identifies repeated full-dictionary validity scans for nullable shared dictionaries, a scaling risk not covered by the new guard.

1 open issue

Review effort: Balanced

@eywalker

Copy link
Copy Markdown
Contributor Author

Copilot's review of 98b38b1 had no inline findings. Its summary notes one scaling edge, recorded here as a known limitation and not changed in this PR:

  • Dictionaries whose values include nulls, shared across many small batches: validity is still computed against the full dictionary once per batch, so the cost is O(batches × dictionary size) in that case.
  • Null-free dictionaries (the common case, e.g. categorical columns) no longer scan the dictionary at all (98b38b1, with a regression test).

The pure-Python package is planned to be replaced by a wrapper around the Rust crate, and the developer ruled out further pure-Python performance work. The cached-dictionary-validity optimisation for this rare input is left for later. Hashes are not affected either way.

Posted by a Claude Code agent on behalf of @eywalker.

Copy tests/golden/format2_v0.5.json byte for byte from starfix e079d56
(sha256 4d5ba202f9fa88e69c94c09947e715c51c5403803b9fa0dfc3ec7dda9582845e;
423 vectors, 114 relations, the previous 339/96 an unchanged prefix) and
raise the pinned minimum counts. format1_to_format2.json is unchanged.

Add golden-independent tests for the two shapes the Rust review found:
a leaf null hidden under a null inner list of a nested list (hash_array,
one batch and single-row batches agree with the clean form), and a
dictionary of nested structs with a null hidden under a null outer value
(dictionary-encoded equals plain), under every option combination.

Refs ITL-898

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A shared dictionary containing any null value triggers a full validity-array rebuild for every batch, causing O(N × D) work and leaving null-containing dictionaries uncovered by the scaling guard.

2 open issues

Review effort: Balanced

Comment thread src/starfix/_format2.py Resolved
Edgar Y. Walker and others added 2 commits November 11, 2026 11:33
…TL-898)

pyarrow 19.0.x and 20.0.0 have no `filter` kernel for decimal32/decimal64, so dropping
nulls from those arrays (directly, or nested under list/struct/dictionary of list or
struct) raised ArrowNotImplementedError on the supported minimum pyarrow.

Fixed-width values are now selected on a zero-copy view of their value buffer as an
unsigned integer of the same width (or FixedSizeBinary(width) for wider values), so no
kernel of the logical type is used. The bytes selected are unchanged: no hash changes
(all 423 golden vectors and 114 relations match with no skips).

tests/test_format2_type_matrix.py runs every supported leaf type, with and without nulls,
sliced, chunked, batched, dictionary-encoded and nested, through hash_array, record batch
and stream hashing, against the Rust golden vectors where they exist and across the
equivalent forms otherwise. CI runs the suite on pyarrow 19.0.0, 20.0.0 and the latest,
and the pytest run asserts the pyarrow it imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er; README memory note (ITL-898)

_field_metadata no longer receives the node path only to build error messages: it raises
_AmbiguousMetadata (a ValueError) and phase2_json re-raises it as ValueError prefixed with
the field path ("Field 's/c': Duplicate metadata key 'a' on the field: ..."). Tests assert
the full messages, including the path.

README: auxiliary memory is not bounded (validity, lengths and list parent indices per
visited element; dictionary-of-struct/list values are materialized with take); only leaf
payload feeding is chunked.

Also the cheap ruff findings in _format2.py (sorted __slots__, unquoted annotation,
itertools.pairwise). No hash changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The breaking hash-format migration depends on the still-open metamorphic-labs/starfix#20 and requires maintainer validation of cross-repository fixture parity, Python/Arrow CI results, and coordinated merge/release sequencing.

1 open issue
1 resolved since last review

Review effort: Balanced

@eywalker
eywalker merged commit e0f0cdc into main Nov 11, 2026
25 of 26 checks passed
@eywalker
eywalker deleted the eywalker/itl-898-starfix-optional-order-sensitive-hashing branch November 11, 2026 23:08
eywalker pushed a commit that referenced this pull request Nov 11, 2026
origin/main's tree is byte-identical to this branch's last #9 commit
(940887c), so every add/add and content conflict (CHANGELOG.md, README.md,
src/starfix/_format2.py, src/starfix/arrow_digester.py,
tests/golden/format2_v0.5.json, tests/test_format2.py,
tests/test_format2_type_matrix.py, tests/test_golden_parity_format2.py)
resolves to main's content plus this branch's FixedSizeList changes, which
is this branch's side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
eywalker added a commit that referenced this pull request Nov 11, 2026
* test: failing tests for ordered and nested-validity hashing options (ITL-898)

Byte-level tests for spec Section 9 worked examples T1-T5, Phase 1 JSON
for every flags value, invariants I1-I8, node-path rejection, streaming
batch checks and keyword-only options. Adds the option table only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: ordered_columns, ordered_struct_fields and nested_validity options (ITL-898)

Implements spec Section 9 of the starfix byte layout: options tag,
flags-aware Phase 1 / hash_array type JSON, nested validity entries,
node-path rejection and the per-batch schema check when streaming.
With all options off every code path and output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: golden parity test for ordered_v0.5.json; docs and CI sync check (ITL-898)

The parity test runs every vector and every relation of the Rust golden
file (to be copied in unchanged). README documents the three options.
The golden-sync check also diffs ordered_v0.5.json against starfix main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: Boolean one-byte layout under nested_validity; Rust Map form with options; golden file (ITL-898)

- nested_validity feeds one byte per valid Boolean value (spec 9.5.1, T6)
- with any option on, Map serializes its entries field and keys_sorted
  as Rust does; the default (flags 0) form is unchanged
- tests/golden/ordered_v0.5.json copied byte for byte from starfix
  (sha256 ff86795d...)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: pin known default-mode divergences from Rust as strict xfails; design note (ITL-898)

Default Map JSON and the default hash_array non-ASCII escaping differ from
Rust in released starfix-python. They are left unchanged (moving released
hashes is the developer's decision) and pinned as xfail(strict=True).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: let the ordered fixture drift step explain a missing upstream file (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: golden file at starfix e5b49c4 (171 vectors, 73 relations); spec 2.1 and 9.7 tests (ITL-898)

- re-copy tests/golden/ordered_v0.5.json byte for byte (sha256 12deacf4...)
- names_array_struct_f0 / _int8_f0 are the known default-mode non-ASCII
  hash_array divergence (strict xfail)
- spec 2.1 FixedSizeList and Map examples, JSON text rules, 9.7 metadata check
- 9.7 default-mode batch check (Rust checks at flags 0, Python does not):
  pinned as strict xfails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: list all three default-mode divergences in the design note (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: show the escaped form in the divergence table (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: tidy ordered hashing tests (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf: hash Booleans under nested_validity in bounded chunks (ITL-898)

Expanding to one byte per value now works on 64Ki-element chunks via
pyarrow compute (filter, cast to uint8) instead of materializing the
whole column as a Python list and byte string. The digest input bytes
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: spec 2.1 type forms and 2.2.1 map metadata paths with options; golden at starfix 544cb1b (ITL-898)

- golden file re-copied byte for byte (226 vectors, 77 relations; sha256 e21dd703...)
- with any option on: Null, Interval, Utf8View, BinaryView schema forms and
  Rust map metadata paths (entries m/, key m//key, value m//value)
- default-mode misses are strict xfails; map entries-field metadata vectors
  are strict xfails because pyarrow drops that metadata on IPC read
- README: qualify Rust parity for default calls; document the Boolean
  effect of nested_validity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: one-line switches for pending default-mode decisions; lossless-fix scope test (ITL-898)

Developer ruling 2026-11-11: the lossless layout becomes the default fix.
Prepares that change and the other pending default-mode decisions as
isolated module switches (all off; behaviour unchanged):
_LOSSLESS_BY_DEFAULT, _FIX_DEFAULT_MAP_JSON, _FIX_DEFAULT_TYPE_JSON_UTF8,
_FIX_DEFAULT_TYPE_FORMS, _FIX_DEFAULT_MAP_METADATA_PATHS,
_FIX_DEFAULT_BATCH_CHECK. Tests show each flipped switch reproduces the
Rust golden values, and that every golden input the 'unaffected' rule
covers hashes identically under the lossless layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: tie default-mode fixes to one hash format version constant (ITL-898)

Developer decision 2026-11-11: bump the hash format version. _FORMAT_VERSION
(1 today) derives VERSION_BYTES and every pending default-mode fix, so
moving to format 2 is a one-line change once the regenerated Rust spec and
golden file land. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: look through dictionaries in option-on metadata paths; guard old-pyarrow golden decoding (ITL-898)

- with any option on, Phase 2 metadata traversal looks through dictionary
  types (spec 2.2.1 paths are the 9.6 node paths); format 1 unchanged
- golden vectors that pyarrow < 19 (Decimal32/64) or < 16 (view types)
  cannot decode are strict xfails on those versions only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: hash format 2, written from the spec, behind _FORMAT_VERSION (ITL-898)

src/starfix/_format2.py implements nauticalab/starfix byte-layout-spec
format 2 (041b8f1, provisional): 12-byte header, canonical type JSON with
the name-free Map form, Phase 1/2 keyed by node path, extension types
(storage type; ARROW:extension:* re-added for registered types), the one
rule (entry = validity, lengths, values; row count after the schema
digest), hash_array, per-batch schema check, rejections in every mode.
Data paths work on Arrow buffers (filter, flatten, bitmaps), linear in rows.

tests/test_format2.py checks the 7 worked examples, every final hash the
spec prints (examples 1 and 5, Map, FixedSizeList, non-ASCII hash_array),
primitive forms, JSON text rules, metadata paths, invariants, rejections,
streaming and extension types, byte for byte from the spec text.

ArrowDigester dispatches to format 2 when _FORMAT_VERSION == 2 (still 1:
nothing changes until the design is approved and the golden lands).
Pending spec answers E1-E6 are marked PENDING in the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): hash_array counts nulls on resolved dictionary values; offset tests (ITL-898)

A valid dictionary key that points at a null dictionary value is a null
element (pyarrow's DictionaryArray.null_count counts keys only), as Rust
resolves it. Adds slicing tests at array offsets 7, 8, 9, 16, 63, 64, 65
for Boolean, Int32 and string leaves under a nullable struct and for a
list of Booleans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(format2): deterministic scaling guards for list hashing (ITL-898, ITL-908)

Ports the ITL-908 guards to format 2: hashing a list column never copies
its shared values buffer (tracemalloc growth, isolated from caller
tracing), and the number of node visits does not grow with the row count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(format2): final spec deltas; final golden files; inputs of unmapped literals (ITL-898)

- struct child node path is always p + "/" + name (spec 3.1, R7)
- absent ARROW:extension:metadata hashed as "" (spec 2.3, E1)
- data hashing of unsupported types rejected before hashing, any row count,
  ValueError (spec 7 item 3, E4)
- tests/golden/format2_v0.5.json and format1_to_format2.json copied byte for
  byte from starfix f7aa441 (sha256 1b6102f4..., 9d40973c...)
- tests/golden/python_literals_format1_inputs.json: the inputs of the 69
  hand-coded format-1 literals of this repo that format1_to_format2.json does
  not cover, recovered by recording the calls that produce each literal (all
  69 reproduce their format-1 value from the IPC), for Rust to supply the
  format-2 values

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat!: hash format 2 is the only format; delete format-1 code (ITL-898)

BREAKING CHANGE: every hash changes; the prefix is 00 00 02 (format 2).

- src/starfix/arrow_digester.py is a thin ArrowDigester over _format2
  (keywords include_metadata, ordered_columns, ordered_struct_fields;
  no nested_validity); all format-1 code and the _FORMAT_VERSION / _FIX_*
  switches are deleted
- removed format-1-only tests and the unreleased ordered_v0.5.json;
  test_golden_parity_metadata.py removed (include_metadata_v0.3.json is
  kept as the historical format-1 record, see tests/golden/README.md)
- hand-coded literals moved to their Rust-produced format-2 values from
  format1_to_format2.json (39); the other 69 await Rust values for the
  inputs in python_literals_format1_inputs.json
- tests/test_golden_parity_format2.py asserts every vector and relation of
  format2_v0.5.json (only version-gated skips for old pyarrow)
- tests/test_api.py: keyword-only options, no third option
- CI golden-sync-check diffs format2_v0.5.json and format1_to_format2.json

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: README, CHANGELOG and design note for hash format 2 (ITL-898)

README documents format 2 (prefix, guarantees, options, rejections,
format history); CHANGELOG lists the breaking format change, the fixed
format-1 losses and Python/Rust differences, the ITL-908 performance fix
and the new options; the superseded options design note is replaced by a
format-2 note of the Python-specific decisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: format2_v0.5.json at starfix d2b26c6 (canonical tensor vector) (ITL-898)

Copied byte for byte (sha256 e7e1ccf1...). Golden parity: 254/254
vectors, 89/89 relations, no skips, no xfails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: format 2 module docstring (approved) (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: move the last 69 literals to Rust format-2 values; golden at starfix a4bd86d (ITL-898)

format2_v0.5.json (323 vectors, 89 relations; sha256 afd7280d...) and
format1_to_format2.json (203 entries; sha256 b06a259e...) copied byte for
byte. Every hand-coded literal now carries its Rust-produced format-2 value
(69 more via the recovered inputs). Parity 323/323 + 89/89, no skips, no
xfails; whole suite green on Python 3.10-3.12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: fetch format-2 fixtures with the raw media type (no 1 MB limit) (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: report a missing upstream fixture as missing, not as drift (ITL-898)

On a 404 gh api prints the error body to stdout; keep the fetched file
only when the request succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build!: require pyarrow >= 19; remove old-pyarrow skips; CI job on pyarrow 19.0.0 (ITL-898)

Developer decision 2026-11-11. Older pyarrow cannot read Decimal32/64 from
Arrow IPC; with the floor at 19 every golden vector is decoded and matched
on every supported version, so the version-gated skips (_pyarrow_compat,
list_view, arrow.uuid) are removed. tests/test_golden_parity_r2.py builds
its struct columns with the schema's field types (pyarrow 19 cannot cast
nullable struct children to non-nullable ones); same data, same hashes.
New CI job test-min-pyarrow runs the suite on Python 3.10 with exactly
pyarrow 19.0.0. CHANGELOG and README state the requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): atomic batch updates, dictionary-of-extension metadata, byte-bounded binary hashing (ITL-898)

Copilot review on PR #9:
- a batch rejected while traversed (unmasked null) no longer leaves earlier
  columns' entries updated: entries are snapshotted in O(nodes) (bit
  length, byte length, last partial byte, copies of the SHA-256 states)
  and restored on failure
- a dictionary of a registered extension type keeps its ARROW:extension:*
  keys in Phase 2 (dictionaries are looked through, extensions are not)
- variable-length values are serialized in updates of about 1 MiB; a value
  of at least that size is hashed straight from the data buffer
- regression tests: unmasked nulls rejected (column, struct child,
  dictionary index to a null value), masked child null accepted, rollback
  with and without partial validity bytes, dictionary-of-extension
  metadata, byte budget

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): bounded memory for nullable leaves and finalization (ITL-898)

Copilot review on PR #9:
- nullable leaves are no longer filtered as a whole column: Booleans and
  fixed-width values are filtered in chunks (fixed-width chunks bounded by
  _BYTE_BUDGET bytes); strings and binary skip nulls while reading the
  original offsets/data buffers, so the valid payload is never copied
- entries are fed to the final SHA-256 one by one (validity bits as a
  memoryview) instead of being joined into one byte string
Same bytes: goldens and a chunk-size sweep test (budgets 1, 7, 64, 1 MiB)
over nullable string, Int32, Boolean and FixedSizeBinary leaves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: legacy 0.4 mode, starfix.legacy.v0_4, byte-identical to starfix-python 0.4.0 (ITL-898)

Developer decision 2026-11-11: keep the 0.4 behaviour available under an
explicitly labelled legacy name; the main API stays format 2.

- src/starfix/legacy/v0_4.py: starfix-python v0.4.0 src/starfix/arrow_digester.py
  (tag v0.4.0, d2ed97e) VERBATIM below a frozen-module header (its sha256 equals
  the PyPI starfix==0.4.0 module); stdlib + pyarrow imports only; nothing shared
  with format 2. starfix/legacy/__init__.py re-exports nothing.
- tests/test_legacy_v0_4_*.py: the five v0.4.0 test files restored with their
  original format-1 values (import pointed at the legacy module; the r2 file
  builds struct inputs so pyarrow 19 can construct them), including the
  include_metadata_v0.3.json golden: 193 passed
- tests/test_legacy_v0_4_frozen.py: sha256 pin of the legacy file, and no imports
  between the legacy and format-2 modules
- scripts/legacy_differential.py + CI job legacy-differential: ~2800 results over
  400 seeded random inputs and the Python-0.4 quirks, against the released
  starfix==0.4.0 in an isolated environment: 0 differ (a one-character mutation
  of the legacy file is caught)
- README 'Legacy 0.4 mode' (purpose, never for new data, defects, Python-vs-Rust
  0.4 divergence table), CHANGELOG, tests/golden/README.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf: legacy 0.4 mode hashes list slices without copying the shared buffer (ITL-908)

One performance-only line in src/starfix/legacy/v0_4.py, marked LEGACY PATCH:
raw = memoryview(data_buf) instead of data_buf.to_pybytes() in
_hash_fixed_size_data (the ITL-908 P1 fix). List columns now hash in linear
time when migration reads large stores. No hash changes: the restored 0.4.0
tests pass and the differential test against the released starfix==0.4.0
still compares 2794 results with 0 differences. The header says the file
differs from 0.4.0 by exactly this line; the freeze guard's sha256 is
updated in this commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): zero-width FixedSizeBinary contributes no value bytes (ITL-898)

Copilot review: a nullable FixedSizeBinary(0) leaf divided by zero when
sizing its chunks. Zero-width values have no payload; their validity is
recorded as usual. Regression test at the root and under a nullable struct.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(format2): dictionary of List and hidden child nulls under non-nullable list items (ITL-898)

Inputs from the independent Rust review: a dictionary whose values are a
List hashes as the plain list (column, struct child, hash_array); child
nulls under a null list slot or outside the offsets are not elements and
are accepted; a visible null under a valid slot is rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: legacy separation guard resolves relative and package-level imports (ITL-898)

Copilot review: the guard missed from .legacy.v0_4 import X, from . import
legacy and from starfix import legacy. Relative imports are now resolved
against the file's package and imported names are included; a self-test
covers each form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): list children are zero-copy slices with a selection mask (ITL-898)

Copilot review: a list with a null row (or under a null struct value) was
filtered as a whole, which copies the child buffers (a second copy of a
list<binary> payload). Now a list node writes the lengths of its present
rows (value_lengths filtered as integers) and visits its child as a
zero-copy slice of the original values with a selection mask
take(present_rows, list_parent_indices): elements outside the selection
are not elements (no validity, lengths or values), as the spec requires.
Same bytes (goldens, spec examples, offset and hidden-null tests).

Regression test on a fresh proxy of the Arrow memory pool (tracemalloc
cannot see Arrow allocations): hashing a nullable list<binary> with a
4 MB payload peaks far below the payload (the old code peaked at 4.03 MB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(format2): entry-point kind in the header; hash_array root always nullable (ITL-898)

Developer decision 2026-11-11 (provisional until the Rust spec and golden):
- header = 0x00 || "starfix" || 00 00 02 || kind || flags (13 bytes), kind
  'S' schema digest, 'R' record batch / stream / table, 'A' hash_array
- the record-batch / stream final digest starts with its own 'R' header
- the hash_array root is always nullable: its validity is always written
Spec-level tests migrated to the new layout; the old printed hashes are
replaced by byte-structure assertions until the new spec prints its values.
Pins: nested dictionaries resolve recursively; FixedSizeBinary(0) 1 row != 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): hash leaf dictionaries through their indices (bounded memory) (ITL-898)

Copilot review: dictionary_decode() materialized the whole logical
payload (10,000 indices to one 1 MiB value = 10 GiB). Leaf dictionaries now
stay encoded: validity is index validity AND the validity of the value it
points to (take over booleans), strings/binary are fed from the dictionary
buffers through the indices, fixed-width and Boolean values are gathered in
byte-bounded chunks. Nested dictionaries resolve inside the dictionary only;
dictionaries of struct/list values are decoded (their children are visited).

Hash-neutral: 1602/1602 identical against the previous module (400 seeded
random batches with string and numeric dictionaries, plus dictionaries in
lists, nested, of structs, with null values, sliced). Regression test on a
proxy of the Arrow pool: 80 references to a 1 MiB value peak far below the
decoded size (the old code peaked at 140 MB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: independent review of PR #9, behaviour items 2, 3, 9, 12, 14 (ITL-898)

2  an Arrow-valid empty list array without an offsets buffer segfaulted:
   empty lists no longer touch offsets or lengths (subprocess test, List
   and LargeList, top level and in a struct)
3  with include_metadata, duplicate metadata keys (field, struct child,
   schema) are rejected with ValueError naming the node and key; metadata
   is read through the Arrow C data interface, which keeps duplicates.
   Explicit ARROW:extension:* keys conflicting with a registered extension
   type are rejected; agreeing ones are accepted
9  wrong container types raise TypeError naming the expected type
   (hash_record_batch(Table), hash_table(RecordBatch), update(Table),
   hash_schema / ArrowDigester with a non-schema); hash_array accepts a
   ChunkedArray, hashed chunk by chunk, equal to the combined array
12 legacy: the performance-only line keeps 0.4.0's AttributeError for a
   missing values buffer (differential now 2795 results, 4 raising the same
   exception, 0 differ); r2 test docstring states the input helpers
14 _Bits appends replace the partial last byte in one slice assignment,
   and restore truncates and resets the byte at its saved index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs/test: independent review of PR #9, items 4, 5, 7, 8, 10, 11, 13, 15 (ITL-898)

4  README/CHANGELOG: extension types are hashed as stored; exact only where
   pyarrow can observe the stored form (list of non-canonical cases; a
   Python-defined type's serializer must reproduce the stored bytes);
   the 'byte-identical to Rust' claim names this one exception
5  ArrowDigester docstring: ValueError (not NotImplementedError) for
   unsupported data, plus every rejection; TypeError for wrong containers;
   not thread-safe; hash_array deliberately without include_metadata
7/8/13 README Performance section (linear; strings ~3-5M rows/s; a few us
   per column per update); _visit docstring states what is still allocated
10 golden test asserts minimum vector/relation counts; pytest
   empty_parameter_set_mark = fail_at_collect
11 test_format2.py: expectations no longer taken from the implementation
   (hand-written Phase 1 / type JSON); docstring says what is independent;
   stale notes removed
15 CI: Python 3.13 in the test matrix; golden-sync also diffs
   python_literals_format1_inputs.json (identical to the Rust copy)
CHANGELOG: the version comes from the git tag at release

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: format2_v0.5.json at starfix b5398ce (entry-point kind, hash_array root); literals remapped (ITL-898)

format2_v0.5.json (339 vectors, 96 relations; sha256 e049bc22...) and
format1_to_format2.json (203 entries, unique ids; sha256 bda80b0b...)
copied byte for byte from starfix b5398ce. Every hand-coded literal (108)
moved to its new Rust value (joined on the format-1 hash). Parity 339/339
vectors and 96/96 relations, no skips, no xfails; minimum counts raised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: point the dictionary-of-List, FixedSizeBinary(0) and hidden-null tests at their golden vectors (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: pin the spec's printed hashes of Examples 1 and 5 (starfix d145d89) (ITL-898)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: read exported metadata through __arrow_c_schema__; pin all spec-printed hashes (starfix bcbe74a) (ITL-898)

- metadata pairs (duplicate detection, extension keys of pyarrow-native
  types) are read from the PyCapsule returned by __arrow_c_schema__(), the
  documented Arrow C data interface route in spec 2.3, instead of the
  private _export_to_c; the capsule releases the exported schema
- Map, FixedSizeList and non-ASCII hash_array printed hashes pinned (all
  five printed values of spec bcbe74a now pinned)
- README extension-type wording aligned with spec 2.3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: design note describes the legacy 0.4 module and the no-skip golden contract (ITL-898)

Copilot review: the Python design note still said no format-1 code is kept
and described version-gated golden-test skips that were removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): refuse big-endian hosts; tests for decoded dictionary validity at depth and the FixedSizeList element path (ITL-898)

Spec re-read at starfix 9c9a88c:
- Section 13: hashes are defined over little-endian buffers; values are
  hashed as stored, so every format-2 entry point raises
  NotImplementedError on a big-endian host (README 'Platforms')
- dictionary validity is that of the decoded array at any depth: tested for
  a column, a struct child and a list element
- the FixedSizeList element node f/ claims its path: tested

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(format2): dictionary validity without scanning a null-free dictionary per batch (ITL-898)

Copilot review (needs a closer look): computing dictionary validity took
the dictionary's validity per batch, O(batches x dictionary size) for many
small batches. When the dictionary has no nulls, validity is the indices'
(identical bytes). Regression test: 50 one-row batches over a 10,000-value
dictionary make no take() over the dictionary validity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(format2): sync golden to starfix e079d56 and pin hidden-null cases

Copy tests/golden/format2_v0.5.json byte for byte from starfix e079d56
(sha256 4d5ba202f9fa88e69c94c09947e715c51c5403803b9fa0dfc3ec7dda9582845e;
423 vectors, 114 relations, the previous 339/96 an unchanged prefix) and
raise the pinned minimum counts. format1_to_format2.json is unchanged.

Add golden-independent tests for the two shapes the Rust review found:
a leaf null hidden under a null inner list of a nested list (hash_array,
one batch and single-row batches agree with the clean form), and a
dictionary of nested structs with a null hidden under a null outer value
(dictionary-encoded equals plain), under every option combination.

Refs ITL-898

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(format2): hash decimal32/decimal64 with nulls on pyarrow 19/20 (ITL-898)

pyarrow 19.0.x and 20.0.0 have no `filter` kernel for decimal32/decimal64, so dropping
nulls from those arrays (directly, or nested under list/struct/dictionary of list or
struct) raised ArrowNotImplementedError on the supported minimum pyarrow.

Fixed-width values are now selected on a zero-copy view of their value buffer as an
unsigned integer of the same width (or FixedSizeBinary(width) for wider values), so no
kernel of the logical type is used. The bytes selected are unchanged: no hash changes
(all 423 golden vectors and 114 relations match with no skips).

tests/test_format2_type_matrix.py runs every supported leaf type, with and without nulls,
sliced, chunked, batched, dictionary-encoded and nested, through hash_array, record batch
and stream hashing, against the Rust golden vectors where they exist and across the
equivalent forms otherwise. CI runs the suite on pyarrow 19.0.0, 20.0.0 and the latest,
and the pytest run asserts the pyarrow it imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(format2): field metadata errors get their path from the caller; README memory note (ITL-898)

_field_metadata no longer receives the node path only to build error messages: it raises
_AmbiguousMetadata (a ValueError) and phase2_json re-raises it as ValueError prefixed with
the field path ("Field 's/c': Duplicate metadata key 'a' on the field: ..."). Tests assert
the full messages, including the path.

README: auxiliary memory is not bounded (validity, lengths and list parent indices per
visited element; dictionary-of-struct/list values are materialized with take); only leaf
payload feeding is chunked.

Also the cheap ruff findings in _format2.py (sorted __slots__, unquoted annotation,
itertools.pairwise). No hash changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(format2): sync golden to starfix e3d5780 (73 FixedSizeList vectors, 34 relations) (ITL-927)

Copied byte for byte from nauticalab/starfix tests/golden/format2_v0.5.json
(sha256 0cd11ea872a64b15db6ae18394b7f606aac21eba65267bb316dd0b9758e7a989).
Floors raised to 496 vectors and 148 relations. The FixedSizeList data
vectors fail until FixedSizeList data is hashed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(format2): hash FixedSizeList data (ITL-927)

A fixed-size-list node's entry is its effective validity if nullable and
empty otherwise (no lengths digest: n is in the schema). Its element node
gets the n items of every effectively valid slot; items of null or
struct-masked slots and child positions past len * n are never visited,
and struct masking restarts below the list, as for List.

Slot i covers values[(offset + i) * n, (offset + i + 1) * n): pyarrow's
FixedSizeListArray.values ignores the array offset. Without null slots
the items are one zero-copy slice; otherwise slots are visited in chunks
of about 1M items, with the slot bits repeated n times as the selection
(list_parent_indices skips null fixed-size-list slots, so it runs on a
validity-free FixedSizeList of nulls instead).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(format2): FixedSizeList in the type matrix; docs for FixedSizeList data (ITL-927)

The type matrix wraps every supported leaf type in fixed-size lists of
size 0, 1 and 2 (a null slot, trailing child positions), nested in
list / struct / dictionary and containing struct / list / fixed-size
list, each compared with a reference that holds other content under the
null slot and no trailing positions.

README, CHANGELOG (0.5.0 section) and the arrow_digester docstring no
longer list FixedSizeList as unsupported; design note added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(format2): reflow docstrings touched by FixedSizeList support (ITL-927)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): release section 0.5.0 (2026-11-11); FixedSizeList items in the rejected-data entry (ITL-927)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Edgar Y. Walker <>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants