Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
bac93fe
test: failing tests for ordered and nested-validity hashing options (…
Nov 11, 2026
2297a25
feat: ordered_columns, ordered_struct_fields and nested_validity opti…
Nov 11, 2026
f7c544b
test: golden parity test for ordered_v0.5.json; docs and CI sync chec…
Nov 11, 2026
3576735
feat: Boolean one-byte layout under nested_validity; Rust Map form wi…
Nov 11, 2026
bee6820
test: pin known default-mode divergences from Rust as strict xfails; …
Nov 11, 2026
05cac6b
ci: let the ordered fixture drift step explain a missing upstream fil…
Nov 11, 2026
98b04c4
test: golden file at starfix e5b49c4 (171 vectors, 73 relations); spe…
Nov 11, 2026
ea770d0
docs: list all three default-mode divergences in the design note (ITL…
Nov 11, 2026
27ab18b
docs: show the escaped form in the divergence table (ITL-898)
Nov 11, 2026
4e9f5fd
test: tidy ordered hashing tests (ITL-898)
Nov 11, 2026
2e82a57
perf: hash Booleans under nested_validity in bounded chunks (ITL-898)
Nov 11, 2026
1a1aa4b
feat: spec 2.1 type forms and 2.2.1 map metadata paths with options; …
Nov 11, 2026
42ca159
refactor: one-line switches for pending default-mode decisions; lossl…
Nov 11, 2026
83a41a1
refactor: tie default-mode fixes to one hash format version constant …
Nov 11, 2026
4e64254
fix: look through dictionaries in option-on metadata paths; guard old…
Nov 11, 2026
6dda4bf
feat: hash format 2, written from the spec, behind _FORMAT_VERSION (I…
Nov 11, 2026
c217677
fix(format2): hash_array counts nulls on resolved dictionary values; …
Nov 11, 2026
02fd2bd
test(format2): deterministic scaling guards for list hashing (ITL-898…
Nov 11, 2026
b9e2964
feat(format2): final spec deltas; final golden files; inputs of unmap…
Nov 11, 2026
7fb5828
feat!: hash format 2 is the only format; delete format-1 code (ITL-898)
Nov 11, 2026
217ced9
docs: README, CHANGELOG and design note for hash format 2 (ITL-898)
Nov 11, 2026
e856379
test: format2_v0.5.json at starfix d2b26c6 (canonical tensor vector) …
Nov 11, 2026
6a4d212
docs: format 2 module docstring (approved) (ITL-898)
Nov 11, 2026
64eee34
test: move the last 69 literals to Rust format-2 values; golden at st…
Nov 11, 2026
c0f99f5
ci: fetch format-2 fixtures with the raw media type (no 1 MB limit) (…
Nov 11, 2026
e3831f3
ci: report a missing upstream fixture as missing, not as drift (ITL-898)
Nov 11, 2026
110aa1a
build!: require pyarrow >= 19; remove old-pyarrow skips; CI job on py…
Nov 11, 2026
40da996
fix(format2): atomic batch updates, dictionary-of-extension metadata,…
Nov 11, 2026
f85a1fa
fix(format2): bounded memory for nullable leaves and finalization (IT…
Nov 11, 2026
3085d95
feat: legacy 0.4 mode, starfix.legacy.v0_4, byte-identical to starfix…
Nov 11, 2026
f65253f
perf: legacy 0.4 mode hashes list slices without copying the shared b…
Nov 11, 2026
277d691
fix(format2): zero-width FixedSizeBinary contributes no value bytes (…
Nov 11, 2026
1d9ae9b
test(format2): dictionary of List and hidden child nulls under non-nu…
Nov 11, 2026
8f145d5
test: legacy separation guard resolves relative and package-level imp…
Nov 11, 2026
da8e945
fix(format2): list children are zero-copy slices with a selection mas…
Nov 11, 2026
64179f9
feat(format2): entry-point kind in the header; hash_array root always…
Nov 11, 2026
7745d2c
fix(format2): hash leaf dictionaries through their indices (bounded m…
Nov 11, 2026
8488dae
fix: independent review of PR #9, behaviour items 2, 3, 9, 12, 14 (IT…
Nov 11, 2026
8082bbe
docs/test: independent review of PR #9, items 4, 5, 7, 8, 10, 11, 13,…
Nov 11, 2026
786b9fe
test: format2_v0.5.json at starfix b5398ce (entry-point kind, hash_ar…
Nov 11, 2026
ca5dbb1
test: point the dictionary-of-List, FixedSizeBinary(0) and hidden-nul…
Nov 11, 2026
376ef56
test: pin the spec's printed hashes of Examples 1 and 5 (starfix d145…
Nov 11, 2026
50aff53
fix: read exported metadata through __arrow_c_schema__; pin all spec-…
Nov 11, 2026
c70c449
docs: design note describes the legacy 0.4 module and the no-skip gol…
Nov 11, 2026
758a26b
fix(format2): refuse big-endian hosts; tests for decoded dictionary v…
Nov 11, 2026
98b38b1
perf(format2): dictionary validity without scanning a null-free dicti…
Nov 11, 2026
b90e196
test(format2): sync golden to starfix e079d56 and pin hidden-null cases
Nov 11, 2026
c440e82
fix(format2): hash decimal32/decimal64 with nulls on pyarrow 19/20 (I…
Nov 11, 2026
940887c
refactor(format2): field metadata errors get their path from the call…
Nov 11, 2026
80c50be
test(format2): sync golden to starfix e3d5780 (73 FixedSizeList vecto…
Nov 11, 2026
fd3d3c5
feat(format2): hash FixedSizeList data (ITL-927)
Nov 11, 2026
895adb0
test(format2): FixedSizeList in the type matrix; docs for FixedSizeLi…
Nov 11, 2026
85b15b1
docs(format2): reflow docstrings touched by FixedSizeList support (IT…
Nov 11, 2026
f4e10a7
merge: origin/main (squash of #9) into FixedSizeList branch (ITL-927)
Nov 11, 2026
cf80096
docs(changelog): release section 0.5.0 (2026-11-11); FixedSizeList it…
Nov 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
# Changelog

## Unreleased (proposed 0.5.0)

The package version comes from the git tag at release (hatch-vcs); nothing in the source
pins it to 0.5.0.
## 0.5.0 (2026-11-11)

### Changed — hash format 2 (every hash changes)

Expand Down Expand Up @@ -56,6 +53,15 @@ buffers: at 1M rows, `list<int32>` takes about 20 ms instead of an estimated 90
(canonical forms; a Python-defined type's serializer must reproduce the stored bytes); see
the README section "Extension types".
- Golden vectors `tests/golden/format2_v0.5.json`, generated by Rust and asserted in full.
- **`FixedSizeList` data** (ITL-927), byte-identical to the Rust crate: record batches,
streams and `hash_array` accept `FixedSizeList<T, n>` data (any size, also 0; at any depth;
dictionary-encoded; as extension storage such as `arrow.fixed_shape_tensor`).
`FixedSizeList<T, n>` is a different type from `List<T>` / `LargeList<T>`, from
`FixedSizeList<T, m>` (`m != n`) and from `FixedSizeBinary(n)`: equal values hash
differently. A fixed-size list node's entry is its validity if nullable and nothing else (no
lengths, as `n` is in the schema); its element node's elements are the `n` items of every
effectively valid slot. Items under null slots and child positions past the last slot never
affect the hash. Pinned by the 73 `fsl_*` golden vectors and 34 relations.

### Changed (breaking)

Expand All @@ -64,6 +70,7 @@ buffers: at 1M rows, `list<int32>` takes about 20 ms instead of an estimated 90
pyarrow version only from 19 on.
- In every mode, `ValueError` is raised for schemas with duplicate or `/`-aliased field paths;
for `ListView`, `LargeListView`, `Union` and `RunEndEncoded` types; for hashing data of
`Null`, `FixedSizeList`, `Map`, `Interval` and view types (previously `NotImplementedError`
or accepted); for a non-nullable field holding a null that no enclosing struct null explains;
and for a streamed batch whose schema differs from the digester's.
`Null`, `Map`, `Interval` and view types, also as the items of a `FixedSizeList`
(previously `NotImplementedError` or accepted); for a non-nullable field holding a null that
no enclosing struct null explains; and for a streamed batch whose schema differs from the
digester's.
25 changes: 18 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Keyword-only, all `False` by default, the same names as the Rust `HasherConfig`
| Option | Effect |
|---|---|
| `ordered_columns` | The order of the top-level columns is part of the hash. |
| `ordered_struct_fields` | The order of struct fields, at every depth (struct children, list elements, map keys and values, dictionary values), is part of the hash. |
| `ordered_struct_fields` | The order of struct fields, at every depth (struct children, list and fixed-size list elements, map keys and values, dictionary values), is part of the hash. |
| `include_metadata` | Schema- and field-level Arrow metadata is part of the hash. A schema without metadata hashes the same either way. |

```python
Expand All @@ -77,13 +77,22 @@ digest = ArrowDigester.hash_table(table, ordered_columns=True, ordered_struct_fi
DataFrame, or a `pa.StructArray` whose type `struct<a, b>` differs from `struct<b, a>`.
- Hashes taken with different order options never coincide, even when order cannot matter.
- With `include_metadata`, metadata is keyed by each field's node path (column `c`, struct
child `c/x`, list element `c/`, map key and value `m//key` and `m//value`). Extension types
child `c/x`, list or fixed-size list element `c/`, map key and value `m//key` and `m//value`). Extension types
are hashed as their storage type; with `include_metadata` their `ARROW:extension:name` and
`ARROW:extension:metadata` are hashed too (see [Extension types](#extension-types)).
- `hash_array` has no `include_metadata` on purpose: an array carries no schema or field
metadata. It accepts a `ChunkedArray` (hashed like the combined array).
- A digester is not thread-safe; a batch rejected by `update` leaves it unchanged.

### Fixed-size lists

`FixedSizeList<T, n>` data is hashed (any size `n`, also 0; at any depth; dictionary-encoded;
as the storage of extension types such as `arrow.fixed_shape_tensor`). It is its own type: it
never hashes like `List<T>` / `LargeList<T>` with the same values, like a fixed-size list of
another size, or like `FixedSizeBinary(n)`. The items under a null slot (or under a slot
masked by a null struct) never affect the hash, and neither do child positions past the last
slot; a null of a non-nullable item there is accepted, a visible one is rejected.

### Extension types

With `include_metadata`, an extension type's field metadata is hashed **as stored**, including
Expand Down Expand Up @@ -117,9 +126,9 @@ rebuilds the type from them, so starfix-python hashes what pyarrow can observe:
within a struct, or a name containing `/` that aliases another node (a column `"a/b"` next to
a struct column `a` with a child `b`). A `/` that aliases nothing (`"km/h"`) is fine;
- `ListView`, `LargeListView`, `Union` and `RunEndEncoded` types;
- hashing data of a schema that contains `Null`, `FixedSizeList`, `Map`, `Interval`,
`Utf8View` or `BinaryView` (their schemas hash; their data does not), whatever the number of
rows;
- hashing data of a schema that contains `Null`, `Map`, `Interval`, `Utf8View` or `BinaryView`
(their schemas hash; their data does not), also inside a fixed-size list, whatever the number
of rows;
- a non-nullable field holding a null that no enclosing struct null explains;
- with `include_metadata`: a duplicate metadata key on a field or the schema, or explicit
`ARROW:extension:*` keys that conflict with the field's registered extension type;
Expand All @@ -144,8 +153,10 @@ crate has the same restriction.
Hashing takes time linear in the data size and works on Arrow buffers. Leaf values are fed to
the digest in bounded chunks without copying the payload, but auxiliary memory is not bounded:
it grows with the number of visited elements and list items (validity bitmaps, list lengths,
list parent indices), and the values of a dictionary of struct or list values are materialized
(decoded with `take`) before they are traversed. Fixed-width, Boolean and list columns hash at tens of millions of rows per second.
list parent indices), and the values of a dictionary of struct, list or fixed-size list values
are materialized (decoded with `take`) before they are traversed. A fixed-size list column
without null slots is hashed from one zero-copy slice of its child; with null slots, its items
are selected in chunks of about one million items. Fixed-width, Boolean and list columns hash at tens of millions of rows per second.
String and binary columns hash at about 3 to 5 million rows per second, because each value is
visited in Python. Each `update` call also costs a few microseconds per column, to check the
batch's schema. starfix-python is a pure-Python implementation; the Rust crate is faster on
Expand Down
65 changes: 65 additions & 0 deletions docs/metamorphic/specs/2026-11-11-fixed-size-list-python.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# FixedSizeList data in starfix-python (ITL-927)

**Date:** 2026-11-11
**Issue:** ITL-927
**Repos:** nauticalab/starfix (Rust, authoritative), nauticalab/starfix-python (this repo)
**Status:** Layout confirmed by the maintainer on 2026-11-11; ships in 0.5.0.

## Source of truth

The layout is starfix's `docs/byte-layout-spec.md` (Section 3.2, Example 3b) and its design
`docs/metamorphic/specs/2026-11-11-fixed-size-list-design.md`. The 73 `fsl_*` vectors and 34
relations of `tests/golden/format2_v0.5.json` are generated by Rust and copied here byte for
byte. This note records only the Python-specific decisions.

## Layout (summary)

- A fixed-size-list node's entry is its effective validity if the field is nullable and empty
otherwise: no lengths digest (`n` is in the schema digest) and no values.
- Its element node `p/` gets the `n` items of every effectively valid slot, in order. Items of
slots that are null (own validity, or masked by a null enclosing struct) and child positions
past `len * n` are never visited. Struct masking restarts below the list, as for List.
- A null of a non-nullable element under a null slot is accepted; a visible one is rejected.

## Implementation (`src/starfix/_format2.py`)

- `_Entry` kind `fixed_size_list`: validity only.
- `_visit_fixed_size_list_items`: slot `i` covers
`arr.values[(arr.offset + i) * n, (arr.offset + i + 1) * n)`, because pyarrow's
`FixedSizeListArray.values` **ignores the array offset**. The items of all slots are one
zero-copy slice of the child.
- No null slot (`present is None`) or `n = 0`: the slice is visited once, without any
selection. No per-item index, mask or copy is built (pinned by
`test_fixed_size_list_without_nulls_builds_no_item_index`).
- Otherwise slots are visited in chunks of about `_ITEM_BUDGET` (1M) items. A chunk without
elements is skipped; a chunk of elements only is visited without a selection; any other
chunk is visited with the slot bits repeated `n` times as the item selection, which the
existing select-aware code applies to every node below (validity, non-nullable check,
lengths, values).
- The repeated bits come from `take(bits, list_parent_indices(FixedSizeList<null>[n]))` over
a validity-free list of nulls: `list_parent_indices` on the real array **skips null slots**
(pyarrow 19 to 26), and `flatten()` skips own-null slots but not struct-masked ones, so
neither is used on the hashed array.
- Leaf values below are hashed by the existing code: fixed-width values through the zero-copy
unsigned-integer / `FixedSizeBinary` view, so no kernel of the logical type is needed (pyarrow
19 and 20 have no `filter` kernel for decimal32 / decimal64).
- A dictionary of fixed-size lists is decoded per row with `take` before it is traversed (as
for lists and structs); hidden child nulls are not validated.

## Tests

- `tests/test_golden_parity_format2.py`: all 496 vectors and 148 relations, no skips. The IPC
payloads are read without full validation (the hidden-null vectors hold nulls of
non-nullable fields under null slots).
- `tests/test_format2.py`: spec Example 3b byte for byte (record batch and `hash_array`), the
non-nullable entry, `n = 0`, slices and trailing child positions on locally built arrays
(IPC normalizes both, so golden vectors cannot pin them), struct masking at two depths with
a different offset at every level, non-nullable slots and items, dictionary of fixed-size
lists, chunked item selection at any chunk size.
- `tests/test_format2_type_matrix.py`: fixed-size lists (`n = 0, 1, 2`, a null slot, trailing
child positions) of every supported leaf type, nested in list / struct / dictionary and
containing struct / list / fixed-size list, each compared with a reference holding other
content under the null slot and no trailing positions, through `hash_array`, record batch
and stream, sliced and chunked. CI runs it on pyarrow 19.0.0, 20.0.0 and the latest.
- A differential run against the Rust branch (random fixed-size-list tables via IPC) is
reported on the PR; it is not part of the repository.
62 changes: 57 additions & 5 deletions src/starfix/_format2.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ def _require_little_endian() -> None:
_U64 = struct.Struct("<Q")
_CHUNK = 1 << 16 # Booleans per chunk when expanding them to one byte each
_BYTE_BUDGET = 1 << 20 # bytes per digest update when serializing variable-length values
_ITEM_BUDGET = 1 << 20 # fixed-size-list items per chunk when some slots are not visited


def flags_of(ordered_columns: bool, ordered_struct_fields: bool) -> int:
Expand Down Expand Up @@ -422,7 +423,14 @@ class _Entry:

def __init__(self, field: pa.Field) -> None:
t = _logical(field.type)
self.kind = "struct" if pa.types.is_struct(t) else "list" if _is_list(t) else "leaf"
if pa.types.is_struct(t):
self.kind = "struct"
elif _is_list(t):
self.kind = "list"
elif pa.types.is_fixed_size_list(t):
self.kind = "fixed_size_list" # Section 3.2: validity only (n is in the schema)
else:
self.kind = "leaf"
self.validity = _Bits() if field.nullable else None
self.lengths = hashlib.sha256() if self.kind == "list" else None
self.values = hashlib.sha256() if self.kind == "leaf" else None
Expand Down Expand Up @@ -663,9 +671,9 @@ def _visit(
"""Feed the elements of the node `field` at `path` into its entry (Section 3.2).

`arr` holds the node's candidate elements; `select` (None: all) marks which of them are
elements at all: below a list, the items of valid list values (items of null or masked
list values are not visited). `enclosing` is the AND of the enclosing struct validities
at this list level (None: all valid). Value payloads (string/binary bytes, list
elements at all: below a list or fixed-size list, the items of valid list values (items
of null or masked list values are not visited). `enclosing` is the AND of the enclosing
struct validities at this list level (None: all valid). Value payloads (string/binary bytes, list
children) are not copied: list children are zero-copy slices of the original values with
a selection mask. Per-element metadata is still allocated: validity bitmaps, list lengths
cast to u64, and list_parent_indices (8 bytes per list item).
Expand Down Expand Up @@ -721,13 +729,57 @@ def _visit(
_visit(arr.type.value_field, items, path + "/", None, entries, item_select)
return

if entry.kind == "fixed_size_list":
_visit_fixed_size_list_items(arr, present, path + "/", entries)
return

_hash_values(arr, present, entry.values)


def _visit_fixed_size_list_items(arr: pa.FixedSizeListArray, present: pa.Array | None, path: str,
entries: dict) -> None:
"""The element node of a fixed-size list (Section 3.2): the n items of every slot marked
in `present` (None: all), in order. Items of other slots and child positions past
len * n are not elements and are never visited.

pyarrow's ``FixedSizeListArray.values`` ignores the array offset, so slot i covers
``values[(offset + i) * n, (offset + i + 1) * n)``. The items are visited zero-copy as
one slice of the child when all slots are elements (no per-item index or mask is built);
otherwise the slots are visited in chunks of about _ITEM_BUDGET items (entries are
streams, so chunks hash like one pass), each chunk with the slot bits repeated n times as
the item selection, skipping chunks without any element and selecting nothing in chunks
of elements only. As for List, a fixed-size-list boundary resets struct masking.
"""
n = arr.type.list_size
element = arr.type.value_field
items = arr.values.slice(arr.offset * n, len(arr) * n)
if present is None or n == 0:
_visit(element, items, path, None, entries)
return
slots = max(1, _ITEM_BUDGET // n)
for start in range(0, len(arr), slots):
chosen = present.slice(start, slots)
count = len(chosen)
valid = chosen.true_count
if valid == 0:
continue
item_select = None
if valid < count:
item_select = _repeat_each(chosen, n)
_visit(element, items.slice(start * n, count * n), path, None, entries, item_select)


def _repeat_each(bits: pa.Array, n: int) -> pa.Array:
"""Every bit of `bits` n times in a row. The parent indices come from a FixedSizeList of
nulls without validity (list_parent_indices skips null fixed-size-list slots)."""
slots = pa.FixedSizeListArray.from_arrays(pa.nulls(len(bits) * n), n)
return pc.take(bits, pc.list_parent_indices(slots))


def _is_unsupported_data(t: pa.DataType) -> bool:
"""Section 8: types whose schema hashes but whose data does not."""
return (
pa.types.is_null(t) or pa.types.is_fixed_size_list(t) or pa.types.is_map(t)
pa.types.is_null(t) or pa.types.is_map(t)
or str(t) in _INTERVALS or str(t) in ("string_view", "binary_view")
)

Expand Down
6 changes: 3 additions & 3 deletions src/starfix/arrow_digester.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ class ArrowDigester:
Raises ``ValueError`` (in every mode) for: schemas in which two nodes share a path
(duplicate names, or a name containing ``/`` that aliases another node); ListView,
LargeListView, Union and RunEndEncoded types; hashing data of a schema containing
Null, FixedSizeList, Map, Interval or view types (their schemas hash), whatever the
number of rows; a non-nullable field holding a null that no enclosing struct null
explains; duplicate metadata keys when ``include_metadata`` is on; and a streamed
Null, Map, Interval or view types (their schemas hash; also inside a FixedSizeList),
whatever the number of rows; a non-nullable field holding a null that no enclosing
struct null explains; duplicate metadata keys when ``include_metadata`` is on; and a streamed
batch whose schema differs from the digester's. Raises ``TypeError`` for a wrong
container type (for example a Table passed to ``hash_record_batch``).

Expand Down
Loading
Loading