Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
bac93fe
test: failing tests for ordered and nested-validity hashing options (…
Nov 11, 2026
2297a25
feat: ordered_columns, ordered_struct_fields and nested_validity opti…
Nov 11, 2026
f7c544b
test: golden parity test for ordered_v0.5.json; docs and CI sync chec…
Nov 11, 2026
3576735
feat: Boolean one-byte layout under nested_validity; Rust Map form wi…
Nov 11, 2026
bee6820
test: pin known default-mode divergences from Rust as strict xfails; …
Nov 11, 2026
05cac6b
ci: let the ordered fixture drift step explain a missing upstream fil…
Nov 11, 2026
98b04c4
test: golden file at starfix e5b49c4 (171 vectors, 73 relations); spe…
Nov 11, 2026
ea770d0
docs: list all three default-mode divergences in the design note (ITL…
Nov 11, 2026
27ab18b
docs: show the escaped form in the divergence table (ITL-898)
Nov 11, 2026
4e9f5fd
test: tidy ordered hashing tests (ITL-898)
Nov 11, 2026
2e82a57
perf: hash Booleans under nested_validity in bounded chunks (ITL-898)
Nov 11, 2026
1a1aa4b
feat: spec 2.1 type forms and 2.2.1 map metadata paths with options; …
Nov 11, 2026
42ca159
refactor: one-line switches for pending default-mode decisions; lossl…
Nov 11, 2026
83a41a1
refactor: tie default-mode fixes to one hash format version constant …
Nov 11, 2026
4e64254
fix: look through dictionaries in option-on metadata paths; guard old…
Nov 11, 2026
6dda4bf
feat: hash format 2, written from the spec, behind _FORMAT_VERSION (I…
Nov 11, 2026
c217677
fix(format2): hash_array counts nulls on resolved dictionary values; …
Nov 11, 2026
02fd2bd
test(format2): deterministic scaling guards for list hashing (ITL-898…
Nov 11, 2026
b9e2964
feat(format2): final spec deltas; final golden files; inputs of unmap…
Nov 11, 2026
7fb5828
feat!: hash format 2 is the only format; delete format-1 code (ITL-898)
Nov 11, 2026
217ced9
docs: README, CHANGELOG and design note for hash format 2 (ITL-898)
Nov 11, 2026
e856379
test: format2_v0.5.json at starfix d2b26c6 (canonical tensor vector) …
Nov 11, 2026
6a4d212
docs: format 2 module docstring (approved) (ITL-898)
Nov 11, 2026
64eee34
test: move the last 69 literals to Rust format-2 values; golden at st…
Nov 11, 2026
c0f99f5
ci: fetch format-2 fixtures with the raw media type (no 1 MB limit) (…
Nov 11, 2026
e3831f3
ci: report a missing upstream fixture as missing, not as drift (ITL-898)
Nov 11, 2026
110aa1a
build!: require pyarrow >= 19; remove old-pyarrow skips; CI job on py…
Nov 11, 2026
40da996
fix(format2): atomic batch updates, dictionary-of-extension metadata,…
Nov 11, 2026
f85a1fa
fix(format2): bounded memory for nullable leaves and finalization (IT…
Nov 11, 2026
3085d95
feat: legacy 0.4 mode, starfix.legacy.v0_4, byte-identical to starfix…
Nov 11, 2026
f65253f
perf: legacy 0.4 mode hashes list slices without copying the shared b…
Nov 11, 2026
277d691
fix(format2): zero-width FixedSizeBinary contributes no value bytes (…
Nov 11, 2026
1d9ae9b
test(format2): dictionary of List and hidden child nulls under non-nu…
Nov 11, 2026
8f145d5
test: legacy separation guard resolves relative and package-level imp…
Nov 11, 2026
da8e945
fix(format2): list children are zero-copy slices with a selection mas…
Nov 11, 2026
64179f9
feat(format2): entry-point kind in the header; hash_array root always…
Nov 11, 2026
7745d2c
fix(format2): hash leaf dictionaries through their indices (bounded m…
Nov 11, 2026
8488dae
fix: independent review of PR #9, behaviour items 2, 3, 9, 12, 14 (IT…
Nov 11, 2026
8082bbe
docs/test: independent review of PR #9, items 4, 5, 7, 8, 10, 11, 13,…
Nov 11, 2026
786b9fe
test: format2_v0.5.json at starfix b5398ce (entry-point kind, hash_ar…
Nov 11, 2026
ca5dbb1
test: point the dictionary-of-List, FixedSizeBinary(0) and hidden-nul…
Nov 11, 2026
376ef56
test: pin the spec's printed hashes of Examples 1 and 5 (starfix d145…
Nov 11, 2026
50aff53
fix: read exported metadata through __arrow_c_schema__; pin all spec-…
Nov 11, 2026
c70c449
docs: design note describes the legacy 0.4 module and the no-skip gol…
Nov 11, 2026
758a26b
fix(format2): refuse big-endian hosts; tests for decoded dictionary v…
Nov 11, 2026
98b38b1
perf(format2): dictionary validity without scanning a null-free dicti…
Nov 11, 2026
b90e196
test(format2): sync golden to starfix e079d56 and pin hidden-null cases
Nov 11, 2026
c440e82
fix(format2): hash decimal32/decimal64 with nulls on pyarrow 19/20 (I…
Nov 11, 2026
940887c
refactor(format2): field metadata errors get their path from the call…
Nov 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 94 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

Expand All @@ -30,6 +30,70 @@ jobs:
- name: Run tests
run: uv run --python ${{ matrix.python-version }} pytest tests/ -v

test-pyarrow:
# The oldest supported pyarrow (pyproject: pyarrow>=19.0.0) on the oldest Python, 20.0.0
# (still lacking some kernels of later versions, e.g. decimal32/64 `filter`), and the
# latest pyarrow on PyPI. The pytest run itself asserts the pyarrow it imports
# (STARFIX_EXPECT_PYARROW) and prints it in the session header.
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- pyarrow: "19.0.0"
python-version: "3.10"
- pyarrow: "20.0.0"
python-version: "3.10"
- pyarrow: "latest"
python-version: "3.13"
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

- name: Install uv
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5

- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}

- name: Install dependencies
run: uv sync --dev --python ${{ matrix.python-version }}

- name: Resolve the pyarrow version
id: pyarrow
run: |
if [ "${{ matrix.pyarrow }}" = latest ]; then
version=$(curl -fsSL https://pypi.org/pypi/pyarrow/json | jq -r .info.version)
else
version="${{ matrix.pyarrow }}"
fi
test -n "$version"
echo "version=$version" >> "$GITHUB_OUTPUT"

- name: Run tests on pyarrow ${{ steps.pyarrow.outputs.version }}
env:
STARFIX_EXPECT_PYARROW: ${{ steps.pyarrow.outputs.version }}
run: |
uv run --python ${{ matrix.python-version }} --with "pyarrow==$STARFIX_EXPECT_PYARROW" python -c "import pyarrow; assert pyarrow.__version__ == '$STARFIX_EXPECT_PYARROW', pyarrow.__version__; print('pyarrow', pyarrow.__version__)"
uv run --python ${{ matrix.python-version }} --with "pyarrow==$STARFIX_EXPECT_PYARROW" pytest tests/ -v

legacy-differential:
# starfix.legacy.v0_4 must stay byte-identical to the released starfix-python 0.4.0 (PyPI).
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4

- name: Install uv
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5

- name: Set up Python 3.12
run: uv python install 3.12

- name: Install dependencies
run: uv sync --dev --python 3.12

- name: Compare starfix.legacy.v0_4 with starfix==0.4.0 from PyPI
run: uv run --python 3.12 python scripts/legacy_differential.py

license-check:
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -107,3 +171,32 @@ jobs:
echo "Copy the updated fixture from the starfix repo and commit it."
exit 1
fi

- name: Fetch authoritative format-2 fixtures from starfix
run: |
# Raw media type: no 1 MB limit (format2_v0.5.json is append-only and grows).
# Tolerate missing files here so the next step prints the explanation.
# On failure gh prints the error body to stdout, so keep the file only on success.
for f in format2_v0.5.json format1_to_format2.json python_literals_format1_inputs.json; do
if ! gh api -H "Accept: application/vnd.github.raw" \
"repos/nauticalab/starfix/contents/tests/golden/$f" > "/tmp/upstream_$f"; then
rm -f "/tmp/upstream_$f"
fi
done
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}

- name: Fail on format-2 fixture drift
run: |
for f in format2_v0.5.json format1_to_format2.json python_literals_format1_inputs.json; do
if [ ! -s "/tmp/upstream_$f" ]; then
echo "ERROR: Could not fetch tests/golden/$f from nauticalab/starfix main."
echo "(the starfix ITL-898 PR must be merged before this check can pass)"
exit 1
fi
if ! diff "tests/golden/$f" "/tmp/upstream_$f" > /dev/null; then
echo "ERROR: tests/golden/$f has drifted from nauticalab/starfix main."
echo "Copy the updated fixture from the starfix repo and commit it."
exit 1
fi
done
69 changes: 69 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Changelog

## Unreleased (proposed 0.5.0)

The package version comes from the git tag at release (hatch-vcs); nothing in the source
pins it to 0.5.0.

### Changed — hash format 2 (every hash changes)

starfix-python now computes hash format 2, byte-identical to the Rust starfix crate 0.5. The
3-byte prefix of every hash is `00 00 02` (format 2); starfix-python 0.4 and earlier produced
format 1 (`00 00 01`). **Every hash changes**, and a format-1 hash can always be told from a
format-2 hash by its first three bytes. Format-1 hashes can be computed with starfix 0.4 in a
separate environment. Specification: starfix `docs/byte-layout-spec.md`.

### Fixed — information losses of format 1 (ITL-898)

These inputs collided, or hashed differently depending on batch boundaries, in format 1:

- Booleans were bit-packed per call without a count (`[true]` == `[true, false, false]` in a
one-column non-nullable table; batch splits changed the hash). Now one byte per valid value.
- A null struct equalled a struct whose fields are all null, and null struct rows vanished
under non-nullable fields. Now every nullable node records its validity.
- A list whose element is a nullable list lost its own lengths, and a list of nullable structs
did not distinguish a null element. Now every list records its lengths and every nullable
element its validity.
- The row count of zero-column tables and empty structs was not hashed. Now it is.
- Python only: a list of dictionary-encoded fixed-width values (for example
`list<dictionary<int32, int64>>`) hashed the dictionary indices instead of the values.

### Fixed — Python/Rust differences of format 1

All are gone in format 2: map schemas (the value type and `keys_sorted` were not hashed),
non-ASCII struct field names in `hash_array`, metadata paths of map fields, schemas of `Null`,
`Interval`, `Utf8View` and `BinaryView` types (they raised), and the streaming schema check.

### Fixed — performance (ITL-908)

Hashing a list column took time quadratic in the number of rows (every row copied the whole
child values buffer). Format 2 hashes every column type in linear time, working on Arrow
buffers: at 1M rows, `list<int32>` takes about 20 ms instead of an estimated 90 s.

### Added

- **Legacy 0.4 mode**, `starfix.legacy.v0_4`: the starfix-python 0.4.0 hasher (format 1),
byte for byte, frozen, for reading or matching data hashed by 0.4.x only. Never use it for
new data. Pinned by the restored 0.4.0 tests and golden file and by a differential test
against the released `starfix==0.4.0`.

- Keyword-only options `ordered_columns` and `ordered_struct_fields` on `ArrowDigester(...)`,
`hash_schema`, `hash_record_batch`, `hash_table` and `hash_array`: make column order and
struct field order (at every depth) part of the hash. Default off.
- Extension types are hashed as their storage type; with `include_metadata`, their
`ARROW:extension:*` keys are hashed as stored. For extension types that pyarrow registers,
this is exact only when the stored metadata and storage are what pyarrow reconstructs
(canonical forms; a Python-defined type's serializer must reproduce the stored bytes); see
the README section "Extension types".
- Golden vectors `tests/golden/format2_v0.5.json`, generated by Rust and asserted in full.

### Changed (breaking)

- **Requires pyarrow >= 19** (was >= 14): older pyarrow cannot read Decimal32/Decimal64 data
from Arrow IPC, so the golden parity (every vector matched) holds for every supported
pyarrow version only from 19 on.
- In every mode, `ValueError` is raised for schemas with duplicate or `/`-aliased field paths;
for `ListView`, `LargeListView`, `Union` and `RunEndEncoded` types; for hashing data of
`Null`, `FixedSizeList`, `Map`, `Interval` and view types (previously `NotImplementedError`
or accepted); for a non-nullable field holding a null that no enclosing struct null explains;
and for a streamed batch whose schema differs from the digester's.
Loading
Loading