Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
11b567f
docs(itl-897): investigation scripts for pandas hash identity and sto…
Nov 11, 2026
ac99a09
docs(itl-897): migration, golden-diff and storage scripts with the pr…
Nov 11, 2026
f6ebaec
docs(itl-897): draft investigation spec and progress notes (WIP)
Nov 11, 2026
5c9259f
docs(itl-897): investigation spec for pandas and Arrow table value ha…
Nov 11, 2026
59e088c
test(hashing): failing tests for order as value identity (ITL-897)
Nov 11, 2026
a8f9f98
docs(itl-897): one full v1 descriptor per property, measured against …
Nov 11, 2026
f31ab59
docs(itl-897): v1 grammar covers every label and dtype that hashes an…
Nov 11, 2026
5a89fda
docs(itl-897): split identity between starfix ordered mode and the pa…
Nov 11, 2026
6c29fad
Merge remote-tracking branch 'origin/main' into eywalker/itl-897-pand…
Nov 11, 2026
e8dd026
docs(itl-897): implementation plan and starfix 0.5 stand-in for local…
Nov 11, 2026
530b424
feat(hashing): shared labelled-hash helper and ordered Arrow digest (…
Nov 11, 2026
925786d
feat(hashing): pandas_dataframe_v1 / pandas_series_v1 value hashes (I…
Nov 11, 2026
03a6bb9
docs(itl-897): changelog, design issue PV1, plan; scripts typed for b…
Nov 11, 2026
4226b51
test(hashing): move the pa.Table value tests to the second ITL-897 PR
Nov 11, 2026
8fda1a6
refactor(hashing): one starfix table call shared by hash_table and ha…
Nov 11, 2026
6bbf3f2
docs(contexts): describe the pandas v1 digest accurately (ITL-897)
Nov 11, 2026
13294b7
docs(itl-897): plan progress and the option A implementation in the spec
Nov 11, 2026
35cfb88
Merge remote-tracking branch 'origin/main' into eywalker/itl-897-pand…
Nov 11, 2026
ccb0f9c
test(hashing): pin pandas labels to the Arrow hasher id; stand-in can…
Nov 11, 2026
b6b5cfc
docs(plans): ITL-897 refresh onto the v0.3 hash step
Nov 12, 2026
18303b4
merge: origin/main (c8e9eb21, v0.3 hash step) into ITL-897
Nov 12, 2026
d24bdea
refactor(hashing): pandas labels follow the ITL-938 label rule; drop …
Nov 12, 2026
277982d
test(golden): pandas value entries move to pandas_dataframe_v1; add s…
Nov 12, 2026
ce78eb4
docs(itl-897): hashing reference section 10, changelog fragment, desi…
Nov 12, 2026
31bb5e1
docs(plans): ITL-897 progress and coordinator hold
Nov 12, 2026
e58c0c9
refactor(hashing): one list of the pandas identity properties, checke…
Nov 12, 2026
431d179
test(arrays): failing tests, design note and plan for the strict arra…
Nov 12, 2026
4bbe5ef
merge: origin/main (25650c42) into ITL-897
Nov 12, 2026
3900ca6
docs(specs): extension name numpy.ndarray and the v0.2 name collision…
Nov 12, 2026
962baa6
docs(specs): scope the Arrow schema hash question (ITL-981)
Nov 12, 2026
ab2684f
docs(plans): record coordinator decisions while waiting for sign-off …
Nov 12, 2026
95a9782
docs(plans): ITL-897 progress after the second merge; push held
Nov 12, 2026
592274a
docs(specs): final extension name orcapod.numpy.ndarray; drop the for…
Nov 12, 2026
1e5e860
Merge remote-tracking branch 'origin/main' into eywalker/itl-981-libr…
Nov 12, 2026
ee8c386
docs(plans): ITL-897 phase 2 decisions (rename, attrs, round-trip fixes)
Nov 12, 2026
16d9630
refactor(logical_types)!: pandas extension names are orcapod.pandas.d…
Nov 12, 2026
64c63bf
feat(logical_types): ignore-list mechanism for library value types (I…
Nov 12, 2026
2756ada
fix(hashing): a pandas value read back from storage keeps its hash; a…
Nov 12, 2026
381a393
test(pandas): a v0.2 table with a pandas.dataframe column is reported…
Nov 12, 2026
99e4d52
docs(itl-897): strict pandas hash, rename and storage/hash split in t…
Nov 12, 2026
1ea760d
docs(handlers): pandas handler docstring states the strict hash (ITL-…
Nov 12, 2026
acb1db3
chore(types): basedpyright-clean pandas hash and tests (ITL-897)
Nov 12, 2026
fdd05a8
merge: ITL-981 ignore-list mechanism (64c63bfe) into ITL-897, local only
Nov 12, 2026
5a0277c
feat(pandas): pandas value types take an ignore list (ITL-981 mechani…
Nov 12, 2026
5684014
chore(types): basedpyright-clean pandas ignore-list tests (ITL-897)
Nov 12, 2026
551d4a3
docs(plans): ITL-897 phase 2 progress
Nov 12, 2026
6e2266f
docs(plans): ITL-897 notes for the merge of ITL-981 from main
Nov 12, 2026
6308943
merge: origin/main (859106f7, with ITL-981 #328) into ITL-897
Nov 13, 2026
774ef65
docs(plans): ITL-897 notes for the final merge of main
Nov 13, 2026
894a2e8
merge: origin/main into ITL-897 (review round 1)
Nov 13, 2026
536c570
fix(pandas): hash the axes and nested cells as storage restores them;…
Nov 13, 2026
0457527
test(pandas): each order tested alone in the pipeline; fixtures state…
Nov 13, 2026
f0f65c1
docs(pandas): hashing reference, changelog and design issue for the r…
Nov 13, 2026
b80272d
merge: origin/main (a53182ce, with Polars value types #286) into ITL-897
Nov 13, 2026
f7252ae
build(deps): require pyarrow>=22.0.0, which restores DataFrame.attrs …
Nov 13, 2026
e7eadae
chore(types): basedpyright-clean order-variant fixture (ITL-897)
Nov 13, 2026
8a8b2ce
chore(types): typed column selection and level swap in the order-vari…
Nov 13, 2026
5277dc4
fix(pandas): describe both axes and every index level as storage rest…
Nov 13, 2026
272f202
fix(pandas): refuse values whose labels read back as duplicates; stat…
Nov 13, 2026
5fc8ed3
fix(pandas): refuse attrs that are not JSON, in the hash and in stora…
Nov 13, 2026
4361495
docs(pandas): the round-two rules in the hashing reference, changelog…
Nov 13, 2026
7be7625
merge: origin/main (f51ce67c) into ITL-897 (review round 2)
Nov 13, 2026
42b0dcd
fix(pandas): categorical categories as storage restores them; attrs k…
Nov 13, 2026
d20d214
test(pandas): keep the storage matrix to a few seconds (ITL-897)
Nov 13, 2026
bbbecea
merge: origin/main (003d2e82) into ITL-897 (review round 3)
Nov 13, 2026
f666b42
merge: origin/main (f6d15fbe, with torch.Tensor #300) into ITL-897
Nov 13, 2026
e76f4d0
fix(pandas): never copy attrs while hashing; Arrow dtype of the colum…
Nov 13, 2026
bcffd33
merge: origin/main (with #343) into ITL-897
Nov 13, 2026
1a49e06
merge: origin/main (with #344) into ITL-897
Nov 13, 2026
af4c1bc
merge: origin/main (with #347) into ITL-897
Nov 13, 2026
8610593
fix(pandas): columns described with the dtype the schema restores; ti…
Nov 13, 2026
f1c7177
merge: origin/main into ITL-897 (review round 5)
Nov 13, 2026
217d72e
chore(types): import tzinfo_to_string from pyarrow.lib (ITL-897)
Nov 13, 2026
f2dcd58
fix(pandas): digest a column that reads back as Arrow null as all-nul…
Nov 13, 2026
c765c0b
Merge remote-tracking branch 'origin/main' into eywalker/itl-897-pand…
Nov 13, 2026
c6fa26d
fix(pandas): keep pd.ArrowDtype(null) distinct from large_string in t…
Nov 13, 2026
f61101e
Merge remote-tracking branch 'origin/main' into eywalker/itl-897-pand…
Nov 13, 2026
58af489
chore(deps): cap pandas below 3 for v0.3.0 (ITL-897, ITL-1054)
Nov 13, 2026
034b362
fix(pandas): a non-nested pd.ArrowDtype is described by its Arrow typ…
Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions changelog.d/ITL-897.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
### Breaking changes and migration

#### pandas values: a strict, content-based hash; column order is identity (ITL-897, ITL-887)

A `pandas.DataFrame` or `pandas.Series` value now hashes under its own label,
`pandas_dataframe_v1` / `pandas_series_v1`, instead of as an Arrow table (`arrow_v1`).
Storage and hashing are separate: a value is stored with pyarrow's own conversion (unchanged),
and the hash is defined by Orcapod and is about the content of the value. The hash is strict:
everything about a value that can be changed is part of its identity, except a property
storage does not restore (developer, November 11 and 12, 2026).

Before, these pairs of values shared a hash, so a cached pod could be handed the result
computed for the other one (for example the wrong `df.columns[0]`):

| Property | Before | Now |
|---|---|---|
| Column order | not hashed | hashed |
| MultiIndex level order (frame and Series) | not hashed | hashed |
| Struct field order in a `pd.ArrowDtype` struct column | not hashed | hashed |
| `columns.name` / `columns.names` | not hashed | hashed |
| Dtype of the column axis, per level (an `int64` versus an `object` Index of the same labels) | not hashed | hashed |
| Column label type (`0` versus `"0"`), MultiIndex columns versus flat labels | not hashed | hashed |
| Series name type (`0` versus `"0"`); a frame versus its one-column Series | not hashed | hashed |
| The exact Arrow type of a non-nested `pd.ArrowDtype` column (`binary` versus `large_binary`, `null` versus `large_string`) | not hashed | hashed, as storage restores it (a nested type, and a decimal, which storage restores as `object`, are not distinguished; ITL-1054 decides nested types) |
| pandas dtype where the Arrow values coincide (`Int64` versus `int64`, `boolean` versus `bool`, `string` versus `object`) | not hashed | hashed |
| Categorical versus string; the category list (unused categories included), category order, `ordered` (ITL-887) | not hashed | hashed |
| `attrs` | not hashed | hashed, as the JSON object storage keeps |
| Values, row order, index values, index names and dtype, column dtype, Series name, `-0.0` versus `0.0` | hashed | hashed |
| `flags`, `RangeIndex` versus an equal `int64` index, an index's `freq`, unused MultiIndex levels, `string[python]` versus `string[pyarrow]`, key order of dicts in an `object` column | not hashed | never hashed: storage does not restore them (a `dict` is unordered) |
| Type of an index level name (`1` versus `"1"`) | hashed | never hashed: storage writes an index name as text, so both read back `"1"` |
| Types of column labels of mixed type (`0` beside `"b"`) | not hashed | never hashed: storage writes labels of mixed type as text |
| A masked, string or Arrow dtype of an index level (`Int64` versus `int64`) | not hashed | never hashed: storage restores an index level as NumPy |
| The dtype of a categorical's categories where storage changes it (`object` categories `[1, 2]` versus `int64`) | not hashed | never hashed: storage restores the categories from the Arrow dictionary |

A value read back from a store now always hashes like the value that was written. Before, an
`object` column of non-strings (`[1, 2]` reads back `int64`) hashed differently after the
round trip; such a column is now hashed in the form storage restores. The same holds for an
`object` column of lists or dicts (a list `[1, None]` reads back `[1.0, nan]`) and for index
level names (the name `1` reads back `"1"`).

Storing a pandas value that storage would corrupt now raises `ValueError`: column labels
and index names that collide once written as text (the index name `1` with a column `"1"`;
integer names of MultiIndex levels), and distinct labels that read back equal (the labels
`True` and `False`). Before, such a value was written and read back without its index (the
levels came back as columns) or with duplicate labels.

`attrs` must be JSON. `attrs` with an entry JSON cannot write (a `set`, a tuple as a key)
are refused by the hash and by storage with `ValueError` naming the key, on every supported
pyarrow version. Before, pyarrow dropped the whole `attrs` with a warning or raised
`TypeError`, depending on its version. A key that is not text is still accepted and is
written as text (`{1: "x"}` reads back `{"1": "x"}`).

The hash is `sha256(descriptor + b"\x00" + D)`: `D` is the Arrow hasher's ordered digest
(starfix 0.5 with `ordered_columns` and `ordered_struct_fields`) of the value's Arrow table
with positional field names, and the descriptor is canonical JSON in an Orcapod-owned grammar
of `attrs`, typed labels, `columns.names`, the column-axis dtype, index levels and pandas
dtypes. The definition and
the grammar are in `docs/reference/hashing.md`, section 10. A label or dtype outside the
grammar raises `ValueError` naming it.

#### pandas logical types are named `orcapod.pandas.dataframe` / `orcapod.pandas.series` (ITL-897)

The Arrow extension names of the two pandas logical types gain the `orcapod.` prefix (they
were `pandas.dataframe` / `pandas.series`), and the metadata is `{"ignore":[],"version":1}`:
the version, and the list of properties the data context configures the value hash to
ignore. Nothing is ignored by default. `hash_ignore` in the logical type registration file
can name `column_order`, `struct_field_order` and `attrs` for a frame, and `series_name`,
`struct_field_order` and `attrs` for a Series; two lists are two logical types, and a column
stored under another list is refused on read. The old names
are not registered any more: a column stored under them is refused on read, never read as
the current types or as raw bytes.

**What moves.** The hash of every pandas value, and with it the data hash and the cache keys
of every row that holds a pandas value. Because the name is part of the logical type
identifier, the type hash of `pd.DataFrame` / `pd.Series`, the schema hash, the pipeline hash
and so the table paths and system-tag column names of every node with a pandas column move
too, and of every node downstream of one. The stored bytes of a pandas value are unchanged.
v0.3 already recomputes everything a v0.2 store holds (ITL-910), and v0.2 tables are still
found by name, reported and left unchanged, so this adds no migration step for a released
store. A store written by an unreleased v0.3 development build computes nodes with a pandas
column once more, in new tables; the older tables stay in place, unchanged, and are not
reported.

#### Public interface: `ArrowHasherProtocol.hash_table_ordered` (ITL-897)

`ArrowHasherProtocol` gains a required method, `hash_table_ordered(table)`, implemented by
`StarfixArrowHasher`: the same procedure as `hash_table`, with column order and struct field
order at every depth part of the digest (`struct_field_order=False` leaves struct field
order out). `hash_table` is unchanged and stays order
independent (streams, datagrams, rows). A custom Arrow hasher must add the method.
`PANDAS_DATAFRAME_HASH_LABEL` and `PANDAS_SERIES_HASH_LABEL` in
`orcapod.hashing.pandas_value_hash` define the two labels, and
`PANDAS_IDENTITY_PROPERTIES` there lists every property and whether it is part of the hash.

#### pyarrow 22.0.0 or newer is required (ITL-897)

The minimum pyarrow version is raised from 20.0.0 to 22.0.0. pyarrow 20 and 21 do not
restore `DataFrame.attrs` when a frame is read back, so a frame with `attrs` would hash
differently after it was stored; from 22.0.0 `attrs` is restored. The locked version
(23.0.1) is unchanged.

### Known limitations

- A `pa.Table` or `pa.RecordBatch` given to the semantic hasher as a value still hashes
without regard to column order and struct field order (`arrow_v1`). Such a table cannot be
a pod input or output (ITL-897, second part).
- pandas is capped below 3 (`pandas>=2.2.3,<3`) for v0.3.0. The pandas value hash and its
round trip through storage are verified on pandas 2 only; pandas 3 changes the default
string dtype and other behaviour the hash depends on, and has not been checked. ITL-1054
lifts the cap.
37 changes: 37 additions & 0 deletions design_issues/ITL-897-PV1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
## `src/orcapod/hashing/semantic_hashing/builtin_handlers.py` — pandas value hash identity

### PV1 — pandas values that pandas tells apart hashed the same
**Status:** resolved
**Severity:** high
**Issue:** ITL-897, ITL-887

`PandasDataFrameHandler` and `PandasSeriesHandler` hashed `pa.Table.from_pandas(value)` with the
Arrow hasher, which ignores column order and struct field order. The hash also never saw what
pyarrow keeps only in its `pandas` schema metadata. So frames that `assert_frame_equal` tells
apart hashed the same when they differed only in column order, MultiIndex level order, struct
field order, label type (`0` vs `"0"`), `columns.name`, nullable vs NumPy dtype, `string` vs
`object`, or categorical vs string / categories / `ordered` (ITL-887). Storage kept every one of
them, so a cached pod returning `df.columns[0]` was handed the result computed for another
column order (sync; async once the cache is warm).

**Fix:** labels `pandas_dataframe_v1` / `pandas_series_v1`,
`sha256(descriptor + b"\x00" + ordered digest)`. The ordered digest is the Arrow hasher's new
`hash_table_ordered` of the positionally renamed `from_pandas` table: the same semantic
preprocessing as `hash_table`, then starfix 0.5's `ordered_columns` and
`ordered_struct_fields` options. Both entry points share one private starfix call. The
descriptor is an Orcapod-owned grammar of `attrs`, typed labels, `columns.names`, the
column-axis dtype, index levels and pandas dtypes (`orcapod.hashing.pandas_value_hash`; `docs/reference/hashing.md`
section 10). Storage and hashing are separate and the hash is strict (developer, November 11
and 12, 2026): every property of a value counts, `attrs` included, except what storage does
not restore (`flags`, RangeIndex kind, index `freq`, unused MultiIndex levels, string
storage, the `object` dtype of a column that reads back typed, the type of an index level
name, the types of labels of mixed type, an extension dtype of an index level), which never
counts, so
`hash(read(write(v))) == hash(v)`. `PANDAS_IDENTITY_PROPERTIES` is the one list of these in
code. The logical types were renamed `orcapod.pandas.dataframe` / `orcapod.pandas.series`
with the shared ignore-list metadata of library value types (`{"ignore":[],"version":1}`,
ITL-981; options `column_order`, `struct_field_order`, `series_name`, `attrs`, none ignored
by default); the old names are no longer registered. Every pandas value hash moved
once, before the v0.3 tag; stored bytes are unchanged. The investigation is in
`superpowers/specs/2026-11-11-itl-897-pandas-hash-identity.md`. `pa.Table` values (same
rule) are not included and remain order independent.
4 changes: 2 additions & 2 deletions docs/concepts/logical-type-identifiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,8 @@ Orcapod's own types follow the same contract as yours. Their metadata is the JSO
| `orcapod.Directory` | `orcapod.directory` | `{"version":1}` |
| `numpy.ndarray` | `orcapod.numpy.ndarray` | `{"ignore":[],"version":1}` |
| `torch.Tensor` (optional, needs PyTorch) | `orcapod.torch.tensor` | `{"ignore":[],"version":1}` |
| `pandas.DataFrame` | `pandas.dataframe` | `{"version":1}` |
| `pandas.Series` | `pandas.series` | `{"version":1}` |
| `pandas.DataFrame` | `orcapod.pandas.dataframe` | `{"ignore":[],"version":1}` |
| `pandas.Series` | `orcapod.pandas.series` | `{"ignore":[],"version":1}` |
| `polars.DataFrame` | `orcapod.polars.dataframe` | `{"ignore":[],"version":1}` |
| `polars.Series` | `orcapod.polars.series` | `{"ignore":[],"version":1}` |

Expand Down
Loading
Loading