diff --git a/changelog.d/ITL-897.md b/changelog.d/ITL-897.md new file mode 100644 index 000000000..ba186c6e0 --- /dev/null +++ b/changelog.d/ITL-897.md @@ -0,0 +1,110 @@ +### Breaking changes and migration + +#### pandas values: a strict, content-based hash; column order is identity (ITL-897, ITL-887) + +A `pandas.DataFrame` or `pandas.Series` value now hashes under its own label, +`pandas_dataframe_v1` / `pandas_series_v1`, instead of as an Arrow table (`arrow_v1`). +Storage and hashing are separate: a value is stored with pyarrow's own conversion (unchanged), +and the hash is defined by Orcapod and is about the content of the value. The hash is strict: +everything about a value that can be changed is part of its identity, except a property +storage does not restore (developer, November 11 and 12, 2026). + +Before, these pairs of values shared a hash, so a cached pod could be handed the result +computed for the other one (for example the wrong `df.columns[0]`): + +| Property | Before | Now | +|---|---|---| +| Column order | not hashed | hashed | +| MultiIndex level order (frame and Series) | not hashed | hashed | +| Struct field order in a `pd.ArrowDtype` struct column | not hashed | hashed | +| `columns.name` / `columns.names` | not hashed | hashed | +| Dtype of the column axis, per level (an `int64` versus an `object` Index of the same labels) | not hashed | hashed | +| Column label type (`0` versus `"0"`), MultiIndex columns versus flat labels | not hashed | hashed | +| Series name type (`0` versus `"0"`); a frame versus its one-column Series | not hashed | hashed | +| The exact Arrow type of a non-nested `pd.ArrowDtype` column (`binary` versus `large_binary`, `null` versus `large_string`) | not hashed | hashed, as storage restores it (a nested type, and a decimal, which storage restores as `object`, are not distinguished; ITL-1054 decides nested types) | +| pandas dtype where the Arrow values coincide (`Int64` versus `int64`, `boolean` versus `bool`, `string` versus `object`) | not hashed | hashed | +| Categorical versus string; the category list (unused categories included), category order, `ordered` (ITL-887) | not hashed | hashed | +| `attrs` | not hashed | hashed, as the JSON object storage keeps | +| Values, row order, index values, index names and dtype, column dtype, Series name, `-0.0` versus `0.0` | hashed | hashed | +| `flags`, `RangeIndex` versus an equal `int64` index, an index's `freq`, unused MultiIndex levels, `string[python]` versus `string[pyarrow]`, key order of dicts in an `object` column | not hashed | never hashed: storage does not restore them (a `dict` is unordered) | +| Type of an index level name (`1` versus `"1"`) | hashed | never hashed: storage writes an index name as text, so both read back `"1"` | +| Types of column labels of mixed type (`0` beside `"b"`) | not hashed | never hashed: storage writes labels of mixed type as text | +| A masked, string or Arrow dtype of an index level (`Int64` versus `int64`) | not hashed | never hashed: storage restores an index level as NumPy | +| The dtype of a categorical's categories where storage changes it (`object` categories `[1, 2]` versus `int64`) | not hashed | never hashed: storage restores the categories from the Arrow dictionary | + +A value read back from a store now always hashes like the value that was written. Before, an +`object` column of non-strings (`[1, 2]` reads back `int64`) hashed differently after the +round trip; such a column is now hashed in the form storage restores. The same holds for an +`object` column of lists or dicts (a list `[1, None]` reads back `[1.0, nan]`) and for index +level names (the name `1` reads back `"1"`). + +Storing a pandas value that storage would corrupt now raises `ValueError`: column labels +and index names that collide once written as text (the index name `1` with a column `"1"`; +integer names of MultiIndex levels), and distinct labels that read back equal (the labels +`True` and `False`). Before, such a value was written and read back without its index (the +levels came back as columns) or with duplicate labels. + +`attrs` must be JSON. `attrs` with an entry JSON cannot write (a `set`, a tuple as a key) +are refused by the hash and by storage with `ValueError` naming the key, on every supported +pyarrow version. Before, pyarrow dropped the whole `attrs` with a warning or raised +`TypeError`, depending on its version. A key that is not text is still accepted and is +written as text (`{1: "x"}` reads back `{"1": "x"}`). + +The hash is `sha256(descriptor + b"\x00" + D)`: `D` is the Arrow hasher's ordered digest +(starfix 0.5 with `ordered_columns` and `ordered_struct_fields`) of the value's Arrow table +with positional field names, and the descriptor is canonical JSON in an Orcapod-owned grammar +of `attrs`, typed labels, `columns.names`, the column-axis dtype, index levels and pandas +dtypes. The definition and +the grammar are in `docs/reference/hashing.md`, section 10. A label or dtype outside the +grammar raises `ValueError` naming it. + +#### pandas logical types are named `orcapod.pandas.dataframe` / `orcapod.pandas.series` (ITL-897) + +The Arrow extension names of the two pandas logical types gain the `orcapod.` prefix (they +were `pandas.dataframe` / `pandas.series`), and the metadata is `{"ignore":[],"version":1}`: +the version, and the list of properties the data context configures the value hash to +ignore. Nothing is ignored by default. `hash_ignore` in the logical type registration file +can name `column_order`, `struct_field_order` and `attrs` for a frame, and `series_name`, +`struct_field_order` and `attrs` for a Series; two lists are two logical types, and a column +stored under another list is refused on read. The old names +are not registered any more: a column stored under them is refused on read, never read as +the current types or as raw bytes. + +**What moves.** The hash of every pandas value, and with it the data hash and the cache keys +of every row that holds a pandas value. Because the name is part of the logical type +identifier, the type hash of `pd.DataFrame` / `pd.Series`, the schema hash, the pipeline hash +and so the table paths and system-tag column names of every node with a pandas column move +too, and of every node downstream of one. The stored bytes of a pandas value are unchanged. +v0.3 already recomputes everything a v0.2 store holds (ITL-910), and v0.2 tables are still +found by name, reported and left unchanged, so this adds no migration step for a released +store. A store written by an unreleased v0.3 development build computes nodes with a pandas +column once more, in new tables; the older tables stay in place, unchanged, and are not +reported. + +#### Public interface: `ArrowHasherProtocol.hash_table_ordered` (ITL-897) + +`ArrowHasherProtocol` gains a required method, `hash_table_ordered(table)`, implemented by +`StarfixArrowHasher`: the same procedure as `hash_table`, with column order and struct field +order at every depth part of the digest (`struct_field_order=False` leaves struct field +order out). `hash_table` is unchanged and stays order +independent (streams, datagrams, rows). A custom Arrow hasher must add the method. +`PANDAS_DATAFRAME_HASH_LABEL` and `PANDAS_SERIES_HASH_LABEL` in +`orcapod.hashing.pandas_value_hash` define the two labels, and +`PANDAS_IDENTITY_PROPERTIES` there lists every property and whether it is part of the hash. + +#### pyarrow 22.0.0 or newer is required (ITL-897) + +The minimum pyarrow version is raised from 20.0.0 to 22.0.0. pyarrow 20 and 21 do not +restore `DataFrame.attrs` when a frame is read back, so a frame with `attrs` would hash +differently after it was stored; from 22.0.0 `attrs` is restored. The locked version +(23.0.1) is unchanged. + +### Known limitations + +- A `pa.Table` or `pa.RecordBatch` given to the semantic hasher as a value still hashes + without regard to column order and struct field order (`arrow_v1`). Such a table cannot be + a pod input or output (ITL-897, second part). +- pandas is capped below 3 (`pandas>=2.2.3,<3`) for v0.3.0. The pandas value hash and its + round trip through storage are verified on pandas 2 only; pandas 3 changes the default + string dtype and other behaviour the hash depends on, and has not been checked. ITL-1054 + lifts the cap. diff --git a/design_issues/ITL-897-PV1.md b/design_issues/ITL-897-PV1.md new file mode 100644 index 000000000..fd1daf428 --- /dev/null +++ b/design_issues/ITL-897-PV1.md @@ -0,0 +1,37 @@ +## `src/orcapod/hashing/semantic_hashing/builtin_handlers.py` — pandas value hash identity + +### PV1 — pandas values that pandas tells apart hashed the same +**Status:** resolved +**Severity:** high +**Issue:** ITL-897, ITL-887 + +`PandasDataFrameHandler` and `PandasSeriesHandler` hashed `pa.Table.from_pandas(value)` with the +Arrow hasher, which ignores column order and struct field order. The hash also never saw what +pyarrow keeps only in its `pandas` schema metadata. So frames that `assert_frame_equal` tells +apart hashed the same when they differed only in column order, MultiIndex level order, struct +field order, label type (`0` vs `"0"`), `columns.name`, nullable vs NumPy dtype, `string` vs +`object`, or categorical vs string / categories / `ordered` (ITL-887). Storage kept every one of +them, so a cached pod returning `df.columns[0]` was handed the result computed for another +column order (sync; async once the cache is warm). + +**Fix:** labels `pandas_dataframe_v1` / `pandas_series_v1`, +`sha256(descriptor + b"\x00" + ordered digest)`. The ordered digest is the Arrow hasher's new +`hash_table_ordered` of the positionally renamed `from_pandas` table: the same semantic +preprocessing as `hash_table`, then starfix 0.5's `ordered_columns` and +`ordered_struct_fields` options. Both entry points share one private starfix call. The +descriptor is an Orcapod-owned grammar of `attrs`, typed labels, `columns.names`, the +column-axis dtype, index levels and pandas dtypes (`orcapod.hashing.pandas_value_hash`; `docs/reference/hashing.md` +section 10). Storage and hashing are separate and the hash is strict (developer, November 11 +and 12, 2026): every property of a value counts, `attrs` included, except what storage does +not restore (`flags`, RangeIndex kind, index `freq`, unused MultiIndex levels, string +storage, the `object` dtype of a column that reads back typed, the type of an index level +name, the types of labels of mixed type, an extension dtype of an index level), which never +counts, so +`hash(read(write(v))) == hash(v)`. `PANDAS_IDENTITY_PROPERTIES` is the one list of these in +code. The logical types were renamed `orcapod.pandas.dataframe` / `orcapod.pandas.series` +with the shared ignore-list metadata of library value types (`{"ignore":[],"version":1}`, +ITL-981; options `column_order`, `struct_field_order`, `series_name`, `attrs`, none ignored +by default); the old names are no longer registered. Every pandas value hash moved +once, before the v0.3 tag; stored bytes are unchanged. The investigation is in +`superpowers/specs/2026-11-11-itl-897-pandas-hash-identity.md`. `pa.Table` values (same +rule) are not included and remain order independent. diff --git a/docs/concepts/logical-type-identifiers.md b/docs/concepts/logical-type-identifiers.md index d7d69daf4..a6db58a16 100644 --- a/docs/concepts/logical-type-identifiers.md +++ b/docs/concepts/logical-type-identifiers.md @@ -79,8 +79,8 @@ Orcapod's own types follow the same contract as yours. Their metadata is the JSO | `orcapod.Directory` | `orcapod.directory` | `{"version":1}` | | `numpy.ndarray` | `orcapod.numpy.ndarray` | `{"ignore":[],"version":1}` | | `torch.Tensor` (optional, needs PyTorch) | `orcapod.torch.tensor` | `{"ignore":[],"version":1}` | -| `pandas.DataFrame` | `pandas.dataframe` | `{"version":1}` | -| `pandas.Series` | `pandas.series` | `{"version":1}` | +| `pandas.DataFrame` | `orcapod.pandas.dataframe` | `{"ignore":[],"version":1}` | +| `pandas.Series` | `orcapod.pandas.series` | `{"ignore":[],"version":1}` | | `polars.DataFrame` | `orcapod.polars.dataframe` | `{"ignore":[],"version":1}` | | `polars.Series` | `orcapod.polars.series` | `{"ignore":[],"version":1}` | diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 19b042f60..107623b94 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -57,6 +57,8 @@ The labels of v0.3: | `arrow_v1` | the Arrow table and schema hash (§9) | `ARROW_HASH_LABEL`, `hashing/arrow_hashers.py` | | `directory_v1` | the directory hash: a SHA-256 Merkle tree over the entries | `DIRECTORY_HASH_LABEL`, `hashing/directory_hashers.py` | | `ndarray_v1` | the array value hash (§8) | `NumpyArrayHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` | +| `pandas_dataframe_v1` | the `pandas.DataFrame` value hash (§10) | `PANDAS_DATAFRAME_HASH_LABEL`, `hashing/pandas_value_hash.py` | +| `pandas_series_v1` | the `pandas.Series` value hash (§10) | `PANDAS_SERIES_HASH_LABEL`, `hashing/pandas_value_hash.py` | | `torch_tensor_v1` | the torch tensor value hash (§8): the element digest, then `requires_grad`, the lazy conjugate and negative bits, the strides and the class | `TORCH_TENSOR_HASH_LABEL`, `logical_types/torch_type.py` | | `polars_expr_v1` | the Polars expression hash (§1c) | `POLARS_EXPR_HASH_LABEL`, `hashing/semantic_hashing/polars_expr.py` | | `polars_dataframe_v1` | the Polars DataFrame value hash (§1e) | `PolarsDataFrameHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` | @@ -569,8 +571,8 @@ struct field order does the same for every struct's fields with `ordered_struct_fields=False`; ignoring a Series' name writes `""` for `"name"` in the descriptor and hashes its values as a column named `""`. -A `pandas.DataFrame` is hashed differently: as the `arrow_v1` hash of its Arrow table, which -ignores column order. A Polars frame and the equal pandas frame never share a hash. +A `pandas.DataFrame` is hashed differently: with the `pandas_dataframe_v1` hash (§10), which +counts column order. A Polars frame and the equal pandas frame never share a hash. --- @@ -1186,7 +1188,7 @@ they hash Python types, where a logical type contributes its identifier text onl Every Arrow-derived hash in the sites above (row data hashes, `record_id`s built from a default `source_id`, base and pipeline entry ids, operator and cached-source record hashes, -`pa.Table` and pandas values) comes from `StarfixArrowHasher.hash_table`, which returns +`pa.Table` values) comes from `StarfixArrowHasher.hash_table`, which returns `ContentHash(method="arrow_v1", digest=D)`. `D` is 35 bytes: the starfix hash format number `00 00 02` followed by a SHA-256 digest, computed by `starfix` 0.5 (hash format 2) and byte-identical to the Rust `starfix` crate. The byte layout is specified in starfix's @@ -1205,7 +1207,12 @@ table would get a different digest. Its number is not kept equal to the starfix strings, binaries and lists hashing alike), nullability at every depth (top-level columns, struct fields and list items, whether or not the column holds data), every value, row order, list element order, list lengths at every depth, null structs (distinct from structs whose fields are all -null) and the row count. Column order and struct field order are not part of the hash. How a +null) and the row count. Column order and struct field order are not part of the hash: +this is the hash of Orcapod's own tables (streams, datagrams, rows), which ignore order on +purpose. A value whose own equality depends on order is hashed with +`StarfixArrowHasher.hash_table_ordered` instead, which is the same procedure with starfix's +`ordered_columns` and `ordered_struct_fields` options on, so that column order and struct +field order at every depth are part of the digest; pandas values use it (§10). How a table is split into chunks or slices is not part of it either. A `fixed_size_list[n]` column hashes differently from a `large_list` column with the same values. Extension columns are hashed as their storage type plus their extension name and metadata, including @@ -1366,3 +1373,233 @@ digest = hasher.hash_table(pa.table({"x": pa.array([1, 2, 3], pa.int64())})) print(digest.method, digest.digest[:3].hex()) # arrow_v1 000002 ``` + +## 10. pandas Value Hash (`pandas_dataframe_v1`, `pandas_series_v1`) + +A `pandas.DataFrame` or `pandas.Series` value (a pod input or output, a value in a source +column, or any object given to the semantic hasher) is hashed by +`pandas_frame_content_hash` / `pandas_series_content_hash` in +`orcapod/hashing/pandas_value_hash.py`. + +The pandas dependency is capped below 3 (`pandas>=2.2.3,<3`) for v0.3.0: the hash and its +round trip are verified on pandas 2 only. ITL-1054 lifts the cap. + +Storage and hashing are separate (developer, November 12, 2026). Storage is the library's +own serialization: the Arrow IPC stream of `pa.Table.from_pandas(value, +preserve_index=True)`, read back with `to_pandas()`. The hash is defined by Orcapod and is +about the content of the value; it is never a hash of the stored bytes. + +```text +ContentHash("pandas_dataframe_v1" | "pandas_series_v1", + sha256(canonical_json(descriptor) + b"\x00" + D)) +``` + +- `D` is the 35-byte digest of `StarfixArrowHasher.hash_table_ordered` (§9) over the + **values table**: `pa.Table.from_pandas(value, preserve_index=True)` with its fields + renamed by position, `c0 .. cN-1` for the columns in frame order, then `i0 .. iM-1` for + the index levels in order. A Series is digested as its one-column frame. `D` covers the + values, the Arrow types, row order, the row count, **column order, index level order and + struct field order**. +- `descriptor` is canonical JSON (sorted keys, `,` and `:` separators, UTF-8) in a grammar + Orcapod owns, built from the pandas objects themselves. It carries what the Arrow values + do not show: `attrs`, typed labels, `columns.names`, the dtype of the column axis, index + level names and the pandas dtypes. +- The label never enters the computation; only the bytes of `D` do. + +```text +frame: {"kind": "orcapod.pandas.dataframe", "height": , "attrs": , + "columns": [[