Skip to content
63 changes: 63 additions & 0 deletions changelog.d/ITL-1054.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
### Breaking changes and migration

#### pandas values storage would not restore are refused at write (ITL-1054)

A pandas `DataFrame` or `Series` that storage would change, or could not read back, is now
refused when it is written, with a `ValueError` that names the column, index level or
column-axis level and the reason. The value hash refuses the same values with the same message,
through one shared check. These values used to be written:

- **Written, then read back changed, with no error:** a MultiIndex whose level names are also
level positions (`names=[1, 0]`; `1.0` and `True` count as `1`). pyarrow wrote it with the
levels swapped, and the hash did not tell the two apart.
- **Written, then failed to read back:** a `pd.ArrowDtype` column or Series of a list, large
list, fixed-size list, list view, struct, map, dictionary, run-end, union or fixed-size binary
type, or of an Arrow extension type (`uuid`, `json`, `bool8`, a fixed-shape tensor). An index level of a `pd.ArrowDtype` list, large list, fixed-size list, struct, map,
dictionary, fixed-size binary or `float16` type. A column axis (any level) that is
categorical, period, timedelta, interval, `datetime.date` objects, or a `pd.ArrowDtype`
timestamp, duration, time, bool, date, decimal or unsigned integer. This includes a Series named with a
`pd.Timedelta`, `pd.Period` or `datetime.date`. A categorical column, Series or index
whose categories are tuples (ITL-1115) or intervals. A `pd.ArrowDtype` `time64[ns]` index
level holding a value that is not a whole number of microseconds (ITL-1118), and a
MultiIndex level holding tuples or lists (ITL-1120); index levels are now converted once, in
full, at write. In a pipeline, such a result stayed in the result
table and could not be loaded on any rerun.
- **Written, then read back with another column label, with no error:** the label `False`
(a one-column bool axis, a Series named `False`, or a `False` in a level of a MultiIndex
column axis) read back as `True` (ITL-1116); an index level named like pyarrow's generated
field names (`__index_level_0__`) read back unnamed (ITL-1119).

A value Arrow cannot convert (for example `complex128` or a sparse column) now raises
`ValueError` naming the column, where it used to raise `ArrowNotImplementedError`, `TypeError`
or `NotImplementedError`.

No hash of a value that is still accepted changes, and the logical type identifiers are
unchanged. Rows already stored before this change are not migrated: a value of the first kind
cannot be told apart from a correct one, and a value of the second kind cannot be read. The
migration tool reports both (ITL-1048).

#### The pandas ignore option `struct_field_order` is removed (ITL-1054)

`struct_field_order` is no longer an option of `orcapod.pandas.dataframe` or
`orcapod.pandas.series`. It acted only on `pd.ArrowDtype` struct columns, which are now
refused. A data context that names it under `hash_ignore` is refused with the unknown-option
error, and so is `ignore=["struct_field_order"]` on the logical type or its handler. No shipped
data context named it, so no identifier or hash moves. A column stored on unreleased `main`
under a custom data context that named it is refused on read (`LogicalTypeIdentifierError`);
such results are recomputed under a data context without it. The pandas types are provisional until
the v0.4 audit (ITL-1069). That audit is where struct columns and this option come back
together, if storage learns to restore them. The Polars types keep their `struct_field_order`
option.

### Fixed

- The pandas storage round-trip test lets only the first write refuse a value. A read failure
of a written value, a failed second round trip, or a column or index level read back in
another position now fails the test, where it used to count as unsupported input.
(ITL-1054)

### Known limitations

- A `pd.ArrowDtype` decimal column is read back as an `object` column of `Decimal` values.
Every value comes back exactly, digits and scale included. The dtype and its declared
precision do not, so two precisions hash equal. (ITL-1054; left for a developer decision)
36 changes: 36 additions & 0 deletions design_issues/ITL-1054-PV3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## `src/orcapod/hashing/semantic_hashing/builtin_handlers.py` — pandas value hash identity

### PV3 — pandas values written that storage changed silently or could not read back
**Status:** resolved
**Severity:** high
**Issue:** ITL-1054

A pandas `DataFrame` or `Series` was written whenever pyarrow could convert it, but two kinds of
written value did not come back. First, a MultiIndex whose level names are also level positions
(`names=[1, 0]`; `1.0` and `True` count as `1`) was written with its levels swapped: pyarrow
fetches level `i` with `index.get_level_values(i)`, which pandas resolves as a level name first.
The value read back was another value, nothing raised, and the hash (computed on the restored
form) gave both the same digest. Second, a nested, dictionary, fixed-size or union
`pd.ArrowDtype` column or index level, a `float16` Arrow index level, and a categorical, period,
timedelta, interval, date or `pd.ArrowDtype` column axis, and categories that are tuples, were
written and then failed to read back, and so were a `time64[ns]` index level holding
sub-microsecond values and a MultiIndex level holding tuples or lists. A `False` column label (or Series name) read back as `True`, and an
index level named `__index_level_<n>__` read back unnamed. In a pipeline, the computed result stayed in the result table and could not be loaded on
any rerun. The storage round-trip matrix test hid the second kind: it treated a failed read
like an input storage does not support.

**Fix:** one check, `require_restorable(schema, frame)` in
`orcapod/hashing/pandas_value_hash.py`, shared by `python_to_storage` and the value hash. It
refuses both kinds at write with a `ValueError` naming the column, index level or column-axis
level. A failed read is detected from a zero-row read (`table.slice(0, 0).to_pandas()`, which
keeps the categories), which agreed with the full read on every value tried, on pyarrow 22.0.0,
23.0.1 and 26.0.1, together with a full conversion of each index level and one assembly of a MultiIndex (the
failures found that depend on values). Column labels and index level names are compared as text with those
read back. Future support: tuple categories ITL-1115, `False` labels ITL-1116, sub-microsecond
`time64[ns]` index ITL-1118, generated-looking index names ITL-1119, tuple or list MultiIndex
levels ITL-1120 (which also carries the untested candidates for the v0.4 audit).
`pandas_frame_to_arrow` wraps every Arrow conversion error into `ValueError`, naming the part
that fails. The matrix test now lets only the first write refuse, and it checks each column and
index level by position. The `struct_field_order` ignore option is removed, because it acted
only on the refused struct columns. Identifiers, labels and every golden value are unchanged.
Rows already stored with either kind are left for the migration tool (ITL-1048).
94 changes: 75 additions & 19 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -1452,10 +1452,11 @@ series: {"kind": "orcapod.pandas.series", "height": <rows>, "attrs": <attrs>,
| {"arrow": <Arrow type text>} pd.ArrowDtype of a non-nested Arrow type:
str(type), "large_binary", "null",
"timestamp[ns, tz=UTC]"
| "arrow" pd.ArrowDtype of a nested, dictionary or
run-end type: the type is in D only
```

A nested, dictionary or run-end `pd.ArrowDtype` has no term: storage cannot read such a
column back, so the value is refused (see "What is refused" below).

`kind` is the name of the value's logical type (its Arrow extension name).

**The hash is strict.** Everything about a value that can be changed is part of its
Expand All @@ -1468,7 +1469,6 @@ the value that was written. `PANDAS_IDENTITY_PROPERTIES` in
|---|---|---|
| Column order | yes | `D` (ordered) |
| MultiIndex level order (frame and Series) | yes | `D` (ordered) |
| Struct field order in a `pd.ArrowDtype` struct column, at every depth | yes | `D` (ordered) |
| Values, row order, row count, Arrow type (int width, datetime unit and timezone) | yes | `D` |
| Index values, index level names (as the text storage writes), index dtype (as storage restores it) | yes | `D`, descriptor |
| `columns.name` / `columns.names` | yes | descriptor |
Expand Down Expand Up @@ -1538,19 +1538,17 @@ More consequences of the same rule:
decimal column and the all-null or empty `object` column is `{"numpy": "object"}`, so they
**hash equal**; the height still counts, and a `string`-dtype column of `None` (descriptor
`"string"`) stays different. Other all-null or empty columns (Arrow string, timestamp,
integer, list or struct, `Int64`, `string`, categorical) keep their own types and are
integer, `Int64`, `string`, categorical) keep their own types and are
unchanged.
A `pd.ArrowDtype` of a non-nested Arrow type is described by its Arrow type text, so
`pd.ArrowDtype(pa.null())` (`{"arrow": "null"}`) never hashes like a
`pd.ArrowDtype(pa.large_string())` column of `None`, and `binary` and `large_binary`
columns (which storage restores as themselves, although D unifies large and non-large
variants) hash differently. A decimal column is the exception that shows the rule: storage
restores it as `object`, so two precisions hash equal. A nested `pd.ArrowDtype` (list,
struct, map, union), a dictionary or a run-end type keeps the bare term `"arrow"`; none
reads back today, and ITL-1054 decides what identifies one (so a `list` and a `large_list`
column hash equal until then).
A `pd.ArrowDtype` of a nested type with a `null` child (`list<null>`, `struct<a: null>`)
is still refused by the hash, as before.
restores it as `object` (each value comes back exactly, digits and scale included; the
dtype and its declared precision do not), so two precisions hash equal. A nested
`pd.ArrowDtype` (list, struct, map, union), a dictionary or a run-end type is refused:
storage cannot read it back (ITL-1054).
- **A timezone is the text storage writes for it** (`"UTC"`, `"Europe/Berlin"`,
`"+01:00"`), whatever class holds it: `datetime.timezone(timedelta(hours=1))` is read
back as another class of the same offset, and both hash equal.
Expand All @@ -1568,15 +1566,74 @@ warning, or raise `TypeError`, depending on its version). With `attrs` in the ig
the hash does not read them (it never copies them either, so `attrs` that cannot be copied
do not matter then).

Storage refuses a value it would corrupt. Column labels and index names that collide
once written as text (the index name `1` with a column `"1"`; integer names of MultiIndex
levels) would be written as two fields of one name, and the value read back would have lost
its index. Distinct labels that read back equal (the labels `True` and `False` both read
back `True`) would come back as duplicates. `python_to_storage` raises `ValueError` in both
cases.

A label or dtype outside the grammar raises `ValueError` naming it (a dtype error names its
column). A value that cannot be converted to Arrow raises `ValueError`.
column).

**What is refused.** Storage refuses, when the value is written, a value it would not
restore, and the hash refuses the same values with the same `ValueError` (one check,
`require_restorable` in `orcapod/hashing/pandas_value_hash.py`; ITL-1054). The message names
the column, index level or column-axis level and the reason. Refused, on every supported
pyarrow version:

- **MultiIndex levels that would be written in another order.** pyarrow fetches index level
`i` with `index.get_level_values(i)`, which pandas resolves as a level *name* first. So
level names that are also level positions (`names=[1, 0]`; `1.0` and `True` count as `1`)
would be written, and read back, with the levels swapped, and nothing would raise.
- **Values storage cannot read back.** A `pd.ArrowDtype` column (or Series) of a list,
large list, fixed-size list, list view, struct, map, dictionary, run-end or union type,
of fixed-size binary, or of an Arrow extension type (`uuid`, `json`, `bool8`, a fixed-shape
tensor); an index level of a `pd.ArrowDtype` list, large list, fixed-size list, struct,
map, dictionary, fixed-size binary or `float16` type; a column axis (any level of it) that
is categorical, period, timedelta, interval, `datetime.date` objects, or a `pd.ArrowDtype`
timestamp, duration, time, bool, date, decimal or unsigned integer, which includes a
Series named with a `pd.Timedelta`, `pd.Period` or `datetime.date`; a categorical column,
Series or index whose categories are tuples (they read back as arrays, which cannot be
categories; ITL-1115) or intervals.
- **Index levels whose values cannot be read back.** A `pd.ArrowDtype` `time64[ns]` index
level holding a value that is not a whole number of microseconds (ITL-1118), and a
MultiIndex level holding tuples or lists, which read back as arrays that cannot be the
values of a MultiIndex level (ITL-1120). These failures depend on the values, so each
index level is converted once, in full, and a MultiIndex is assembled once, when a value is
written or hashed; columns are checked without converting any row.
- **Index level names that would read back as another name.** pyarrow drops a name of the
form `__index_level_<n>__` (its own name for an unnamed level), so such a level reads back
unnamed (ITL-1119). Names are compared as the text storage writes for them.
- **Column labels that would read back as another label.** pyarrow restores a bool label
from its text, so the label `False` (a one-column bool axis, a Series named `False`, a
`False` in a level of a MultiIndex column axis) reads back `True` (ITL-1116). Labels are
compared as text, so mixed-type labels, which storage writes as text, are not a change.
- **Labels and names that collide once written as text** (the index name `1` with a column
`"1"`; integer names of MultiIndex levels): two fields of one name, and the value read
back would have lost its index. Distinct labels that read back equal (the labels `True`
and `False` both read back `True`) would come back as duplicates.
- **Values Arrow cannot convert** (mixed or non-Arrow values, `complex128`, sparse columns,
a categorical of periods, a `float16` column axis, an index of a type Arrow does not
support).

Before v0.3 such values were written: a value of the first kind reads back changed, and one
of the second kind cannot be read at all.

```python
import pandas as pd
from orcapod.contexts import get_default_context
from orcapod.logical_types.pandas_type import LogicalPandasDataFrame

semantic_hasher = get_default_context().semantic_hasher
swapped = pd.DataFrame(
{"a": [10, 20]},
index=pd.MultiIndex.from_tuples([("p", 7), ("q", 8)], names=[1, 0]),
)
try:
LogicalPandasDataFrame().python_to_storage(swapped)
except ValueError as error:
print(str(error).split(":")[0])
# index level 0 (name 1)
try:
semantic_hasher.hash_object(swapped)
except ValueError as error:
print(str(error).split(":")[0])
# index level 0 (name 1)
```

**What a data context can make the hash ignore.** Nothing is ignored by default. The
registration file (`orcapod/contexts/logical_types.json`, `hash_ignore`) can name, per type,
Expand All @@ -1588,7 +1645,6 @@ The preimage keeps its shape: an ignored property is hashed with a fixed value.
|---|---|---|
| `column_order` | DataFrame | the columns are hashed sorted by the canonical JSON of their typed labels |
| `series_name` | Series | `name` is hashed as `null` |
| `struct_field_order` | both | starfix's `ordered_struct_fields` option is off for `D` |
| `attrs` | both | `attrs` is hashed as `{}` |

Every other property always counts, and the properties storage does not restore never do;
Expand Down
Loading
Loading