Repository navigation
fix(hashing): dict and set values hash the same whatever order they were built in (ITL-905) #301
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
459934a
1c888a2
502e0fa
072f61d
39179f5
40eb82b
bf28f71
210a75d
24732a5
984c524
44e7aa8
f63602c
2994073
bb53363
cb8eb1d
9ee3854
fcb9f75
fcd819b
d0ff008
388bb60
67fd4d2
66c6aac
c0f40fb
a32387c
37c8913
f4897ad
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| ### Breaking changes and migration | ||
|
|
||
| #### A `dict` or `set` value hashes the same whatever order it was built in (ITL-905) | ||
|
|
||
| A `dict[K, V]` value is stored as `large_list<struct<key, value>>` and a plain `set[T]` | ||
| (element type without a logical type) as `large_list<T>`. Both were stored in Python | ||
| iteration order, so `{"a": 1, "b": 2}` and `{"b": 2, "a": 1}` had different row hashes, | ||
| and the same `set[str]` had a different row hash in every process (`PYTHONHASHSEED`). | ||
| Orcapod now stores dict entries sorted by key and set elements sorted, wherever a Python | ||
| value becomes Arrow; see the hashing reference, section 1e, for the order. A NaN key or | ||
| element, values that cannot be compared, and two keys that convert to the same stored | ||
| value raise `CanonicalOrderError` (a `ValueError`) naming the column. A datagram built from | ||
| Python values (and values set with `update` or `with_columns`) holds its dicts in the same | ||
| order, so a newly computed pod output and one served from the cache iterate alike. Dicts | ||
| nested in dataclass or pydantic instances are the exception: a fresh instance keeps insertion | ||
| order, while a read-back instance uses key order. An Arrow table is still hashed exactly as | ||
| stored: a key/value list that arrives in Arrow form is never re-sorted. | ||
|
|
||
| Set values passed to `update` after Arrow materialization, or to `with_columns` with an | ||
| explicit Arrow list field, are also stored in canonical order. The existing Arrow field | ||
| (width, nullability and metadata) is preserved; actual list values keep their supplied order. | ||
|
|
||
| Which hashes move because of this change, on top of the v0.3 hash step (ITL-910, ITL-945), | ||
| which moves every stored hash: only those of values holding a dict whose insertion order was | ||
| not key order, or a `set[T]` of two or more elements whose iteration order was not sorted | ||
| (for `str` and `bytes` that order changed with every process, so it had no stable value). | ||
| For such a row, its data hash moves, and with it the function pod cache key, the pipeline | ||
| record and entry ids, and the default `source_id` (with its per-row record ids) of a | ||
| source built without an explicit `source_id`. Schema hashes, `pipeline_hash` values and | ||
| explicit `source_id`s do not move for this reason, and a dict already built in key order is | ||
| stored byte-for-byte as before if every nested unordered container is also canonical. | ||
| In the golden hash suite exactly two entries moved: | ||
| `pipeline:index_pick.index.row[{"subject":"s1"}].data_hash` and `...s2...`, whose rows hold | ||
| `meta={"k": ..., "j": ...}`. | ||
|
|
||
| What a `0.2.0` store sees (checked against a Delta store written by `0.2.0`): nothing | ||
| changes because of this fix. v0.3 reads no table `0.2.0` wrote (every table is at a new path, | ||
| see "Data context `std:v1:default`", ITL-944), so every input is computed again whether its | ||
| dict was in key order or not, and no row stored in `0.2.0`'s insertion order is ever served. | ||
| The `0.2.0` tables stay where they are, untouched. Every dict and plain set a v0.3 store | ||
| holds from a pod output or a source built from Python values is in canonical order. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| ## `src/orcapod/semantic_types/canonical_order.py` — dict and set values in canonical order | ||
|
|
||
| ### DO1 — A `dict` value's insertion order, or a `set`'s iteration order, changed its row hash | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-905 | ||
|
|
||
| A `dict[K, V]` value is stored as `large_list<struct<key, value>>` and a plain `set[T]` as | ||
| `large_list<T>`. The converter wrote both in Python iteration order, so `{"a": 1, "b": 2}` | ||
| and `{"b": 2, "a": 1}` had different row hashes (and missed each other's function pod cache | ||
| entries), while the semantic hasher, which sorts dict keys, gave them the same hash. A | ||
| `set[str]` was stored in an order that depends on `PYTHONHASHSEED`, so its row hash changed | ||
| from process to process. `set[<logical type>]` already sorted (`ListLogicalType`). | ||
|
|
||
| **Fix:** the developer ruled (November 11, 2026) that a dict and a set are order independent, | ||
| that Orcapod sorts them before storing them into Arrow, and that hashing an Arrow table never | ||
| sorts anything (starfix ignores only column order and struct field order). The converter's | ||
| `dict` branch sorts entries by storage key and a new `set` branch sorts elements, both with | ||
| the one ordering rule in `semantic_types/canonical_order.py` (None first; numeric; `str` by | ||
| code point; `bytes` bytewise; dates and times naturally). NaN, incomparable values and two | ||
| keys or elements with the same storage value raise `CanonicalOrderError`, wrapped with the | ||
| column name by the caller (decided November 12, 2026). `Datagram` built from Python values | ||
| holds key-ordered copies (also decided November 12, 2026) of | ||
| its dict values (inside `list`, `dict`, `Optional`), and `update` and `with_columns` do the | ||
| same, so a newly computed output and one served from the cache iterate alike (v0.3 reads no | ||
| table `0.2.0` wrote, so no row in `0.2.0`'s insertion order is ever served); the caller's dict | ||
| is never mutated. Set inputs written through an existing Arrow list schema also use the | ||
| same ordering rule, preserving the exact field; list inputs retain their supplied order. | ||
| On top of the v0.3 hash step, hashes move only for values not already | ||
| in canonical order (two golden entries, `pipeline:index_pick.index` row data hashes); see | ||
| the `0.3.0` changelog. This storage order is a separate layer from the semantic hasher's | ||
| ordering of its hash input (ITL-942), which sorts by canonical JSON text and stores nothing. Regressions: `tests/test_hashing/test_dict_order_hash.py` (including a | ||
| Delta store written by `0.2.0`, which v0.3 does not read) and `tests/test_hashing/test_set_order_hash.py`. Open: a | ||
| key/value list that arrives already in Arrow form keeps its stored order and hash, by the | ||
| same ruling; dicts inside dataclass or pydantic objects keep their own order in memory | ||
| (their stored and hashed form is canonical). A fresh user-class instance iterates its | ||
| nested dict in insertion order, while a read-back instance iterates it in key order; | ||
| an order-sensitive pod can observe that difference for equal hashed inputs. Reconstructing | ||
| user-class instances is outside this fix (coordinator decision, November 12, 2026). | ||
| `test_dataclass_dict_storage_is_canonical_but_fresh_object_is_unchanged` pins equal stored | ||
| values and hashes, the fresh/read-back iteration difference, and caller non-mutation. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,8 +29,10 @@ | |
| from orcapod import contexts | ||
| from orcapod.config import OrcapodConfig | ||
| from orcapod.core.base import ContentIdentifiableBase | ||
| from orcapod.errors import CanonicalOrderError | ||
| from orcapod.protocols.semantic_types_protocols import TypeConverterProtocol | ||
| from orcapod.semantic_types import infer_python_schema_from_pylist_data | ||
| from orcapod.semantic_types.canonical_order import canonical_python_value | ||
| from orcapod.system_constants import constants | ||
| from orcapod.types import ColumnConfig, DataType, DataValue, Schema, SchemaLike | ||
| from orcapod.utils import arrow_utils | ||
|
|
@@ -119,6 +121,9 @@ def _init_from_dict( | |
| if python_schema | ||
| else inferred | ||
| ) | ||
| self._data_dict = self._canonical_dict_values( | ||
| data_columns, self._data_python_schema | ||
| ) | ||
| self._data_arrow_schema: pa.Schema | None = None | ||
|
|
||
| if meta_info is not None: | ||
|
|
@@ -130,6 +135,28 @@ def _init_from_dict( | |
| self._meta_table: pa.Table | None = None | ||
| self._context_table: pa.Table | None = None | ||
|
|
||
| def _canonical_dict_values( | ||
| self, columns: dict[str, DataValue], schema: Schema | ||
| ) -> dict[str, DataValue]: | ||
| """Return ``columns`` with every dict value in canonical key order. | ||
|
|
||
| A dict is order independent (ITL-905): its entries are stored and hashed in | ||
| canonical key order, so a value built here holds the same order a stored copy | ||
| reads back with. Dicts are copied, never mutated; dicts inside other objects | ||
| (dataclass or pydantic fields) are left as given. | ||
| """ | ||
| out: dict[str, DataValue] = {} | ||
| for name, value in columns.items(): | ||
| try: | ||
| out[name] = canonical_python_value( | ||
| value, | ||
| schema.get(name), | ||
| self.converter.get_python_to_arrow_converter, | ||
| ) | ||
|
Comment on lines
+151
to
+155
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| except CanonicalOrderError as exc: | ||
| raise CanonicalOrderError(f"Column {name!r}: {exc}") from exc | ||
| return out | ||
|
|
||
| def _init_from_table( | ||
| self, | ||
| table: pa.Table, | ||
|
|
@@ -822,8 +849,13 @@ def update(self, **updates: DataValue) -> Self: | |
| return new_d | ||
| else: | ||
| assert self._data_dict is not None | ||
| # Updated dict values are held in canonical key order, as at | ||
| # construction (ITL-905); the caller's dicts are copied, not mutated. | ||
| canonical_updates = self._canonical_dict_values( | ||
| updates, self._ensure_python_schema() | ||
| ) | ||
| new_d = self.copy(include_cache=False) | ||
| new_d._data_dict = {**self._data_dict, **updates} | ||
| new_d._data_dict = {**self._data_dict, **canonical_updates} | ||
| new_d._data_table = None | ||
| return new_d | ||
|
|
||
|
|
@@ -982,7 +1014,9 @@ def with_columns( | |
| {k: schema.get(k, inferred.get(k, str)) for k in new_data} | ||
| ) | ||
| new_d = self.copy(include_cache=False) | ||
| new_d._data_dict = new_data | ||
| # New dict values are held in canonical key order, as at construction | ||
| # (ITL-905); the caller's dicts are copied, not mutated. | ||
| new_d._data_dict = self._canonical_dict_values(new_data, new_schema) | ||
| new_d._data_python_schema = new_schema | ||
| new_d._data_table = None | ||
| new_d._data_arrow_schema = None | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 42561
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 13970
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 29623
Web query:
starfix 0.4.0 ArrowDigester float signed zero hash canonicalization source codeResult:
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 42666
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 24381
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 3509
Document the signed-zero exception.
Python considers
{0.0: 1}and{-0.0: 1}equal. The converter preserves the key representation, andStarfixArrowHasherhashes its bytes. These dicts can therefore have different row hashes.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Accepted, documented (fcd819b). Confirmed:
{0.0: 1} == {-0.0: 1}but the row hashes differ, because a float is hashed by its stored bytes. This is not specific to dicts and sets: a plainfloatcolumn holding0.0vs-0.0hashes differently too. Orcapod deliberately does not rewrite-0.0to0.0: pod code can tell them apart (math.copysign,repr, division), so the finer identity can cost a recompute but never serves a wrong result.docs/reference/hashing.mdsection 1c now states this as a general float rule, and thecanonical_order.pymodule docstring no longer says the two are "one value" (they collide only within one dict or set).Posted by a Claude Code agent on behalf of @eywalker.