From 459934a02462c4de8be4ee06041121f37fd33b87 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:30:45 -0800 Subject: [PATCH 01/22] docs(plan): ITL-905 dict order hash plan and migration assessment Co-Authored-By: Claude Opus 5.5 --- .../2026-11-11-itl-905-dict-order-hash.md | 197 ++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100644 superpowers/plans/2026-11-11-itl-905-dict-order-hash.md diff --git a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md new file mode 100644 index 000000000..155481b9f --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md @@ -0,0 +1,197 @@ +# ITL-905: dict insertion order changes the row hash — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Equal dicts give equal row hashes on every path (Python and Arrow inputs, nested, sync and async, fresh and cached), because a `dict` is order independent (developer, November 11, 2026). + +**Architecture:** One ordering rule, defined once on Arrow *storage* key values (`semantic_types/dict_order.py`). It is applied in three places: (1) the converter sorts entries when a Python dict becomes `large_list>`; (2) the Arrow hasher sorts the entries of every dict-encoded list before starfix sees them, so tables that arrive already holding unsorted entries (Arrow/Delta sources, v0.2.0 stores, operator rebuilds) hash canonically; (3) a datagram built from Python values holds its dict values in the same canonical order, so a fresh value and a cached read-back value iterate identically. + +**Tech Stack:** Python, PyArrow, starfix 0.4 (unchanged), pytest via `uv run`. + +**Status:** STEP 1 (plan + migration assessment) — awaiting coordinator approval. No library code yet. + +--- + +## 0. Findings from the investigation (main `b197544e`) + +Reproduced: `{"a": 1, "b": 2}` and `{"b": 2, "a": 1}` give different row data hashes through `DictSource`. + +Extra observations: + +* An `ArrowTableSource` holding `large_list>` gets Python type `dict[K, V]`, and the two entry orders also hash differently. So the Arrow-native path is affected independently of the converter. +* Duplicate keys in an Arrow-native key/value list are accepted silently: `[{a,1},{a,2}]` hashes both entries but `as_dict()` returns `{'a': 2}`. The hash and the Python value disagree. +* `map` columns are rejected by the hasher today (`NotImplementedError: Unsupported leaf type: map<...>`), so they are not a path to fix here. +* A Python `TypedDict` with fields `key`/`value` is not a supported type, and no other supported Python type maps to `list>`. In Orcapod's type system that layout always reads back as `dict[K, V]` (`arrow_type_to_python_type`, `_create_arrow_to_python_converter`). +* **Out of scope, but serious (reported, not fixed):** a plain `set[str]` value is stored as `list` in Python set iteration order, which depends on `PYTHONHASHSEED`. The same set gives a different row hash in every process (verified with seeds 1, 2, 3). `set[]` already sorts (`ListLogicalType`), plain `set[T]` does not. +* Out of scope: `record_id_column` on a dict-valued column builds its token from `str(dict)`, which is insertion ordered. + +## 1. Every place a dict becomes Arrow (brief item a) + +Python → Arrow, all through `UniversalTypeConverter.get_python_to_arrow_converter` → `_create_python_to_arrow_converter`, `origin is dict` branch (`universal_converter.py:1596-1606`): + +| Path | Entry point | +|---|---| +| `DictSource`, `ListSource`, `DataFrameSource`, `PollingSource` | `python_dicts_to_arrow_table` | +| `Datagram` / `Data` / `Tag` from a Python dict (incl. function pod outputs, sync and async) | `Datagram._ensure_data_table` → `python_dicts_to_arrow_table` | +| `FunctionPodStream` / `FunctionNode` output tables | `python_dicts_to_struct_dicts` | +| dict nested in list / dict / tuple / Optional | the converter's own recursion | +| dict inside a dataclass / pydantic field | `DataclassLogicalType.python_to_storage` → `converter.python_to_storage` → same branch | +| dict as an element of `list[]` / `set[]` | not possible: a dict is not a logical type | + +So fix (1) in that one branch covers every Python-origin path. + +Arrow → hash without the converter: every `content_hash` of a table goes through `StarfixArrowHasher.hash_table` → `_process_table_columns`, which already walks every list/struct column value by value with `SemanticHashingVisitor`. Paths that reach it with entries in whatever order the table has: `ArrowTableSource` / `DeltaTableSource` / `CSVSource` input tables, rows read back from a result or pipeline store (incl. v0.2.0 stores), `DerivedSource`, and operators that rebuild tables (`Batch`, `GroupBy`, `MergeJoin`, `Join`, Polars round trips — none of them reorder list contents, but they pass through whatever order they were given). Fix (2) in the visitor covers all of them, sync and async (async operators hash through the same hasher). + +## 2. Canonical form (brief item b) + +**Rule:** entries of a dict are sorted ascending by key; the order is defined on the *converted (storage) key*, i.e. the value that is stored in Arrow and read back by `to_pylist()`. + +Why the storage key and not the Python key: +* The hasher only has storage values (fix 2). Defining the order on storage values gives one function used by both the converter and the hasher, so the converter's order and the hasher's order cannot drift. +* Python keys that convert to the same storage value (naive datetime under `coerce_utc` vs the same UTC-aware datetime) must collide, not sort apart. +* Converted datetimes are all timezone-aware, so they are mutually comparable; raw Python datetimes may mix naive and aware and raise. + +Sort key `dict_entry_sort_key(storage_key)`, total and independent of process, locale and platform: + +| Key type (storage) | Order | +|---|---| +| `None` (key type `K | None`) | first | +| `bool` | `False < True` | +| `int` (any width) | numeric | +| `float` | numeric; `-0.0` and `0.0` compare equal (they are one Python dict key); **NaN rejected** | +| `str` | Unicode code point (Python `str` comparison; equals UTF-8 byte order; never locale) | +| `bytes` | unsigned bytewise lexicographic | +| `date`, `datetime` (aware after conversion), `time`, `timedelta`, `Decimal` | natural order (datetime by instant) | +| `tuple[...]` (stored as `struct`) | lexicographic over fields in field order, each field by this table | + +* NaN keys: NaN is not equal to itself, so equality of dicts with NaN keys is identity based and has no canonical form. Raise `ValueError` naming the column and the key. +* Keys whose storage values are not mutually comparable (anything outside the table above): raise `TypeError` naming the column and the key type. Logical/extension key types are already rejected when the type is registered (ET1), so in practice this is a guard. +* **Duplicate keys** after conversion (Arrow-native duplicates, or two Python keys that convert to one storage value): raise `ValueError` naming the column and key. A dict cannot hold two entries for one key, and silently keeping both (hash) while `as_dict()` keeps one is the current inconsistency. +* Nested dicts: values are canonicalised recursively first (the converter recurses into values before sorting; the visitor visits elements before sorting), then the outer entries are sorted by key only. Values never take part in the order (keys are unique). + +**Why sorted dicts keep their hash:** canonicalisation only permutes entries; for a dict whose insertion order already equals the sorted order the stored list and the hashed bytes are byte-identical, so its hash does not move. This holds for str keys inserted in code-point order, ints ascending, etc. The golden suite proves it (section 5). + +**Alternative considered — order-independent hash in starfix (stored order left alone).** starfix would need a way to hash a list column as a multiset/map: a per-field option (e.g. field metadata `orcapod:dict` or an `unordered_list` flag) that combines per-entry digests order-independently (sort entry digests, or a commutative combine). That is a cross-language spec change in the Rust crate plus `starfix-python`, a release, and a dependency bump, and ITL-898 is adding the opposite option (order-sensitive columns/struct fields) right now. It would also leave stored and read-back values in insertion order, so a fresh value and a cached value of the same graph still iterate differently (item c). **Recommendation: canonical sort in Orcapod (fixes 1+2+3), no starfix change.** Hash-time sorting reuses the visitor's existing per-value walk, so its cost is one sort per dict value. + +## 3. What a pod reads back (brief item c) + +With sort-on-write alone, a cached/read-back value is key-sorted while a fresh in-memory value keeps the user's insertion order (`Data.as_dict()` returns the dict the function returned). `==` agrees, iteration order and `repr` do not. + +**Recommendation: canonicalise in memory too (fix 3).** When a `Datagram` is built from Python values, each field whose Python type contains a `dict` (directly or inside `list`/`tuple`/`dict`/`Optional`) is replaced by a **new** canonically ordered copy (no mutation of the user's object). Values inside dataclass/pydantic objects are not rebuilt (that would replace the user's object); their hash is canonical through fix 1, and this residual difference is documented on the hashing page. Then fresh and cached runs hand the downstream pod dicts that iterate identically. + +## 4. Tables that already hold unsorted entries (brief item d) + +* **On hash: yes** (fix 2). **On read: no.** Orcapod does not rewrite a user's Arrow table or stored rows (no mutation, no silent repair). A value read from an unsorted Arrow source or a v0.2.0 store keeps its stored order in `as_dict()`; only its hash is canonical. (Optionally a later issue can canonicalise at `ArrowTableSource` ingest; not proposed here.) +* **Is `list>` always a dict?** Yes in Orcapod's type system: the converter maps exactly that layout (a list — large, regular or fixed-size — whose element is a struct with exactly the two fields `key` and `value`, in either field order) to `dict[K, V]`, and no supported Python type produces it otherwise. A user who needs an ordered list of pairs uses `list[tuple[K, V]]` (stored `struct`, never sorted) or other field names. The detection becomes one helper, `is_dict_arrow_type`, used by the converter's two `arrow → python` branches and by the hasher, so "what is a dict" has a single definition. Any other list of structs (other field names, or extra fields) is never sorted. The hashing reference page will state this rule. + +## 5. Migration assessment (brief item e) + +Hashes that move — **only for values containing a dict whose stored entry order was not already key-sorted**: + +| Hash | Moves when | +|---|---| +| row data content hash (`Data.content_hash`, `INPUT_DATA_HASH_COL`, `OUTPUT_DATA_HASH_COL` of new rows) | the row holds such a dict at any depth | +| function pod cache key / result-store lookup | its input row holds such a dict | +| pipeline record id / entry id (system tags + input data hash) | same | +| source table hash → default `source_id` → per-row record ids (`_tag::source...`) | a source built without an explicit `source_id` holds such a dict in any row | +| stream / node `content_hash` that commit to the above | transitively | + +Hashes that do **not** move: every schema hash, every `pipeline_hash` (schema + topology only), `source_id` given explicitly, any value with no dict or with dicts already in key order, all semantic-hasher hashes (it already sorts dict keys). + +What a v0.2.0 store sees (pipeline path is unchanged because `pipeline_hash` is unchanged, so the same tables are read): +* Inputs whose dicts were key-sorted: same input hash → cache hit, same record id, served as before. +* Inputs with an unsorted dict: new input hash → result-cache miss → the pod recomputes once and a new row is written next to the old one. Old rows are never deleted or rewritten, so nothing looks dropped. A store-wide read (`get_all_records`, `FunctionJobNode` hot-load from the DB) shows **both** the old entry and the new one for such an input — duplicated, not dropped. This will be stated in the CHANGELOG with the remedy (drop the old rows, or start a fresh store). No automatic repair (standing rule). +* `EmptyData` replay of an old pipeline entry carries its stored (pre-fix) output hash to the downstream lookup; that is self-consistent with the downstream entries computed in the same old store, so nothing is mis-paired; it is replaced once the upstream entry is recomputed. +* Old stored rows with duplicate keys (only possible through an Arrow-native source) now raise on hashing, naming the column. Flagged for the coordinator: acceptable, or warn instead? + +Golden suite (`tests/test_hashing/golden`): +* **No existing entry changes.** The only dict values in the golden cases are `datagram/nested` (`{"a": 1}`, `{"b": 2, "c": 3}` — already key-sorted) and `semantic/dict.ab`/`dict.ba` (semantic hasher, unaffected). The arrow group has no key/value list tables. The unchanged suite passing after the fix is the proof that sorted dicts keep their hash. +* **Added entries only** (regenerate, check the diff only adds lines): `datagram/dict_order` (rows `{"b": 2, "a": 1}` and `{"a": 1, "b": 2}` → equal hashes), nested `dict[str, dict[str, int]]` reordered, `dict[int, str]` with keys `10, 9`, a tuple-key dict; `arrow/table` `kv.sorted` and `kv.unsorted` (must be equal), each pinning the canonical value. + +--- + +## File structure + +* Create `src/orcapod/semantic_types/dict_order.py` — `is_dict_arrow_type(arrow_type)`, `dict_entry_sort_key(storage_key)`, `sort_dict_entries(entries, column)` (sort + NaN / incomparable / duplicate checks). Single definition of the rule. +* Modify `src/orcapod/semantic_types/universal_converter.py` — `origin is dict` converter branch sorts via `sort_dict_entries`; the two `arrow → python` dict detections use `is_dict_arrow_type`. +* Modify `src/orcapod/hashing/visitors.py` — `SemanticHashingVisitor.visit_list`: after visiting elements, if `is_dict_arrow_type(list_type)`, sort the processed entries with `sort_dict_entries` (column path from `_current_field_path`). +* Modify `src/orcapod/core/datagrams/datagram.py` — dict-backed init canonicalises dict-containing fields into new objects (fix 3), via a helper `canonicalize_dict_values(value, python_type, converter)` in `dict_order.py`. +* Tests: `tests/test_hashing/test_dict_order_hash.py` (public interface), golden additions in `tests/test_hashing/golden/cases.py` (new case functions only; no existing fixture moved or renamed). +* Docs: `docs/reference/hashing.md` (dict canonical order, what is a dict, duplicate/NaN errors), `CHANGELOG.md` `[0.3.0] — unreleased` (hash change + migration note), `DESIGN_ISSUES.md` (one entry at the end). + +## Tasks (STEP 2, after approval) + +### Task 1: failing public-interface regression tests + +**Files:** Create `tests/test_hashing/test_dict_order_hash.py` + +- [ ] Write tests (all through public API: sources, `Data.content_hash()`, `as_dict()`, `FunctionPod`/`CachedFunctionPod.process(stream).iter_data()`, `async_process_data` for the async pod path, databases' `get_all_records`): + * `DictSource` inferred schema: `{"a":1,"b":2}` vs `{"b":2,"a":1}` → equal `content_hash`. + * `DictSource` with `data_schema={"m": dict[str, int]}` → equal. + * function pod `-> dict[str, int]` returning the two orders → equal output hashes; sync and async. + * nested `dict[str, dict[str, int]]`, `list[dict[str, int]]`, `dict[str, list[dict[str,int]]]` reordered at inner and outer level → equal. + * `ArrowTableSource` with `large_list>` in both orders → equal; and equal to the `DictSource` hash. + * key types: int (`{10:..., 9:...}`), bool, bytes, float incl. `-0.0`, date, aware datetime, tuple keys, Optional key with `None` → reorder gives equal hash. + * NaN key → `ValueError` naming the column; Arrow-native duplicate key → `ValueError` naming the column. + * `list[tuple[str,int]]` (ordered pairs) in two orders → hashes **differ** (non-dict lists are never sorted). + * cached pod hit: `CachedFunctionPod` over `InMemoryArrowDatabase` and `DeltaTableDatabase(tmp_path)`; process `{"b":2,"a":1}` then `{"a":1,"b":2}` → one stored record, second run is a hit. + * fresh vs cached values: the pod output dict read via `as_dict()` iterates in key order on the fresh run and on the cached run. + * a dict already in key order keeps the exact hash it had before (assert against the hash pinned by the existing golden `datagram/nested` entry by running the same rows). +- [ ] Run `uv run pytest tests/test_hashing/test_dict_order_hash.py -v`; expected: reorder/Arrow/cache/NaN/duplicate tests FAIL, the `list[tuple]` and already-sorted tests PASS. Paste output into the PR description. +- [ ] Commit `test(hashing): failing regression tests for dict insertion order (ITL-905)`. + +### Task 2: the ordering rule + +**Files:** Create `src/orcapod/semantic_types/dict_order.py`; unit tests in `tests/test_semantic_types/test_dict_order.py` (these are about the helper itself). + +```python +def is_dict_arrow_type(arrow_type: pa.DataType) -> bool: + """True for a list (any kind) whose element is struct{key, value}, exactly those fields.""" + +def dict_entry_sort_key(storage_key: Any) -> tuple: + # None first; struct (dict from to_pylist) → tuple of field keys in field order; + # float NaN → ValueError; otherwise (1, value). + +def sort_dict_entries(entries: list[dict], column: str) -> list[dict]: + # Returns a new list sorted by dict_entry_sort_key(entry["key"]); raises ValueError on + # NaN or duplicate keys and TypeError on incomparable keys, naming ``column``. +``` + +- [ ] Write the helper tests (ordering table from section 2, NaN, duplicates, incomparable), see them fail, implement, see them pass, commit. + +### Task 3: converter sorts on write (fix 1) + +**Files:** Modify `src/orcapod/semantic_types/universal_converter.py:1596-1606`, `1390-1401`, `1681-1702`. + +- [ ] In the `origin is dict` converter: build `[{"key": key_converter(k), "value": value_converter(v)} ...]`, then return `sort_dict_entries(entries, column=...)`. Replace the two `set(field_names) == {"key", "value"}` checks with `is_dict_arrow_type`. +- [ ] Run Task 1 tests (Python-origin cases now pass; Arrow-native still fail) and `uv run pytest tests/test_semantic_types -q`. Commit. + +### Task 4: hasher sorts dict entries (fix 2) + +**Files:** Modify `src/orcapod/hashing/visitors.py` (`SemanticHashingVisitor.visit_list`). + +- [ ] After `_visit_list_elements`, if `is_dict_arrow_type(list_type)`, return `sort_dict_entries(processed, column=".".join(self._current_field_path))`. Run Task 1 tests (Arrow-native, nested, store cases pass) and `uv run pytest tests/test_hashing -q` (golden must pass unchanged). Commit. + +### Task 5: canonical in-memory values (fix 3) + +**Files:** Modify `src/orcapod/core/datagrams/datagram.py` dict-backed init; add `canonicalize_dict_values` to `dict_order.py`. + +- [ ] Walk the Python type: `dict[K, V]` → new dict sorted by `dict_entry_sort_key(key_converter(k))` with values canonicalised; `list`/`tuple`/`Optional` → rebuilt only if a child changed; anything else returned as is. Run the fresh-vs-cached test. Commit. + +### Task 6: golden additions + +- [ ] Add the case functions listed in section 5 to `tests/test_hashing/golden/cases.py` (new functions only), regenerate with `uv run python tests/test_hashing/golden/generate_golden_hashes.py`, check `git diff tests/test_hashing/golden/data` only adds lines. Commit. + +### Task 7: docs, CHANGELOG, DESIGN_ISSUES + +- [ ] `docs/reference/hashing.md`: section "Dict values are order independent" (rule table, what is a dict, NaN/duplicate errors, read-back order, dataclass caveat). Runs under `tests/test_docs`. +- [ ] `CHANGELOG.md` `[0.3.0] — unreleased`: Fixed + "Hash changes" note with the migration text of section 5. +- [ ] `DESIGN_ISSUES.md`: one entry appended at the end, status resolved with **Fix:**. +- [ ] Full checks: `uv run pytest tests/`, `uv run --with ruff ruff check .`, `uv run --with ruff ruff format --check .`, `uv run --with basedpyright basedpyright` (exit code). Commit, merge origin/main, push, PR. + +## Open questions for the coordinator + +1. Approve canonical sort in Orcapod (converter + hasher + in-memory) over a starfix order-independent hash? +2. Approve fix 3 (in-memory canonical copy of dict values in plain containers; dataclass/pydantic internals left as given)? +3. NaN keys and duplicate keys raise (including when hashing a v0.2.0 stored row that has duplicates, only possible via Arrow-native input) — OK? +4. The `set[T]` per-process hash bug: separate issue (recommended), or fold into this PR (same mechanism: sort plain `set[T]` on write)? From 1c888a264fb963c6a0b9a47af5115ce9292a45c3 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:39:06 -0800 Subject: [PATCH 02/22] test(hashing): failing regression tests for dict insertion order (ITL-905) Public-interface tests: DictSource (inferred and declared schemas), key types, nested dicts, dataclass fields, Arrow-native key/value lists, function pod outputs sync and async, cached pod hits on in-memory and Delta stores, and a Delta result store written by the released v0.2.0 code. Co-Authored-By: Claude Opus 5.5 --- .../data/generate_v0_2_0_dict_store.py | 47 ++ .../data/v0_2_0_dict_store.tar.gz | Bin 0 -> 4870 bytes tests/test_hashing/test_dict_order_hash.py | 413 ++++++++++++++++++ 3 files changed, 460 insertions(+) create mode 100644 tests/test_hashing/data/generate_v0_2_0_dict_store.py create mode 100644 tests/test_hashing/data/v0_2_0_dict_store.tar.gz create mode 100644 tests/test_hashing/test_dict_order_hash.py diff --git a/tests/test_hashing/data/generate_v0_2_0_dict_store.py b/tests/test_hashing/data/generate_v0_2_0_dict_store.py new file mode 100644 index 000000000..d64fb77f3 --- /dev/null +++ b/tests/test_hashing/data/generate_v0_2_0_dict_store.py @@ -0,0 +1,47 @@ +"""Build a Delta result store with the released v0.2.0 code (ITL-905 test fixture). + +Run from a checkout of the v0.2.0 tag, then pack the store: + + uv run python generate_v0_2_0_dict_store.py + tar -czf v0_2_0_dict_store.tar.gz -C . + +The process can hang at interpreter exit inside deltalake after printing; the store is +complete once the record count is printed. +""" + +import sys +import orcapod +from orcapod.core.cached_function_pod import CachedFunctionPod +from orcapod.core.function_pod import FunctionPod +from orcapod.core.data_function import PythonDataFunction +from orcapod.databases import DeltaTableDatabase +from orcapod.sources import DictSource + + +def total(m: dict[str, int]) -> int: + return sum(m.values()) + + +out = sys.argv[1] +db = DeltaTableDatabase(out) +pod = CachedFunctionPod( + FunctionPod(PythonDataFunction(total, output_keys="total")), result_database=db +) +src = DictSource( + data=[ + {"k": "unsorted", "m": {"b": 2, "a": 1}}, + {"k": "sorted", "m": {"a": 1, "c": 3}}, + ], + tag_columns=["k"], + data_schema={"k": str, "m": dict[str, int]}, + source_id="itl905_v020", +) +res = list(pod.process(src).iter_data()) +db.flush() +for t, d in res: + print(t.as_dict(), d.as_dict(), d.content_hash().to_string()) +for t, d in src.iter_data(): + print("input", t.as_dict(), d.content_hash().to_string()) +print(pod.record_path) +print(db.get_all_records(pod.record_path).num_rows) +print("version", orcapod.__version__ if hasattr(orcapod, "__version__") else "?") diff --git a/tests/test_hashing/data/v0_2_0_dict_store.tar.gz b/tests/test_hashing/data/v0_2_0_dict_store.tar.gz new file mode 100644 index 0000000000000000000000000000000000000000..5a59fd95160a4a5cadf5a3f49cb540a4f1d805f5 GIT binary patch literal 4870 zcmV+h6Zz~PiwFQfMf7R_1MOUWY#Zl!rx@8*65EkH9Zd(at1va=EFyV#4k%BD1yMrQZF!q2J?b6=sd3|hW=3$8CGPO;XjP6 zD7vjMnvVt9kZoyE1a07Xd*Ao&$UENg%b{#*<3T3x-sgRv=Xsy!d7t;=cc#;2+V%^_ zaUVquqi?SleS=H+mL3R-^ijBn^pMmrPLL$-HVsqTubRnJDvFE%RN*(uOlFN2>eVZ! zcs|pBkG508omX^wE~iV(i%e!)qo95NyWOO>>;KzvDgH0=Stci@+1NVnBt|J5_xq^; zMfv@N*YBZl9%l&F&%4<~!V_n4j`IdsmLNQCDjx7s9yZ_&B=|T%dKvH(@My!SZU2*A z(nECpe><*i^8Y%n3V~bjKjrO?|J!l_M-?{2R6gfSlyaeE<|86k<<2&5{ z+cU5_-hE^L3tT+5PIM{^+V{WPO;KI{-*NvJ7(sNv?hpwgbLkYN7DJKkG^ z>;Jj`7w8ay^P(!d-4^WIhb>%xCwCafPD907Iu zvEA72A+PCJe{CJBu4G^Uu<9E!gHLmh$qWQ>SV|E@{@4l189~UGCF208!|$JZKJc?| zJ@L-k+mHAC=Al1ddUxdW55Mr87yjt={Ns;%GNOSR&h|K1+pMqe!U4>pbm89NU7#E} z+CZ~ih+vhzS}=t+RK$l5y8rE)A6|Utoy!-#^XN}XSAO&#)8AjT|2?s1`C2M`_#cf- z(H;kDoAul7!r8^XU!I@&{k0@&4G|BbbudeH}+ewg4f->r}OtLCBw_8`BkfLkfP_LY|3>58CXx7S| zUR7P1D~jAG3LK74FeStM(qCK0sw>e8x@5-w+N_BBR?o*Ty`bcY(auf?K#x)V1U@>7 z6Mzjva$KDC`6z;5m_!0FW!<#u*A%mKE(WPyH^I(v>TeIj}Bm`EhcpP z%AiK=n+LI%ccI7EP^aELEY$=(y>W1QuQb?;YzkNv8!};gkD9Ib$6^A{<^?X6=A@E= zSS+0@l*Aavh)gWS6jM^UB^E0rV(W|$Q_^KD$7K1Vrr9AV0#+H;S1G2GIYulAyo%Ei zeKj_p6ZxE|sx(lkluS$IYDotXl|RF?=%Xa&g_yWeP_&_pYov3DJjlemXMYdY>9zJ| z(nS!wu=*Z8!)Jj8Sg$1Do;7|$EpXR5lPPIBYpn{9rjgh>FBCy%syYm+br1!na|-mr z5;R66>R!o$XT(@dvwg8*z9g`G3{_S)?*q!^xtLfyIyn2yfuRvJ_(6^sI&`DoZW_Y& z9->UU`wwX%1T$OAmMcA$f9SC<*!x=F2+mrD(Lgg<&W&JTj2yudCbbNG%pplP&6a2H zv0pda|GDS(?Dl3YtFoS#MzAB6^Pj`6_h=NYGYF8jU+b|v-(x?w%l?aAqaEO^!yQ8Ws+a-wtu|q_U!tqCixJW~ulno@Cd<#Ej1P}u zp7v!(*{CAr;sf?)AE+bcH&Dt2yH-m5?5T3Y%l-EEAE>txt-9ewClGMWu64t-VeYMO zr&k8-&kWS*!DT4m+#@x1YA&G0NH+%T*XxP63Prr7i0GA#q%5ofgL6HWmwW6V9kBoK zz>w8oziX+t3!V+gqt}U9ET2AtO={ernKx9~{gpxcPk*QN9dI?j-*kfE_U5sg{H~pa zkev7TSU%okzj@I9I@aBJw!8D31q(dc;1^uju;sN!F|guOhfZa7S4Gd;hwbMN4_OWN zpO$*NQYCtR=mODm^)V1VmP>~%W_zFIjU$6_32NO-he9XAqtdFB;4# z#j?aTLIlcjGL+z^5qo-NVbkS0gSt8=Hwp$9%N5Q`kY2`Nae^fS1e=}DMBJfuZfdd! z>1$2eCsF${YQI7TO5D`4hnoxt(TJpe8MRNUl@8LWaz%O2|5=F?tnidQGYLPqsQ|tb zhV{oit_7ECO5!S4oR~W;eOIjNL-I}s^!6l@71NC`R_aeIxLvNzO;~?PTYpLEFK{KI zOZsFdi(D914X(KoF4v@1pUFylI=BSsh%_Z;_(c#J;EIONWa8oF5(i8VpP|2^nLvi0 zT0R{if$gS`ry^vg6wUe(mm<`!f&KG{j_j9+xPw{9BO0Dhu+tfwnF^F5?r4V1%@^V% zidkSErGAy)_)M8r9Dw5M4zJXwPdx_emtp<-_MnrH9)w+) zj;7+%%NbZ0Uhp;S-;wgrPpV%st=9myhKu& z1MIjFoeUJBH2npW^^gn=JA~q}yx<$AC@1d3!LAc*P22i2{j6UCF-_g;R<2g@f09kG zZpIyO_~W?G;b8)9N5BW}eDOHv^ZDa!g7Due{!db#TKu1c?|*jj|2vHT|C#uU^Y8Jm zW=ntbP+k0ARmZ9;X&e84SF#1&04vukNK-6 zQG=q!n18b=t?dR~%>Vxy=C7nk)h%M%CfpwS${b5z>4{D(P z%U!6yRj9u=E%MnrhV!=}z4xd|1@fEjG|1m(x+ zDzLBL$Xe=dZ!K>^NdMhOurD@&^ncW2c?;yMLHm{VAblX>r%?TGAH$w*4bK1Ip#Awf z3g_?4O{EIX|CJLPw%k0{3Y>rWu>Bbb=j*qYmU0q*?g9fZ%)&gpOShMVc*3&49!+2>wExs&GENeQ^Gg0UST6 zg3lJ4!1=fyexudx*(PxQ1ca@nPAFLnM%CZbi!)tb2|N22l1qYu%bksEsJnarr+*E2ko(YiA$ni}G;nI?QD)p=U z!V*8!74!~7683i-!PR9=|HcuJE17PdR_fCe5Wb%?fWOuDm_cwodOd<%37;V#jF3*31;Z|&Bpr9A$myV4+-ujGkAndrC1W#%LtA?rHVHM$4@8e`fxlszaGzpGT>hY z`d%`C&nL?2@ek-5G+p!hL6!?sr@5&Nj1>?kTxlf|0=Z$i3aLZ;==D|GZvxR9>Pw~K z@YR^PP5tWvp}DI4W)WSnUm>0g5+KK{0i9B6`@!S@vcyS{n&lLel(I%pn;$05RBalR zVSXCDR)qL(fOuJFv%rSoMY_`v|Ftf}zin4T{6EI=8L&;yWrECbDR2El3!;w{6HZ4m6&w#Ew@&U!GHu+>{ z)ixs6FgTh(73C9%YDL{KAgYG^D^*hMwh^gzFO6!+^+!D#`I@66)EJ!+DZ2(kU_&Y`5qi*jgPB}dukXWgxR9U?ruPS(Vy|=z<+k)<5&S>0qE9Ps%FfIFfXIQYE z@M$4-8;=#}u3RaT8OxV)qH1&*3di+h0oQs9S2+?lFge~xZM?BF@jAW1x3)k^5}72( z92Fk|r*=$-|mY6DJHUnk}UP$MX3qpQ9&GCXn+1k4Atc^u2 zW^#!R*io5`Pw*{MJjoKgrFC?pCKq;6Olf39u zfkMMy7?#|nE(8%`6L^NJm<)&ZO7pp@5p8H00rmpU!iIwYy8u$%^*LS?(k$w?6i?9O z(2TJxC?v=cnC@fwk_cfVAzy}tQkfh(nlef2SR$R_i!m^@@tiS9;IsL4NW=|@MV86s z>a{Gcr3(d~i*Y3w5U6e~7pM^!2Zgj=03rc<0ACx^ByZ`3BVwrl=0&-oz?X%z$X87T zECrRR7J$ItYBv9WhV>ubq4l2+_xN<{KR4c8|J|PTUqQ%=c{ZPc0}!mELSQWOLS^iN zAOwYB7-`i)5+y#6ub6SLOoI(U9vHPH{5WiKA)eb0LjM@ zFmo|*y{9DJA}gc|qAJ-Y@&ZUwXh~X0wzlfhGLI^wB_&)_qBH6zMK~0twP9r*EP`bt zD=i%oX+DG2iZDA?=0H^vlNK0C%Xm7+2pgyjs>H-IupF&IS5dFw;&fHgK+CWO+wK%w z>}(Tr2r{aft0S5b1V*xq{GW?Y>FYF& zdbBL-gz^agyk%BT!G$G^0tQTY^PsX^aqHG?Si2zXw1*T7xa*!a)TnMWR2vk%5(y?# zY~~8h#-e$@T4AZ(eRL>bW;FHjPB=7ks=@IN$d2%Uxt=UfrvE?Xc^A0o-Kp1ORRT0At0gxc~qF literal 0 HcmV?d00001 diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py new file mode 100644 index 000000000..dd411be56 --- /dev/null +++ b/tests/test_hashing/test_dict_order_hash.py @@ -0,0 +1,413 @@ +"""A dict's identity does not depend on its key order (ITL-905). + +A ``dict[K, V]`` column is stored as ``large_list>``. Equal dicts must +give equal row hashes whatever order their entries were inserted or stored in, on every +path: Python values through the converter, Arrow tables that already hold key/value lists, +nested dicts, function pod outputs (sync and async), and cached results read back from a +store. Ordered layouts (``list[tuple[K, V]]``) keep order as identity. +""" + +from __future__ import annotations + +import tarfile +from dataclasses import dataclass +from datetime import date, datetime, timezone +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod.core.cached_function_pod import CachedFunctionPod +from orcapod.core.data_function import PythonDataFunction +from orcapod.core.function_pod import FunctionPod +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.sources import ArrowTableSource, DictSource + +V020_STORE = Path(__file__).parent / "data" / "v0_2_0_dict_store.tar.gz" + +KV_TYPE = pa.large_list( + pa.struct([pa.field("key", pa.large_string()), pa.field("value", pa.int64())]) +) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _source(value: Any, value_type: Any = None) -> DictSource: + schema = None if value_type is None else {"k": str, "m": value_type} + return DictSource( + data=[{"k": "r", "m": value}], tag_columns=["k"], data_schema=schema + ) + + +def _data(value: Any, value_type: Any = None) -> Any: + _, data = next(iter(_source(value, value_type).iter_data())) + return data + + +def _row_hash(value: Any, value_type: Any = None) -> str: + return _data(value, value_type).content_hash().to_string() + + +def _arrow_source(entries: list[dict[str, Any]]) -> ArrowTableSource: + table = pa.table( + { + "k": pa.array(["r"], pa.large_string()), + "m": pa.array([entries], KV_TYPE), + } + ) + return ArrowTableSource(table, tag_columns=["k"], source_id="itl905") + + +def _arrow_row_hash(entries: list[dict[str, Any]]) -> str: + _, data = next(iter(_arrow_source(entries).iter_data())) + return data.content_hash().to_string() + + +def total(m: dict[str, int]) -> int: + return sum(m.values()) + + +def echo(m: dict[str, int]) -> dict[str, int]: + return m + + +def reorder(m: dict[str, int]) -> dict[str, int]: + """Return an equal dict in reverse insertion order.""" + return dict(reversed(list(m.items()))) + + +def _pod(fn: Any, output_key: str) -> FunctionPod: + return FunctionPod(PythonDataFunction(fn, output_keys=output_key)) + + +@dataclass +class Holder: + """A dataclass with a dict field: dicts reach Arrow through the logical type.""" + + name: str + counts: dict[str, int] + + +# --------------------------------------------------------------------------- +# Python values through the converter +# --------------------------------------------------------------------------- + + +class TestPythonDictValues: + def test_inferred_schema_reordered_dicts_hash_equal(self): + assert _row_hash({"a": 1, "b": 2}) == _row_hash({"b": 2, "a": 1}) + + def test_declared_schema_reordered_dicts_hash_equal(self): + t = dict[str, int] + assert _row_hash({"a": 1, "b": 2}, t) == _row_hash({"b": 2, "a": 1}, t) + + def test_stored_entries_are_in_key_order(self): + table = _source({"b": 2, "a": 1}).as_table() + assert [e["key"] for e in table.column("m").to_pylist()[0]] == ["a", "b"] + + def test_str_keys_order_by_code_point_not_locale(self): + table = _source({"é": 1, "z": 2, "A": 3}).as_table() + assert [e["key"] for e in table.column("m").to_pylist()[0]] == ["A", "z", "é"] + + def test_different_values_still_hash_differently(self): + assert _row_hash({"a": 1, "b": 2}) != _row_hash({"a": 2, "b": 1}) + + @pytest.mark.parametrize( + ("value_type", "first", "second"), + [ + (dict[int, str], {10: "a", 9: "b", -1: "c"}, {-1: "c", 9: "b", 10: "a"}), + (dict[bool, int], {True: 1, False: 0}, {False: 0, True: 1}), + (dict[bytes, int], {b"\xff": 1, b"\x00": 2}, {b"\x00": 2, b"\xff": 1}), + (dict[float, int], {1.5: 1, -0.0: 2, -3.0: 3}, {-3.0: 3, -0.0: 2, 1.5: 1}), + ( + dict[date, int], + {date(2026, 1, 2): 1, date(2025, 5, 1): 2}, + {date(2025, 5, 1): 2, date(2026, 1, 2): 1}, + ), + ( + dict[datetime, int], + { + datetime(2026, 1, 2, tzinfo=timezone.utc): 1, + datetime(2025, 5, 1, tzinfo=timezone.utc): 2, + }, + { + datetime(2025, 5, 1, tzinfo=timezone.utc): 2, + datetime(2026, 1, 2, tzinfo=timezone.utc): 1, + }, + ), + ], + ) + def test_key_types_reordered_hash_equal(self, value_type, first, second): + assert _row_hash(first, value_type) == _row_hash(second, value_type) + + def test_nan_key_is_rejected_naming_the_column(self): + with pytest.raises(ValueError, match="'m'.*NaN|NaN.*'m'"): + _row_hash({float("nan"): 1, 1.0: 2}, dict[float, int]) + + +class TestNestedDicts: + def test_dict_of_dicts_inner_and_outer_reordered(self): + t = dict[str, dict[str, int]] + first = {"x": {"a": 1, "b": 2}, "y": {"c": 3}} + second = {"y": {"c": 3}, "x": {"b": 2, "a": 1}} + assert _row_hash(first, t) == _row_hash(second, t) + + def test_list_of_dicts_reordered_inside(self): + t = list[dict[str, int]] + assert _row_hash([{"a": 1, "b": 2}, {"c": 3}], t) == _row_hash( + [{"b": 2, "a": 1}, {"c": 3}], t + ) + + def test_list_of_dicts_keeps_list_order(self): + t = list[dict[str, int]] + assert _row_hash([{"a": 1}, {"c": 3}], t) != _row_hash([{"c": 3}, {"a": 1}], t) + + def test_dict_of_list_of_dicts_reordered(self): + t = dict[str, list[dict[str, int]]] + first = {"x": [{"a": 1, "b": 2}], "w": []} + second = {"w": [], "x": [{"b": 2, "a": 1}]} + assert _row_hash(first, t) == _row_hash(second, t) + + def test_dict_inside_dataclass_field_reordered(self): + first = Holder("h", {"a": 1, "b": 2}) + second = Holder("h", {"b": 2, "a": 1}) + assert _row_hash(first, Holder) == _row_hash(second, Holder) + + +# --------------------------------------------------------------------------- +# Arrow tables that already hold key/value lists +# --------------------------------------------------------------------------- + + +class TestArrowNativeKeyValueLists: + def test_unsorted_and_sorted_entries_hash_equal(self): + sorted_entries = [{"key": "a", "value": 1}, {"key": "b", "value": 2}] + unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] + assert _arrow_row_hash(sorted_entries) == _arrow_row_hash(unsorted_entries) + + def test_arrow_native_matches_python_dict_hash(self): + unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] + assert _arrow_row_hash(unsorted_entries) == _row_hash( + {"a": 1, "b": 2}, dict[str, int] + ) + + def test_user_table_is_not_rewritten(self): + unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] + source = _arrow_source(unsorted_entries) + _, data = next(iter(source.iter_data())) + data.content_hash() + assert source.as_table().column("m").to_pylist()[0] == unsorted_entries + + def test_duplicate_keys_are_rejected_naming_the_column(self): + entries = [{"key": "a", "value": 1}, {"key": "a", "value": 2}] + with pytest.raises(Exception, match="(?s)'m'.*duplicate key"): + _arrow_row_hash(entries) + + def test_struct_keys_sort_field_by_field(self): + key_type = pa.struct( + [pa.field("f0", pa.int64()), pa.field("f1", pa.large_string())] + ) + kv_type = pa.large_list( + pa.struct([pa.field("key", key_type), pa.field("value", pa.int64())]) + ) + ordered = [ + {"key": {"f0": 0, "f1": "z"}, "value": 3}, + {"key": {"f0": 1, "f1": "a"}, "value": 2}, + {"key": {"f0": 1, "f1": "b"}, "value": 1}, + ] + + def kv_hash(entries: list[dict[str, Any]]) -> str: + table = pa.table( + { + "k": pa.array(["r"], pa.large_string()), + "m": pa.array([entries], kv_type), + } + ) + src = ArrowTableSource(table, tag_columns=["k"], source_id="struct_keys") + _, data = next(iter(src.iter_data())) + return data.content_hash().to_string() + + assert kv_hash(ordered) == kv_hash(ordered[::-1]) + assert kv_hash(ordered) == kv_hash([ordered[1], ordered[2], ordered[0]]) + + def test_ordered_struct_pairs_are_not_sorted(self): + pair_type = pa.large_list( + pa.struct([pa.field("f0", pa.large_string()), pa.field("f1", pa.int64())]) + ) + + def pair_hash(entries: list[dict[str, Any]]) -> str: + table = pa.table( + { + "k": pa.array(["r"], pa.large_string()), + "m": pa.array([entries], pair_type), + } + ) + src = ArrowTableSource(table, tag_columns=["k"], source_id="pairs") + _, data = next(iter(src.iter_data())) + return data.content_hash().to_string() + + assert pair_hash([{"f0": "a", "f1": 1}, {"f0": "b", "f1": 2}]) != pair_hash( + [{"f0": "b", "f1": 2}, {"f0": "a", "f1": 1}] + ) + + +# --------------------------------------------------------------------------- +# Function pods: outputs, async, caches +# --------------------------------------------------------------------------- + + +class TestFunctionPodDictOutputs: + def test_pod_output_reordered_dicts_hash_equal(self): + source = _source({"a": 1, "b": 2}, dict[str, int]) + [(_, plain)] = list(_pod(echo, "out").process(source).iter_data()) + [(_, flipped)] = list(_pod(reorder, "out").process(source).iter_data()) + assert plain.content_hash() == flipped.content_hash() + + def test_fresh_pod_output_iterates_in_key_order(self): + source = _source({"b": 2, "a": 1}, dict[str, int]) + [(_, out)] = list(_pod(echo, "out").process(source).iter_data()) + assert list(out.as_dict()["out"]) == ["a", "b"] + + def test_pod_does_not_mutate_the_returned_dict(self): + returned: list[dict[str, int]] = [] + + def keep(m: dict[str, int]) -> dict[str, int]: + result = {"b": m["b"], "a": m["a"]} + returned.append(result) + return result + + source = _source({"a": 1, "b": 2}, dict[str, int]) + list(_pod(keep, "out").process(source).iter_data()) + assert list(returned[0]) == ["b", "a"] + + @pytest.mark.asyncio + async def test_async_output_matches_sync_and_is_order_independent(self): + source = _source({"a": 1, "b": 2}, dict[str, int]) + tag, data = next(iter(source.iter_data())) + _, sync_out = _pod(echo, "out").process_data(tag, data) + _, async_plain = await _pod(echo, "out").async_process_data(tag, data) + _, async_flipped = await _pod(reorder, "out").async_process_data(tag, data) + assert sync_out is not None + assert async_plain is not None + assert async_flipped is not None + assert sync_out.content_hash() == async_plain.content_hash() + assert async_plain.content_hash() == async_flipped.content_hash() + assert list(async_flipped.as_dict()["out"]) == ["a", "b"] + + +def _delta_db(tmp_path: Path) -> DeltaTableDatabase: + return DeltaTableDatabase(tmp_path / "store") + + +@pytest.fixture(params=["in_memory", "delta"]) +def result_db(request, tmp_path): + if request.param == "in_memory": + return InMemoryArrowDatabase() + return _delta_db(tmp_path) + + +class TestCachedPodHit: + def test_reordered_equal_dict_is_a_cache_hit(self, result_db): + pod = CachedFunctionPod(_pod(total, "total"), result_database=result_db) + first = _source({"b": 2, "a": 1}, dict[str, int]) + second = _source({"a": 1, "b": 2}, dict[str, int]) + + [(_, out1)] = list(pod.process(first).iter_data()) + result_db.flush() + [(_, out2)] = list(pod.process(second).iter_data()) + + assert out1.get_meta_value(pod.RESULT_COMPUTED_FLAG) is True + assert out2.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False + records = result_db.get_all_records(pod.record_path) + assert records is not None + assert records.num_rows == 1 + + @pytest.mark.asyncio + async def test_reordered_equal_dict_is_an_async_cache_hit(self, result_db): + pod = CachedFunctionPod(_pod(total, "total"), result_database=result_db) + tag1, data1 = next(iter(_source({"b": 2, "a": 1}, dict[str, int]).iter_data())) + tag2, data2 = next(iter(_source({"a": 1, "b": 2}, dict[str, int]).iter_data())) + + _, out1 = await pod.async_process_data(tag1, data1) + result_db.flush() + _, out2 = await pod.async_process_data(tag2, data2) + + assert out1 is not None + assert out2 is not None + assert out1.get_meta_value(pod.RESULT_COMPUTED_FLAG) is True + assert out2.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False + + def test_fresh_and_cached_dict_outputs_iterate_identically(self, result_db): + pod = CachedFunctionPod(_pod(reorder, "out"), result_database=result_db) + source = _source({"a": 1, "b": 2, "c": 3}, dict[str, int]) + + [(_, fresh)] = list(pod.process(source).iter_data()) + result_db.flush() + [(_, cached)] = list(pod.process(source).iter_data()) + + assert cached.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False + assert list(fresh.as_dict()["out"]) == list(cached.as_dict()["out"]) + assert list(cached.as_dict()["out"]) == ["a", "b", "c"] + assert fresh.content_hash() == cached.content_hash() + + +class TestV020Store: + """A Delta result store written by the released v0.2.0 code. + + ``data/v0_2_0_dict_store.tar.gz`` was produced by v0.2.0 running ``total`` over two + inputs: ``k=unsorted`` holding ``{"b": 2, "a": 1}`` (stored in that order, so its + v0.2.0 input hash is order dependent) and ``k=sorted`` holding ``{"a": 1, "c": 3}`` + (already in key order, so its hash does not move). ``data/generate_v0_2_0_dict_store.py`` + regenerates it from a checkout of the ``v0.2.0`` tag. + """ + + @pytest.fixture + def store(self, tmp_path): + # The store is shipped as a tarball: its directory names contain ":". + target = tmp_path / "v020_store" + with tarfile.open(V020_STORE) as archive: + archive.extractall(target, filter="data") + return DeltaTableDatabase(target) + + def _inputs(self) -> DictSource: + return DictSource( + data=[ + {"k": "unsorted", "m": {"b": 2, "a": 1}}, + {"k": "sorted", "m": {"a": 1, "c": 3}}, + ], + tag_columns=["k"], + data_schema={"k": str, "m": dict[str, int]}, + source_id="itl905_v020", + ) + + def test_unsorted_row_recomputes_once_and_old_rows_stay(self, store): + pod = CachedFunctionPod(_pod(total, "total"), result_database=store) + before = store.get_all_records(pod.record_path) + assert before is not None + assert before.num_rows == 2 + + results = { + tag.as_dict()["k"]: out + for tag, out in pod.process(self._inputs()).iter_data() + } + store.flush() + + assert results["sorted"].get_meta_value(pod.RESULT_COMPUTED_FLAG) is False + assert results["unsorted"].get_meta_value(pod.RESULT_COMPUTED_FLAG) is True + assert results["unsorted"].as_dict()["total"] == 3 + after = store.get_all_records(pod.record_path) + assert after is not None + assert after.num_rows == 3 # both old rows kept, one new row + + again = { + tag.as_dict()["k"]: out + for tag, out in pod.process(self._inputs()).iter_data() + } + assert all( + out.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False + for out in again.values() + ) From 502e0fa224b0dcd6cf7a0fc76aac8b12ef7fc116 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:39:06 -0800 Subject: [PATCH 03/22] fix(semantic_types): store dict entries in canonical key order (ITL-905) A dict is order independent, so the converter now sorts the entries of a dict value by storage key when it builds large_list>. The order has one definition (semantic_types/dict_order.py); NaN keys, incomparable keys and duplicate keys raise InvalidDictKeysError, which python_dicts_to_struct_dicts wraps with the column name. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/errors.py | 9 + src/orcapod/semantic_types/dict_order.py | 174 ++++++++++++++++++ .../semantic_types/universal_converter.py | 71 +++---- 3 files changed, 220 insertions(+), 34 deletions(-) create mode 100644 src/orcapod/semantic_types/dict_order.py diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 81c653627..e016ffcb5 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -26,6 +26,15 @@ class DuplicateTagError(ValueError): pass +class InvalidDictKeysError(ValueError): + """Raised when a dict value's keys have no canonical order. + + A dict is stored with its entries in canonical key order (ITL-905), so its keys must + be totally ordered and unique. Raised for a NaN key, keys that cannot be compared, + two keys with the same stored value, or a null entry in a stored key/value list. + """ + + class DataFunctionUnavailableError(RuntimeError): """Raised when a data function proxy is invoked without a bound function. diff --git a/src/orcapod/semantic_types/dict_order.py b/src/orcapod/semantic_types/dict_order.py new file mode 100644 index 000000000..37eece72a --- /dev/null +++ b/src/orcapod/semantic_types/dict_order.py @@ -0,0 +1,174 @@ +"""Canonical entry order for dict values. + +A ``dict[K, V]`` value is stored as ``large_list>``. A dict is order +independent, so two equal dicts must store and hash identically whatever order their +entries were inserted in (ITL-905). This module holds the one definition of that order. +It is applied when a Python dict is converted to Arrow, when a datagram is built from +Python values, and by the Arrow hasher to key/value lists that arrive already stored. + +The order is defined on storage key values (what the converter produces and what +``to_pylist`` reads back), so the converter and the hasher cannot disagree: + +* ``None`` sorts first. +* ``bool``, ``int``, ``float``, ``Decimal``: numeric. A NaN key has no order and is + rejected. ``-0.0`` and ``0.0`` are one key. +* ``str``: Unicode code point (never locale). ``bytes``: unsigned bytewise. +* ``date``, ``datetime`` (by instant), ``time``, ``timedelta``: natural order. +* struct keys (a ``dict`` of field values): field by field, in field order. + +Keys that cannot be ordered against each other, duplicate keys and null entries raise +``InvalidDictKeysError``. +""" + +from __future__ import annotations + +import math +import types +import typing +from collections.abc import Callable, Mapping +from typing import TYPE_CHECKING, Any, get_args, get_origin + +from orcapod.errors import InvalidDictKeysError +from orcapod.utils.lazy_module import LazyModule + +if TYPE_CHECKING: + import pyarrow as pa +else: + pa = LazyModule("pyarrow") + + +def is_dict_arrow_type(arrow_type: pa.DataType) -> bool: + """Return whether ``arrow_type`` is Orcapod's encoding of a ``dict``. + + A dict is a list (regular, large or fixed-size) whose element is a struct with + exactly two fields named ``key`` and ``value``. Any other list of structs is an + ordered list, whose order is part of its identity. + + Args: + arrow_type: The Arrow type to test. + + Returns: + True for a dict-encoded list type. + """ + if not ( + pa.types.is_list(arrow_type) + or pa.types.is_large_list(arrow_type) + or pa.types.is_fixed_size_list(arrow_type) + ): + return False + element = arrow_type.value_type + return ( + pa.types.is_struct(element) + and element.num_fields == 2 + and {element.field(0).name, element.field(1).name} == {"key", "value"} + ) + + +def _key_rank(key: Any) -> tuple[Any, ...]: + """Return the sort key of one storage key value. + + Raises: + InvalidDictKeysError: If the key is a NaN float. + """ + if key is None: + return (0,) + if isinstance(key, float) and math.isnan(key): + raise InvalidDictKeysError("a NaN dict key has no canonical order") + if isinstance(key, Mapping): + return (1, tuple(_key_rank(v) for v in key.values())) + return (1, key) + + +def _sorted_by_rank(items: list[tuple[tuple[Any, ...], Any]]) -> list[Any]: + """Sort ``(rank, payload)`` pairs by rank and return the payloads. + + Raises: + InvalidDictKeysError: If two ranks cannot be compared or are equal. + """ + try: + items = sorted(items, key=lambda item: item[0]) + except TypeError as exc: + raise InvalidDictKeysError(f"dict keys cannot be ordered: {exc}") from exc + for (previous, _), (current, payload) in zip(items, items[1:]): + if previous == current: + raise InvalidDictKeysError(f"duplicate key in dict value: {payload!r}") + return [payload for _, payload in items] + + +def sort_dict_entries(entries: list[Any]) -> list[Any]: + """Return storage entries ``{"key": ..., "value": ...}`` in canonical key order. + + Args: + entries: The entries of one dict value, as the converter produces them or as + ``to_pylist`` reads them back. + + Returns: + A new list holding the same entries, sorted by key. + + Raises: + InvalidDictKeysError: For a null entry, a NaN key, keys that cannot be ordered + against each other, or a duplicate key. + """ + ranked = [] + for entry in entries: + if entry is None: + raise InvalidDictKeysError("a dict value holds a null entry") + ranked.append((_key_rank(entry["key"]), entry)) + return _sorted_by_rank(ranked) + + +def canonical_python_value( + value: Any, + python_type: Any, + key_converter: Callable[[Any], Callable[[Any], Any]], +) -> Any: + """Return ``value`` with every dict in it in canonical key order. + + Walks ``python_type`` through ``dict``, ``list`` and ``Optional``. A dict whose order + changes, or whose values change, is returned as a new dict; everything else is + returned as given. The input is never mutated. Values of other types (including + dataclass and pydantic objects) are returned unchanged. + + Args: + value: A Python value of type ``python_type``. + python_type: The value's declared Python type. + key_converter: Returns the Python-to-storage converter for a key type, so keys + are ordered by their storage value, as the converter stores them. + + Returns: + ``value`` itself when nothing changes, otherwise a canonically ordered copy. + + Raises: + InvalidDictKeysError: For a key that has no canonical order (NaN), keys that + cannot be ordered, or two keys that convert to the same storage key. + """ + if value is None: + return None + origin = get_origin(python_type) + args = get_args(python_type) + if origin is typing.Union or origin is types.UnionType: + non_none = [a for a in args if a is not type(None)] + if len(non_none) != 1: + return value + return canonical_python_value(value, non_none[0], key_converter) + if origin is dict and len(args) == 2 and isinstance(value, dict): + to_storage = key_converter(args[0]) + items = [ + ( + _key_rank(to_storage(k)), + (k, canonical_python_value(v, args[1], key_converter)), + ) + for k, v in value.items() + ] + ordered = _sorted_by_rank(items) + unchanged = all( + new_k is old_k and new_v is old_v + for (new_k, new_v), (old_k, old_v) in zip(ordered, value.items()) + ) + return value if unchanged else dict(ordered) + if origin is list and len(args) == 1 and isinstance(value, list): + new_items = [canonical_python_value(v, args[0], key_converter) for v in value] + if all(new is old for new, old in zip(new_items, value)): + return value + return new_items + return value diff --git a/src/orcapod/semantic_types/universal_converter.py b/src/orcapod/semantic_types/universal_converter.py index b68a6f39c..c3b3f4113 100644 --- a/src/orcapod/semantic_types/universal_converter.py +++ b/src/orcapod/semantic_types/universal_converter.py @@ -23,6 +23,8 @@ from typing import TYPE_CHECKING, Any, TypedDict, get_args, get_origin from orcapod.contexts import DataContext, resolve_context +from orcapod.errors import InvalidDictKeysError +from orcapod.semantic_types.dict_order import is_dict_arrow_type, sort_dict_entries from orcapod.semantic_types.type_inference import infer_python_schema_from_pylist_data from orcapod.types import DataType, Schema, SchemaLike from orcapod.utils import arrow_utils @@ -1006,7 +1008,12 @@ def python_dicts_to_struct_dicts( converted_record = {} for field_name, converter in converters.items(): if field_name in record: - converted_record[field_name] = converter(record[field_name]) + try: + converted_record[field_name] = converter(record[field_name]) + except InvalidDictKeysError as exc: + raise InvalidDictKeysError( + f"Column {field_name!r}: {exc}" + ) from exc else: converted_record[field_name] = None converted_data.append(converted_record) @@ -1388,17 +1395,14 @@ def _convert_arrow_to_python(self, arrow_type: pa.DataType) -> type | Any: element_type = arrow_type.value_type # Check if this is a dict representation: list> - if pa.types.is_struct(element_type): - field_names = [field.name for field in element_type] - if set(field_names) == {"key", "value"}: - # This is a dict - key_field = next(f for f in element_type if f.name == "key") - value_field = next(f for f in element_type if f.name == "value") + if is_dict_arrow_type(arrow_type): + key_field = element_type.field("key") + value_field = element_type.field("value") - key_python_type = self.arrow_type_to_python_type(key_field.type) - value_python_type = self.arrow_type_to_python_type(value_field.type) + key_python_type = self.arrow_type_to_python_type(key_field.type) + value_python_type = self.arrow_type_to_python_type(value_field.type) - return dict[key_python_type, value_python_type] + return dict[key_python_type, value_python_type] # Regular list element_python_type = self.arrow_type_to_python_type(element_type) @@ -1596,11 +1600,15 @@ def _convert_datetime(dt: datetime) -> datetime: elif origin is dict: key_converter = self.get_python_to_arrow_converter(args[0]) value_converter = self.get_python_to_arrow_converter(args[1]) + # A dict is order independent: entries are stored in canonical key order + # so equal dicts store and hash identically (ITL-905). return lambda value: ( - [ - {"key": key_converter(k), "value": value_converter(v)} - for k, v in value.items() - ] + sort_dict_entries( + [ + {"key": key_converter(k), "value": value_converter(v)} + for k, v in value.items() + ] + ) if value is not None else [] ) @@ -1679,27 +1687,22 @@ def _create_arrow_to_python_converter( element_type = arrow_type.value_type # Check if this is a dict representation - if pa.types.is_struct(element_type): - field_names = [field.name for field in element_type] - if set(field_names) == {"key", "value"}: - # Dict representation - key_field = next(f for f in element_type if f.name == "key") - value_field = next(f for f in element_type if f.name == "value") - - key_converter = self.get_arrow_to_python_converter(key_field.type) - value_converter = self.get_arrow_to_python_converter( - value_field.type - ) + if is_dict_arrow_type(arrow_type): + key_field = element_type.field("key") + value_field = element_type.field("value") - return lambda value: ( - { - key_converter(item["key"]): value_converter(item["value"]) - for item in value - if item is not None - } - if value - else {} - ) + key_converter = self.get_arrow_to_python_converter(key_field.type) + value_converter = self.get_arrow_to_python_converter(value_field.type) + + return lambda value: ( + { + key_converter(item["key"]): value_converter(item["value"]) + for item in value + if item is not None + } + if value + else {} + ) # Regular list element_converter = self.get_arrow_to_python_converter(element_type) From 072f61d03690a5895a35760f78e1f595dd1951a8 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:44:46 -0800 Subject: [PATCH 04/22] fix(hashing): hash dict entries in canonical key order (ITL-905) Tables can already hold a dict's key/value entries in any order (Arrow and Delta sources, rows read back from a store). The Arrow hasher now puts the entries of every dict in a column, at any depth and inside extension storage, in canonical key order before hashing. Columns whose type holds no dict skip the step after one type check. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/hashing/arrow_hashers.py | 15 ++++ src/orcapod/semantic_types/dict_order.py | 88 ++++++++++++++++++++-- tests/test_hashing/test_dict_order_hash.py | 9 ++- 3 files changed, 106 insertions(+), 6 deletions(-) diff --git a/src/orcapod/hashing/arrow_hashers.py b/src/orcapod/hashing/arrow_hashers.py index 54c27a528..5f67dbda0 100644 --- a/src/orcapod/hashing/arrow_hashers.py +++ b/src/orcapod/hashing/arrow_hashers.py @@ -5,11 +5,16 @@ import pyarrow as pa from starfix import ArrowDigester +from orcapod.errors import InvalidDictKeysError from orcapod.hashing.schema_cleaner import ( clean_schema_for_hashing, has_extension_metadata, ) from orcapod.hashing.visitors import SemanticHashingVisitor +from orcapod.semantic_types.dict_order import ( + canonical_storage_value, + contains_dict_arrow_type, +) from orcapod.utils.arrow_utils import normalize_extension_columns from orcapod.types import ContentHash @@ -96,6 +101,14 @@ def _process_table_columns(self, table: "pa.Table | pa.RecordBatch") -> "pa.Tabl ) try: + # A dict is order independent: put the entries of every dict in the + # column in canonical key order before hashing, whatever order the + # table stores them in (ITL-905). + if contains_dict_arrow_type(field.type): + column_data = [ + canonical_storage_value(field.type, value) + for value in column_data + ] new_type: pa.DataType | None = None processed_data: list[Any] = [] for value in column_data: @@ -110,6 +123,8 @@ def _process_table_columns(self, table: "pa.Table | pa.RecordBatch") -> "pa.Tabl new_columns.append(pa.array(processed_data, type=new_type)) new_fields.append(field.with_type(new_type)) + except InvalidDictKeysError as exc: + raise InvalidDictKeysError(f"Column {field.name!r}: {exc}") from exc except Exception as exc: raise RuntimeError( f"Failed to process column '{field.name}': {exc}" diff --git a/src/orcapod/semantic_types/dict_order.py b/src/orcapod/semantic_types/dict_order.py index 37eece72a..0c55756dd 100644 --- a/src/orcapod/semantic_types/dict_order.py +++ b/src/orcapod/semantic_types/dict_order.py @@ -22,6 +22,7 @@ from __future__ import annotations +import functools import math import types import typing @@ -50,11 +51,7 @@ def is_dict_arrow_type(arrow_type: pa.DataType) -> bool: Returns: True for a dict-encoded list type. """ - if not ( - pa.types.is_list(arrow_type) - or pa.types.is_large_list(arrow_type) - or pa.types.is_fixed_size_list(arrow_type) - ): + if not _is_list_type(arrow_type): return False element = arrow_type.value_type return ( @@ -64,6 +61,87 @@ def is_dict_arrow_type(arrow_type: pa.DataType) -> bool: ) +def _is_list_type(arrow_type: pa.DataType) -> bool: + return ( + pa.types.is_list(arrow_type) + or pa.types.is_large_list(arrow_type) + or pa.types.is_fixed_size_list(arrow_type) + ) + + +def contains_dict_arrow_type(arrow_type: pa.DataType) -> bool: + """Return whether a dict-encoded list occurs anywhere in ``arrow_type``. + + Looks through list elements, struct fields and extension storage. + + Args: + arrow_type: The Arrow type to search. + + Returns: + True if ``arrow_type`` is or contains a dict-encoded list. + """ + try: + return _contains_dict_cached(arrow_type) + except TypeError: # an extension type that does not implement __hash__ + return _contains_dict(arrow_type) + + +@functools.lru_cache(maxsize=1024) +def _contains_dict_cached(arrow_type: pa.DataType) -> bool: + return _contains_dict(arrow_type) + + +def _contains_dict(arrow_type: pa.DataType) -> bool: + if isinstance(arrow_type, pa.ExtensionType): + return contains_dict_arrow_type(arrow_type.storage_type) + if _is_list_type(arrow_type): + return is_dict_arrow_type(arrow_type) or contains_dict_arrow_type( + arrow_type.value_type + ) + if pa.types.is_struct(arrow_type): + return any( + contains_dict_arrow_type(arrow_type.field(i).type) + for i in range(arrow_type.num_fields) + ) + return False + + +def canonical_storage_value(arrow_type: pa.DataType, value: Any) -> Any: + """Return a storage value with the entries of every dict in canonical key order. + + ``value`` is one value of an Arrow column of type ``arrow_type`` as ``to_pylist`` + reads it. Dicts at any depth (inside lists, struct fields and extension storage) + are sorted innermost first; every other list keeps its order. The input is not + mutated. + + Args: + arrow_type: The Arrow type of ``value``. + value: The storage value. + + Returns: + ``value`` itself when ``arrow_type`` holds no dict, otherwise a canonical copy. + + Raises: + InvalidDictKeysError: For a dict with a null entry, a NaN key, keys that cannot + be ordered against each other, or a duplicate key. + """ + if value is None or not contains_dict_arrow_type(arrow_type): + return value + if isinstance(arrow_type, pa.ExtensionType): + return canonical_storage_value(arrow_type.storage_type, value) + if _is_list_type(arrow_type): + element_type = arrow_type.value_type + elements = [canonical_storage_value(element_type, v) for v in value] + if is_dict_arrow_type(arrow_type): + return sort_dict_entries(elements) + return elements + # A struct holding a dict somewhere in its fields. + return { + field.name: canonical_storage_value(field.type, value.get(field.name)) + for field in arrow_type + } + + def _key_rank(key: Any) -> tuple[Any, ...]: """Return the sort key of one storage key value. diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index dd411be56..ca5e57952 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -53,11 +53,18 @@ def _row_hash(value: Any, value_type: Any = None) -> str: def _arrow_source(entries: list[dict[str, Any]]) -> ArrowTableSource: + # Non-nullable like a DictSource column declared ``dict[str, int]``. table = pa.table( { "k": pa.array(["r"], pa.large_string()), "m": pa.array([entries], KV_TYPE), - } + }, + schema=pa.schema( + [ + pa.field("k", pa.large_string(), nullable=False), + pa.field("m", KV_TYPE, nullable=False), + ] + ), ) return ArrowTableSource(table, tag_columns=["k"], source_id="itl905") From 39179f5540573dae4b11544a94dbaf81b5a547ac Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:45:19 -0800 Subject: [PATCH 05/22] fix(datagrams): hold dict values in canonical key order in memory (ITL-905) A datagram built from Python values now holds each dict value (also inside lists, dicts and Optional) as a canonically ordered copy, so a fresh pod output and the same output read back from a cache iterate identically. The caller's dict is never mutated. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/datagrams/datagram.py | 27 ++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/src/orcapod/core/datagrams/datagram.py b/src/orcapod/core/datagrams/datagram.py index e44c1fc57..6c69550ee 100644 --- a/src/orcapod/core/datagrams/datagram.py +++ b/src/orcapod/core/datagrams/datagram.py @@ -29,8 +29,10 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.base import ContentIdentifiableBase +from orcapod.errors import InvalidDictKeysError from orcapod.protocols.semantic_types_protocols import TypeConverterProtocol from orcapod.semantic_types import infer_python_schema_from_pylist_data +from orcapod.semantic_types.dict_order import canonical_python_value from orcapod.system_constants import constants from orcapod.types import ColumnConfig, DataType, DataValue, Schema, SchemaLike from orcapod.utils import arrow_utils @@ -119,6 +121,9 @@ def _init_from_dict( if python_schema else inferred ) + self._data_dict = self._canonical_dict_values( + data_columns, self._data_python_schema + ) self._data_arrow_schema: pa.Schema | None = None if meta_info is not None: @@ -130,6 +135,28 @@ def _init_from_dict( self._meta_table: pa.Table | None = None self._context_table: pa.Table | None = None + def _canonical_dict_values( + self, columns: dict[str, DataValue], schema: Schema + ) -> dict[str, DataValue]: + """Return ``columns`` with every dict value in canonical key order. + + A dict is order independent (ITL-905): its entries are stored and hashed in + canonical key order, so a value built here holds the same order a stored copy + reads back with. Dicts are copied, never mutated; dicts inside other objects + (dataclass or pydantic fields) are left as given. + """ + out: dict[str, DataValue] = {} + for name, value in columns.items(): + try: + out[name] = canonical_python_value( + value, + schema.get(name), + self.converter.get_python_to_arrow_converter, + ) + except InvalidDictKeysError as exc: + raise InvalidDictKeysError(f"Column {name!r}: {exc}") from exc + return out + def _init_from_table( self, table: pa.Table, From 40eb82b107adb052185d2790a9bb10c461b1936c Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:46:05 -0800 Subject: [PATCH 06/22] test(golden): re-pin the two index_pick row hashes that hold an unsorted dict (ITL-905) The pipeline index_pick case builds rows with meta={'k': ..., 'j': ...}, not in key order. With dicts stored and hashed in canonical key order, the two Index output row data hashes move; nothing else does. Approved by the coordinator under the developer's November 11, 2026 ruling. Co-Authored-By: Claude Opus 5.5 --- tests/test_hashing/golden/data/pipeline.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_hashing/golden/data/pipeline.json b/tests/test_hashing/golden/data/pipeline.json index b35b12ae3..cd5c6ba3c 100644 --- a/tests/test_hashing/golden/data/pipeline.json +++ b/tests/test_hashing/golden/data/pipeline.json @@ -74,10 +74,10 @@ "index_pick.index.pipeline_hash": "semantic_v0.1:375b6916dd1d0c46e5553b471809ef66120dc092a19192a1f7b052e2481299c1", "index_pick.index.row[{\"subject\":\"s1\"}]._tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "dd65b531e3d957a0ae3f2fba11e021d5", "index_pick.index.row[{\"subject\":\"s1\"}]._tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "lists", - "index_pick.index.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000189f1942fbc40737ff78f6428f7be73a5dc531b1426c075ae3d5bc790b4ef8fcd", + "index_pick.index.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001e070353e8d329a2b9d9924105603730a2348626034ae79ccca0473c189415feb", "index_pick.index.row[{\"subject\":\"s2\"}]._tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "00c12388e8c4522893b029b39a8ec7aa", "index_pick.index.row[{\"subject\":\"s2\"}]._tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "lists", - "index_pick.index.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000164b528c22b7183231910091256b6cb1d63bef2ded631aa330114a2fef997ed8c", + "index_pick.index.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000119e013179645c0e80233af8ab0406ac1e04b789ce333197ed7a6d11b9e95d1bd", "index_pick.index.system_tag_columns": [ "_tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a", "_tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a" From bf28f7131be52b340092616d42d4ef35d917bd97 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:46:33 -0800 Subject: [PATCH 07/22] test(golden): pin dict order independence (ITL-905) New cases only: arrow dict_order (sorted and unsorted key/value lists pin one value; ordered struct pairs keep distinct values) and datagram dict_order (str keys, int keys, nested dicts in two insertion orders). Co-Authored-By: Claude Opus 5.5 --- tests/test_hashing/golden/cases.py | 53 ++++++++++++++++++++ tests/test_hashing/golden/data/arrow.json | 4 ++ tests/test_hashing/golden/data/datagram.json | 36 +++++++++++++ 3 files changed, 93 insertions(+) diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index a2e12d808..ab1457b68 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -550,6 +550,31 @@ def arrow_schema_hashes() -> dict[str, Any]: } +@case("arrow", "dict_order") +def arrow_dict_order_hashes() -> dict[str, Any]: + # ITL-905: a dict-encoded list hashes in canonical key order, so the sorted and + # unsorted tables pin the same value; a list of other structs keeps its order. + hasher = _ctx().arrow_hasher + kv = pa.large_list( + pa.struct([pa.field("key", pa.large_string()), pa.field("value", pa.int64())]) + ) + pairs = pa.large_list( + pa.struct([pa.field("f0", pa.large_string()), pa.field("f1", pa.int64())]) + ) + ab = [{"key": "a", "value": 1}, {"key": "b", "value": 2}] + tables = { + "kv.sorted": pa.table({"m": pa.array([ab], kv)}), + "kv.unsorted": pa.table({"m": pa.array([ab[::-1]], kv)}), + "pairs.ab": pa.table( + {"m": pa.array([[{"f0": "a", "f1": 1}, {"f0": "b", "f1": 2}]], pairs)} + ), + "pairs.ba": pa.table( + {"m": pa.array([[{"f0": "b", "f1": 2}, {"f0": "a", "f1": 1}]], pairs)} + ), + } + return {k: hasher.hash_table(t).to_string() for k, t in tables.items()} + + # --------------------------------------------------------------------------- # Group: datagram — public sources carrying each logical type # --------------------------------------------------------------------------- @@ -620,6 +645,34 @@ def _arrow_source(column: pa.Array[Any], nullable: bool = False) -> Any: return ArrowTableSource(table, tag_columns=["k"], source_id="golden_arrow") +@case("datagram", "dict_order") +def datagram_dict_order() -> dict[str, Any]: + # ITL-905: rows holding equal dicts in different insertion orders pin equal hashes. + from orcapod.sources import DictSource + + rows: dict[str, tuple[Any, list[Any]]] = { + "str_keys": (dict[str, int], [{"b": 2, "a": 1}, {"a": 1, "b": 2}]), + "int_keys": (dict[int, str], [{10: "x", 9: "y"}, {9: "y", 10: "x"}]), + "nested": ( + dict[str, dict[str, int]], + [ + {"y": {"d": 4, "c": 3}, "x": {"a": 1}}, + {"x": {"a": 1}, "y": {"c": 3, "d": 4}}, + ], + ), + } + out: dict[str, Any] = {} + for name, (value_type, values) in rows.items(): + source = DictSource( + data=[{"k": f"r{i}", "m": v} for i, v in enumerate(values)], + tag_columns=["k"], + data_schema={"k": str, "m": value_type}, + source_id=f"golden_dict_{name}", + ) + out.update(_prefixed(name, stream_summary(source))) + return out + + @case("datagram", "arrow_source") def datagram_arrow_source() -> dict[str, Any]: ints = [1, 2, 3] diff --git a/tests/test_hashing/golden/data/arrow.json b/tests/test_hashing/golden/data/arrow.json index ff71bb164..615d6ef9e 100644 --- a/tests/test_hashing/golden/data/arrow.json +++ b/tests/test_hashing/golden/data/arrow.json @@ -1,4 +1,8 @@ { + "dict_order.kv.sorted": "arrow_v0.1:000001a2e624080089c3ee324cdded473a7fb482ccd86ddb8a00cb736671aa8a30498f", + "dict_order.kv.unsorted": "arrow_v0.1:000001a2e624080089c3ee324cdded473a7fb482ccd86ddb8a00cb736671aa8a30498f", + "dict_order.pairs.ab": "arrow_v0.1:0000018236ae50c0879c0da86fa4640be236f101bb0487e88bc935a089952abef915f9", + "dict_order.pairs.ba": "arrow_v0.1:00000158d0d63f1a04a1d6fa1f1ca72561a74ea0024933aea63948c0dc81a7c26b282c", "extension_table.extension.path": "arrow_v0.1:000001e32d9b28a94c4f47e33a647ed137a7316d4cf80d5bd216cc74e2ea4e46bc8d48", "extension_table.extension.unregistered_passthrough": "arrow_v0.1:00000197e4832ff6efd7436451ab12fbbcde498ae4871ab74b13c9b80c32a4ba9ee44a", "extension_table.extension.uuid": "arrow_v0.1:0000016efd6e3b9a4c03e2c78bc410a5e04bd5e4bfd7800e3a4054b67805c22fd6436f", diff --git a/tests/test_hashing/golden/data/datagram.json b/tests/test_hashing/golden/data/datagram.json index e9bf5b544..5071e6eef 100644 --- a/tests/test_hashing/golden/data/datagram.json +++ b/tests/test_hashing/golden/data/datagram.json @@ -113,6 +113,42 @@ "_tag_source_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14", "_tag_record_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14" ], + "dict_order.int_keys.content_hash": "semantic_v0.1:f1f37a6683326aff03d599f3c5d6d3c323d2b8410499957e9ce8b85135e79087", + "dict_order.int_keys.pipeline_hash": "semantic_v0.1:1ed51934abd8eb8fd765c716cf4934bfd42f43409a5f4b94de1d9aaad7582ca2", + "dict_order.int_keys.row[{\"k\":\"r0\"}]._tag_record_id::1ed51934abd8eb8fd765c716cf4934bfd42f43409a5f4b94de1d9aaad7582ca2": "a7a545a63e0d5cfdbbb7444d2ff924c4", + "dict_order.int_keys.row[{\"k\":\"r0\"}]._tag_source_id::1ed51934abd8eb8fd765c716cf4934bfd42f43409a5f4b94de1d9aaad7582ca2": "golden_dict_int_keys", + "dict_order.int_keys.row[{\"k\":\"r0\"}].data_hash": "arrow_v0.1:0000010ab832e68f5e92143d2f6f03f48f61849e7520bf0958b1eb7a90a2bb7480311c", + "dict_order.int_keys.row[{\"k\":\"r1\"}]._tag_record_id::1ed51934abd8eb8fd765c716cf4934bfd42f43409a5f4b94de1d9aaad7582ca2": "7e203bc987835298ac9b02baf41ca4cf", + "dict_order.int_keys.row[{\"k\":\"r1\"}]._tag_source_id::1ed51934abd8eb8fd765c716cf4934bfd42f43409a5f4b94de1d9aaad7582ca2": "golden_dict_int_keys", + "dict_order.int_keys.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000010ab832e68f5e92143d2f6f03f48f61849e7520bf0958b1eb7a90a2bb7480311c", + "dict_order.int_keys.system_tag_columns": [ + "_tag_source_id::1ed51934abd8eb8fd765c716cf4934bfd42f43409a5f4b94de1d9aaad7582ca2", + "_tag_record_id::1ed51934abd8eb8fd765c716cf4934bfd42f43409a5f4b94de1d9aaad7582ca2" + ], + "dict_order.nested.content_hash": "semantic_v0.1:abb3a76012b824b50a86dbd35148094767d1cc6cc83b77f1b5ef8774d9f22613", + "dict_order.nested.pipeline_hash": "semantic_v0.1:dd792c45f9d33453d960e667968eb8d6c3b723c6b9929fdc33aa1d1998cb53c1", + "dict_order.nested.row[{\"k\":\"r0\"}]._tag_record_id::dd792c45f9d33453d960e667968eb8d6c3b723c6b9929fdc33aa1d1998cb53c1": "f2163b15031857e7b0edc5c8092fe897", + "dict_order.nested.row[{\"k\":\"r0\"}]._tag_source_id::dd792c45f9d33453d960e667968eb8d6c3b723c6b9929fdc33aa1d1998cb53c1": "golden_dict_nested", + "dict_order.nested.row[{\"k\":\"r0\"}].data_hash": "arrow_v0.1:000001404020d569538a434976cbf5970df8a23155a078020124f53b7b0db4d608e221", + "dict_order.nested.row[{\"k\":\"r1\"}]._tag_record_id::dd792c45f9d33453d960e667968eb8d6c3b723c6b9929fdc33aa1d1998cb53c1": "f6ab4ac474e750cb89debd9cb2697ccb", + "dict_order.nested.row[{\"k\":\"r1\"}]._tag_source_id::dd792c45f9d33453d960e667968eb8d6c3b723c6b9929fdc33aa1d1998cb53c1": "golden_dict_nested", + "dict_order.nested.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001404020d569538a434976cbf5970df8a23155a078020124f53b7b0db4d608e221", + "dict_order.nested.system_tag_columns": [ + "_tag_source_id::dd792c45f9d33453d960e667968eb8d6c3b723c6b9929fdc33aa1d1998cb53c1", + "_tag_record_id::dd792c45f9d33453d960e667968eb8d6c3b723c6b9929fdc33aa1d1998cb53c1" + ], + "dict_order.str_keys.content_hash": "semantic_v0.1:1fbc6226941e201ddcb3e13ca09d1ced7df3fb4d68b07090e451fbeb8da4c3ed", + "dict_order.str_keys.pipeline_hash": "semantic_v0.1:7f992e15265162404ce4f6dcb05344659b2ae9f060678dd3b9631e9073732636", + "dict_order.str_keys.row[{\"k\":\"r0\"}]._tag_record_id::7f992e15265162404ce4f6dcb05344659b2ae9f060678dd3b9631e9073732636": "bf6c81a8f6a75d7a9615d4f2034b772f", + "dict_order.str_keys.row[{\"k\":\"r0\"}]._tag_source_id::7f992e15265162404ce4f6dcb05344659b2ae9f060678dd3b9631e9073732636": "golden_dict_str_keys", + "dict_order.str_keys.row[{\"k\":\"r0\"}].data_hash": "arrow_v0.1:00000196dfc79e137a9b9c52285a345af1003c3835e32cc0fd07e2f0d0218b7356192c", + "dict_order.str_keys.row[{\"k\":\"r1\"}]._tag_record_id::7f992e15265162404ce4f6dcb05344659b2ae9f060678dd3b9631e9073732636": "2cec7386d5645ec48bf8dfa5d661196a", + "dict_order.str_keys.row[{\"k\":\"r1\"}]._tag_source_id::7f992e15265162404ce4f6dcb05344659b2ae9f060678dd3b9631e9073732636": "golden_dict_str_keys", + "dict_order.str_keys.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000196dfc79e137a9b9c52285a345af1003c3835e32cc0fd07e2f0d0218b7356192c", + "dict_order.str_keys.system_tag_columns": [ + "_tag_source_id::7f992e15265162404ce4f6dcb05344659b2ae9f060678dd3b9631e9073732636", + "_tag_record_id::7f992e15265162404ce4f6dcb05344659b2ae9f060678dd3b9631e9073732636" + ], "list_path.content_hash": "semantic_v0.1:7be2936fd87936bea5dce849f7e947c7e360c4f08e8dbf2d18e18948fbb3d389", "list_path.pipeline_hash": "semantic_v0.1:3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020", "list_path.row[{\"k\":\"r1\"}]._tag_record_id::3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020": "059c77251e845bafb5a3839bf547a2f5", From 210a75db37732cf98b61bfaf0e1aae0c66fcb9e8 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:47:52 -0800 Subject: [PATCH 08/22] fix(semantic_types): store plain set[T] elements in canonical order (ITL-905) A set[T] of a non-logical element type was stored as large_list in Python's set iteration order, which depends on PYTHONHASHSEED for str and bytes, so the same set had a different row hash in every process. Its elements are now sorted with the same ordering rule as dict keys. set[T] of a logical type already sorted (ListLogicalType) and is unchanged. The ordering module becomes semantic_types/canonical_order.py and its error CanonicalOrderError, since it now covers sets as well as dicts. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/datagrams/datagram.py | 8 +-- src/orcapod/errors.py | 11 +-- src/orcapod/hashing/arrow_hashers.py | 8 +-- .../{dict_order.py => canonical_order.py} | 61 +++++++++++------ .../semantic_types/universal_converter.py | 22 ++++-- tests/test_hashing/test_set_order_hash.py | 68 +++++++++++++++++++ 6 files changed, 140 insertions(+), 38 deletions(-) rename src/orcapod/semantic_types/{dict_order.py => canonical_order.py} (78%) create mode 100644 tests/test_hashing/test_set_order_hash.py diff --git a/src/orcapod/core/datagrams/datagram.py b/src/orcapod/core/datagrams/datagram.py index 6c69550ee..029bd0a4e 100644 --- a/src/orcapod/core/datagrams/datagram.py +++ b/src/orcapod/core/datagrams/datagram.py @@ -29,10 +29,10 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.base import ContentIdentifiableBase -from orcapod.errors import InvalidDictKeysError +from orcapod.errors import CanonicalOrderError from orcapod.protocols.semantic_types_protocols import TypeConverterProtocol from orcapod.semantic_types import infer_python_schema_from_pylist_data -from orcapod.semantic_types.dict_order import canonical_python_value +from orcapod.semantic_types.canonical_order import canonical_python_value from orcapod.system_constants import constants from orcapod.types import ColumnConfig, DataType, DataValue, Schema, SchemaLike from orcapod.utils import arrow_utils @@ -153,8 +153,8 @@ def _canonical_dict_values( schema.get(name), self.converter.get_python_to_arrow_converter, ) - except InvalidDictKeysError as exc: - raise InvalidDictKeysError(f"Column {name!r}: {exc}") from exc + except CanonicalOrderError as exc: + raise CanonicalOrderError(f"Column {name!r}: {exc}") from exc return out def _init_from_table( diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index e016ffcb5..40b89ff3d 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -26,12 +26,13 @@ class DuplicateTagError(ValueError): pass -class InvalidDictKeysError(ValueError): - """Raised when a dict value's keys have no canonical order. +class CanonicalOrderError(ValueError): + """Raised when a dict or set value has no canonical order. - A dict is stored with its entries in canonical key order (ITL-905), so its keys must - be totally ordered and unique. Raised for a NaN key, keys that cannot be compared, - two keys with the same stored value, or a null entry in a stored key/value list. + Dict entries and set elements are stored in canonical order (ITL-905), so dict keys + and set elements must be totally ordered and unique. Raised for a NaN key or element, + values that cannot be compared, two keys or elements with the same stored value, or + a null entry in a stored key/value list. """ diff --git a/src/orcapod/hashing/arrow_hashers.py b/src/orcapod/hashing/arrow_hashers.py index 5f67dbda0..351bdc783 100644 --- a/src/orcapod/hashing/arrow_hashers.py +++ b/src/orcapod/hashing/arrow_hashers.py @@ -5,13 +5,13 @@ import pyarrow as pa from starfix import ArrowDigester -from orcapod.errors import InvalidDictKeysError +from orcapod.errors import CanonicalOrderError from orcapod.hashing.schema_cleaner import ( clean_schema_for_hashing, has_extension_metadata, ) from orcapod.hashing.visitors import SemanticHashingVisitor -from orcapod.semantic_types.dict_order import ( +from orcapod.semantic_types.canonical_order import ( canonical_storage_value, contains_dict_arrow_type, ) @@ -123,8 +123,8 @@ def _process_table_columns(self, table: "pa.Table | pa.RecordBatch") -> "pa.Tabl new_columns.append(pa.array(processed_data, type=new_type)) new_fields.append(field.with_type(new_type)) - except InvalidDictKeysError as exc: - raise InvalidDictKeysError(f"Column {field.name!r}: {exc}") from exc + except CanonicalOrderError as exc: + raise CanonicalOrderError(f"Column {field.name!r}: {exc}") from exc except Exception as exc: raise RuntimeError( f"Failed to process column '{field.name}': {exc}" diff --git a/src/orcapod/semantic_types/dict_order.py b/src/orcapod/semantic_types/canonical_order.py similarity index 78% rename from src/orcapod/semantic_types/dict_order.py rename to src/orcapod/semantic_types/canonical_order.py index 0c55756dd..5a420d886 100644 --- a/src/orcapod/semantic_types/dict_order.py +++ b/src/orcapod/semantic_types/canonical_order.py @@ -1,12 +1,15 @@ -"""Canonical entry order for dict values. - -A ``dict[K, V]`` value is stored as ``large_list>``. A dict is order -independent, so two equal dicts must store and hash identically whatever order their -entries were inserted in (ITL-905). This module holds the one definition of that order. -It is applied when a Python dict is converted to Arrow, when a datagram is built from -Python values, and by the Arrow hasher to key/value lists that arrive already stored. - -The order is defined on storage key values (what the converter produces and what +"""Canonical order for the unordered containers ``dict`` and ``set``. + +A ``dict[K, V]`` value is stored as ``large_list>`` and a plain +``set[T]`` as ``large_list``. Both are order independent, so two equal values must +store and hash identically whatever order their entries were inserted or iterated in +(ITL-905). This module holds the one definition of that order. Dict entries are sorted +by key when a Python dict is converted to Arrow, when a datagram is built from Python +values, and by the Arrow hasher for key/value lists that arrive already stored. Set +elements are sorted when a Python set is converted to Arrow (a stored set is a plain +list, so it cannot be recognised later). + +The order is defined on storage values (what the converter produces and what ``to_pylist`` reads back), so the converter and the hasher cannot disagree: * ``None`` sorts first. @@ -16,8 +19,8 @@ * ``date``, ``datetime`` (by instant), ``time``, ``timedelta``: natural order. * struct keys (a ``dict`` of field values): field by field, in field order. -Keys that cannot be ordered against each other, duplicate keys and null entries raise -``InvalidDictKeysError``. +Keys or elements that cannot be ordered against each other, duplicate dict keys and +null dict entries raise ``CanonicalOrderError``. """ from __future__ import annotations @@ -29,7 +32,7 @@ from collections.abc import Callable, Mapping from typing import TYPE_CHECKING, Any, get_args, get_origin -from orcapod.errors import InvalidDictKeysError +from orcapod.errors import CanonicalOrderError from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -122,7 +125,7 @@ def canonical_storage_value(arrow_type: pa.DataType, value: Any) -> Any: ``value`` itself when ``arrow_type`` holds no dict, otherwise a canonical copy. Raises: - InvalidDictKeysError: For a dict with a null entry, a NaN key, keys that cannot + CanonicalOrderError: For a dict with a null entry, a NaN key, keys that cannot be ordered against each other, or a duplicate key. """ if value is None or not contains_dict_arrow_type(arrow_type): @@ -146,12 +149,12 @@ def _key_rank(key: Any) -> tuple[Any, ...]: """Return the sort key of one storage key value. Raises: - InvalidDictKeysError: If the key is a NaN float. + CanonicalOrderError: If the key is a NaN float. """ if key is None: return (0,) if isinstance(key, float) and math.isnan(key): - raise InvalidDictKeysError("a NaN dict key has no canonical order") + raise CanonicalOrderError("a NaN key or set element has no canonical order") if isinstance(key, Mapping): return (1, tuple(_key_rank(v) for v in key.values())) return (1, key) @@ -161,15 +164,15 @@ def _sorted_by_rank(items: list[tuple[tuple[Any, ...], Any]]) -> list[Any]: """Sort ``(rank, payload)`` pairs by rank and return the payloads. Raises: - InvalidDictKeysError: If two ranks cannot be compared or are equal. + CanonicalOrderError: If two ranks cannot be compared or are equal. """ try: items = sorted(items, key=lambda item: item[0]) except TypeError as exc: - raise InvalidDictKeysError(f"dict keys cannot be ordered: {exc}") from exc + raise CanonicalOrderError(f"values cannot be ordered: {exc}") from exc for (previous, _), (current, payload) in zip(items, items[1:]): if previous == current: - raise InvalidDictKeysError(f"duplicate key in dict value: {payload!r}") + raise CanonicalOrderError(f"duplicate key or set element: {payload!r}") return [payload for _, payload in items] @@ -184,17 +187,33 @@ def sort_dict_entries(entries: list[Any]) -> list[Any]: A new list holding the same entries, sorted by key. Raises: - InvalidDictKeysError: For a null entry, a NaN key, keys that cannot be ordered + CanonicalOrderError: For a null entry, a NaN key, keys that cannot be ordered against each other, or a duplicate key. """ ranked = [] for entry in entries: if entry is None: - raise InvalidDictKeysError("a dict value holds a null entry") + raise CanonicalOrderError("a dict value holds a null entry") ranked.append((_key_rank(entry["key"]), entry)) return _sorted_by_rank(ranked) +def sort_set_elements(elements: list[Any]) -> list[Any]: + """Return the storage elements of one set value in canonical order. + + Args: + elements: The converted elements of a set. + + Returns: + A new list holding the same elements, sorted. + + Raises: + CanonicalOrderError: For a NaN element, elements that cannot be ordered against + each other, or two elements with the same storage value. + """ + return _sorted_by_rank([(_key_rank(element), element) for element in elements]) + + def canonical_python_value( value: Any, python_type: Any, @@ -217,7 +236,7 @@ def canonical_python_value( ``value`` itself when nothing changes, otherwise a canonically ordered copy. Raises: - InvalidDictKeysError: For a key that has no canonical order (NaN), keys that + CanonicalOrderError: For a key that has no canonical order (NaN), keys that cannot be ordered, or two keys that convert to the same storage key. """ if value is None: diff --git a/src/orcapod/semantic_types/universal_converter.py b/src/orcapod/semantic_types/universal_converter.py index c3b3f4113..6824d631c 100644 --- a/src/orcapod/semantic_types/universal_converter.py +++ b/src/orcapod/semantic_types/universal_converter.py @@ -23,8 +23,12 @@ from typing import TYPE_CHECKING, Any, TypedDict, get_args, get_origin from orcapod.contexts import DataContext, resolve_context -from orcapod.errors import InvalidDictKeysError -from orcapod.semantic_types.dict_order import is_dict_arrow_type, sort_dict_entries +from orcapod.errors import CanonicalOrderError +from orcapod.semantic_types.canonical_order import ( + is_dict_arrow_type, + sort_dict_entries, + sort_set_elements, +) from orcapod.semantic_types.type_inference import infer_python_schema_from_pylist_data from orcapod.types import DataType, Schema, SchemaLike from orcapod.utils import arrow_utils @@ -1010,8 +1014,8 @@ def python_dicts_to_struct_dicts( if field_name in record: try: converted_record[field_name] = converter(record[field_name]) - except InvalidDictKeysError as exc: - raise InvalidDictKeysError( + except CanonicalOrderError as exc: + raise CanonicalOrderError( f"Column {field_name!r}: {exc}" ) from exc else: @@ -1613,6 +1617,16 @@ def _convert_datetime(dt: datetime) -> datetime: else [] ) + elif origin is set: + # A set has no order: elements are stored sorted, never in iteration + # order, which depends on PYTHONHASHSEED for str and bytes (ITL-905). + element_converter = self.get_python_to_arrow_converter(args[0]) + return lambda value: ( + sort_set_elements([element_converter(item) for item in value]) + if value is not None + else [] + ) + elif origin is tuple: if len(set(args)) == 1: # Homogeneous tuple diff --git a/tests/test_hashing/test_set_order_hash.py b/tests/test_hashing/test_set_order_hash.py new file mode 100644 index 000000000..dc4c7253d --- /dev/null +++ b/tests/test_hashing/test_set_order_hash.py @@ -0,0 +1,68 @@ +"""A plain ``set[T]`` value hashes the same in every process (ITL-905). + +A ``set[T]`` of a non-logical element type is stored as ``large_list``. Python's set +iteration order depends on ``PYTHONHASHSEED`` for str and bytes elements, so the stored +order, and the row hash, must not come from iteration order. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import textwrap + +from orcapod.sources import DictSource + +_ROW_HASH_SCRIPT = textwrap.dedent( + """ + from orcapod.sources import DictSource + + source = DictSource( + data=[{"k": "r", "s": {"apple", "banana", "cherry", "date", "elder"}}], + tag_columns=["k"], + data_schema={"k": str, "s": set[str]}, + source_id="set_order", + ) + _, data = next(iter(source.iter_data())) + print(data.content_hash().to_string()) + """ +) + + +def _row_hash_in_subprocess(seed: str) -> str: + env = {**os.environ, "PYTHONHASHSEED": seed} + result = subprocess.run( + [sys.executable, "-c", _ROW_HASH_SCRIPT], + env=env, + capture_output=True, + text=True, + check=True, + ) + return result.stdout.strip() + + +def _source(value: set[str]) -> DictSource: + return DictSource( + data=[{"k": "r", "s": value}], + tag_columns=["k"], + data_schema={"k": str, "s": set[str]}, + source_id="set_order", + ) + + +class TestPlainSetOrder: + def test_row_hash_is_independent_of_hash_seed(self): + assert _row_hash_in_subprocess("1") == _row_hash_in_subprocess("2") + + def test_stored_elements_are_sorted(self): + table = _source({"cherry", "apple", "banana"}).as_table() + assert table.column("s").to_pylist() == [["apple", "banana", "cherry"]] + + def test_int_set_elements_are_sorted(self): + source = DictSource( + data=[{"k": "r", "s": {300, -1, 7}}], + tag_columns=["k"], + data_schema={"k": str, "s": set[int]}, + ) + assert source.as_table().column("s").to_pylist() == [[-1, 7, 300]] From 24732a5578903a7840405a254132e4f7e244bb7d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:49:32 -0800 Subject: [PATCH 09/22] fix(hashing): hash Arrow tables exactly as stored; sort only at conversion (ITL-905) Developer ruling (November 11, 2026): starfix hashing never sorts list contents, and a list> that arrives in Arrow form is an ordered list whose order is part of its hash. Canonical order is established only where a Python dict or set becomes Arrow. Revert the hash-time canonicalisation in StarfixArrowHasher, drop the Arrow-side helpers (is_dict_arrow_type, contains_dict_arrow_type, canonical_storage_value) and the converter's arrow-to-python refactor, and pin that unsorted Arrow-native key/value lists hash differently from sorted ones (golden arrow dict_order.kv.unsorted, added on this branch). Co-Authored-By: Claude Opus 5.5 --- src/orcapod/hashing/arrow_hashers.py | 15 -- src/orcapod/semantic_types/canonical_order.py | 160 +++--------------- .../semantic_types/universal_converter.py | 51 +++--- tests/test_hashing/golden/cases.py | 5 +- tests/test_hashing/golden/data/arrow.json | 2 +- tests/test_hashing/test_dict_order_hash.py | 63 +++---- 6 files changed, 71 insertions(+), 225 deletions(-) diff --git a/src/orcapod/hashing/arrow_hashers.py b/src/orcapod/hashing/arrow_hashers.py index 351bdc783..54c27a528 100644 --- a/src/orcapod/hashing/arrow_hashers.py +++ b/src/orcapod/hashing/arrow_hashers.py @@ -5,16 +5,11 @@ import pyarrow as pa from starfix import ArrowDigester -from orcapod.errors import CanonicalOrderError from orcapod.hashing.schema_cleaner import ( clean_schema_for_hashing, has_extension_metadata, ) from orcapod.hashing.visitors import SemanticHashingVisitor -from orcapod.semantic_types.canonical_order import ( - canonical_storage_value, - contains_dict_arrow_type, -) from orcapod.utils.arrow_utils import normalize_extension_columns from orcapod.types import ContentHash @@ -101,14 +96,6 @@ def _process_table_columns(self, table: "pa.Table | pa.RecordBatch") -> "pa.Tabl ) try: - # A dict is order independent: put the entries of every dict in the - # column in canonical key order before hashing, whatever order the - # table stores them in (ITL-905). - if contains_dict_arrow_type(field.type): - column_data = [ - canonical_storage_value(field.type, value) - for value in column_data - ] new_type: pa.DataType | None = None processed_data: list[Any] = [] for value in column_data: @@ -123,8 +110,6 @@ def _process_table_columns(self, table: "pa.Table | pa.RecordBatch") -> "pa.Tabl new_columns.append(pa.array(processed_data, type=new_type)) new_fields.append(field.with_type(new_type)) - except CanonicalOrderError as exc: - raise CanonicalOrderError(f"Column {field.name!r}: {exc}") from exc except Exception as exc: raise RuntimeError( f"Failed to process column '{field.name}': {exc}" diff --git a/src/orcapod/semantic_types/canonical_order.py b/src/orcapod/semantic_types/canonical_order.py index 5a420d886..9525fbe3d 100644 --- a/src/orcapod/semantic_types/canonical_order.py +++ b/src/orcapod/semantic_types/canonical_order.py @@ -1,152 +1,38 @@ """Canonical order for the unordered containers ``dict`` and ``set``. A ``dict[K, V]`` value is stored as ``large_list>`` and a plain -``set[T]`` as ``large_list``. Both are order independent, so two equal values must -store and hash identically whatever order their entries were inserted or iterated in -(ITL-905). This module holds the one definition of that order. Dict entries are sorted -by key when a Python dict is converted to Arrow, when a datagram is built from Python -values, and by the Arrow hasher for key/value lists that arrive already stored. Set -elements are sorted when a Python set is converted to Arrow (a stored set is a plain -list, so it cannot be recognised later). +``set[T]`` as ``large_list``. A dict and a set are order independent, so the order +they are stored in must not come from insertion or iteration order (ITL-905). Orcapod +puts them in canonical order where a Python value becomes Arrow: in the converter, and +in a datagram built from Python values. An Arrow table is always hashed exactly as it +is stored; a ``list>`` that arrives already in Arrow form is an +ordered list at the Arrow level, and its order is part of its hash. -The order is defined on storage values (what the converter produces and what -``to_pylist`` reads back), so the converter and the hasher cannot disagree: +The order is defined on storage values (what the converter produces): * ``None`` sorts first. -* ``bool``, ``int``, ``float``, ``Decimal``: numeric. A NaN key has no order and is - rejected. ``-0.0`` and ``0.0`` are one key. +* ``bool``, ``int``, ``float``, ``Decimal``: numeric. A NaN has no order and is + rejected. ``-0.0`` and ``0.0`` are one value. * ``str``: Unicode code point (never locale). ``bytes``: unsigned bytewise. * ``date``, ``datetime`` (by instant), ``time``, ``timedelta``: natural order. -* struct keys (a ``dict`` of field values): field by field, in field order. -Keys or elements that cannot be ordered against each other, duplicate dict keys and -null dict entries raise ``CanonicalOrderError``. +Keys or elements that cannot be ordered against each other, and two keys or elements +with the same storage value, raise ``CanonicalOrderError``. """ from __future__ import annotations -import functools import math import types import typing -from collections.abc import Callable, Mapping -from typing import TYPE_CHECKING, Any, get_args, get_origin +from collections.abc import Callable +from typing import Any, get_args, get_origin from orcapod.errors import CanonicalOrderError -from orcapod.utils.lazy_module import LazyModule - -if TYPE_CHECKING: - import pyarrow as pa -else: - pa = LazyModule("pyarrow") - - -def is_dict_arrow_type(arrow_type: pa.DataType) -> bool: - """Return whether ``arrow_type`` is Orcapod's encoding of a ``dict``. - - A dict is a list (regular, large or fixed-size) whose element is a struct with - exactly two fields named ``key`` and ``value``. Any other list of structs is an - ordered list, whose order is part of its identity. - - Args: - arrow_type: The Arrow type to test. - - Returns: - True for a dict-encoded list type. - """ - if not _is_list_type(arrow_type): - return False - element = arrow_type.value_type - return ( - pa.types.is_struct(element) - and element.num_fields == 2 - and {element.field(0).name, element.field(1).name} == {"key", "value"} - ) - - -def _is_list_type(arrow_type: pa.DataType) -> bool: - return ( - pa.types.is_list(arrow_type) - or pa.types.is_large_list(arrow_type) - or pa.types.is_fixed_size_list(arrow_type) - ) - - -def contains_dict_arrow_type(arrow_type: pa.DataType) -> bool: - """Return whether a dict-encoded list occurs anywhere in ``arrow_type``. - - Looks through list elements, struct fields and extension storage. - - Args: - arrow_type: The Arrow type to search. - - Returns: - True if ``arrow_type`` is or contains a dict-encoded list. - """ - try: - return _contains_dict_cached(arrow_type) - except TypeError: # an extension type that does not implement __hash__ - return _contains_dict(arrow_type) - - -@functools.lru_cache(maxsize=1024) -def _contains_dict_cached(arrow_type: pa.DataType) -> bool: - return _contains_dict(arrow_type) - - -def _contains_dict(arrow_type: pa.DataType) -> bool: - if isinstance(arrow_type, pa.ExtensionType): - return contains_dict_arrow_type(arrow_type.storage_type) - if _is_list_type(arrow_type): - return is_dict_arrow_type(arrow_type) or contains_dict_arrow_type( - arrow_type.value_type - ) - if pa.types.is_struct(arrow_type): - return any( - contains_dict_arrow_type(arrow_type.field(i).type) - for i in range(arrow_type.num_fields) - ) - return False - - -def canonical_storage_value(arrow_type: pa.DataType, value: Any) -> Any: - """Return a storage value with the entries of every dict in canonical key order. - - ``value`` is one value of an Arrow column of type ``arrow_type`` as ``to_pylist`` - reads it. Dicts at any depth (inside lists, struct fields and extension storage) - are sorted innermost first; every other list keeps its order. The input is not - mutated. - - Args: - arrow_type: The Arrow type of ``value``. - value: The storage value. - - Returns: - ``value`` itself when ``arrow_type`` holds no dict, otherwise a canonical copy. - - Raises: - CanonicalOrderError: For a dict with a null entry, a NaN key, keys that cannot - be ordered against each other, or a duplicate key. - """ - if value is None or not contains_dict_arrow_type(arrow_type): - return value - if isinstance(arrow_type, pa.ExtensionType): - return canonical_storage_value(arrow_type.storage_type, value) - if _is_list_type(arrow_type): - element_type = arrow_type.value_type - elements = [canonical_storage_value(element_type, v) for v in value] - if is_dict_arrow_type(arrow_type): - return sort_dict_entries(elements) - return elements - # A struct holding a dict somewhere in its fields. - return { - field.name: canonical_storage_value(field.type, value.get(field.name)) - for field in arrow_type - } def _key_rank(key: Any) -> tuple[Any, ...]: - """Return the sort key of one storage key value. + """Return the sort key of one storage key or set element. Raises: CanonicalOrderError: If the key is a NaN float. @@ -155,8 +41,6 @@ def _key_rank(key: Any) -> tuple[Any, ...]: return (0,) if isinstance(key, float) and math.isnan(key): raise CanonicalOrderError("a NaN key or set element has no canonical order") - if isinstance(key, Mapping): - return (1, tuple(_key_rank(v) for v in key.values())) return (1, key) @@ -177,25 +61,19 @@ def _sorted_by_rank(items: list[tuple[tuple[Any, ...], Any]]) -> list[Any]: def sort_dict_entries(entries: list[Any]) -> list[Any]: - """Return storage entries ``{"key": ..., "value": ...}`` in canonical key order. + """Return converted entries ``{"key": ..., "value": ...}`` in canonical key order. Args: - entries: The entries of one dict value, as the converter produces them or as - ``to_pylist`` reads them back. + entries: The entries of one dict value, as the converter produces them. Returns: A new list holding the same entries, sorted by key. Raises: - CanonicalOrderError: For a null entry, a NaN key, keys that cannot be ordered - against each other, or a duplicate key. + CanonicalOrderError: For a NaN key, keys that cannot be ordered against each + other, or two keys with the same storage value. """ - ranked = [] - for entry in entries: - if entry is None: - raise CanonicalOrderError("a dict value holds a null entry") - ranked.append((_key_rank(entry["key"]), entry)) - return _sorted_by_rank(ranked) + return _sorted_by_rank([(_key_rank(entry["key"]), entry) for entry in entries]) def sort_set_elements(elements: list[Any]) -> list[Any]: diff --git a/src/orcapod/semantic_types/universal_converter.py b/src/orcapod/semantic_types/universal_converter.py index 6824d631c..7a8b7f838 100644 --- a/src/orcapod/semantic_types/universal_converter.py +++ b/src/orcapod/semantic_types/universal_converter.py @@ -25,7 +25,6 @@ from orcapod.contexts import DataContext, resolve_context from orcapod.errors import CanonicalOrderError from orcapod.semantic_types.canonical_order import ( - is_dict_arrow_type, sort_dict_entries, sort_set_elements, ) @@ -1399,14 +1398,17 @@ def _convert_arrow_to_python(self, arrow_type: pa.DataType) -> type | Any: element_type = arrow_type.value_type # Check if this is a dict representation: list> - if is_dict_arrow_type(arrow_type): - key_field = element_type.field("key") - value_field = element_type.field("value") + if pa.types.is_struct(element_type): + field_names = [field.name for field in element_type] + if set(field_names) == {"key", "value"}: + # This is a dict + key_field = next(f for f in element_type if f.name == "key") + value_field = next(f for f in element_type if f.name == "value") - key_python_type = self.arrow_type_to_python_type(key_field.type) - value_python_type = self.arrow_type_to_python_type(value_field.type) + key_python_type = self.arrow_type_to_python_type(key_field.type) + value_python_type = self.arrow_type_to_python_type(value_field.type) - return dict[key_python_type, value_python_type] + return dict[key_python_type, value_python_type] # Regular list element_python_type = self.arrow_type_to_python_type(element_type) @@ -1701,22 +1703,27 @@ def _create_arrow_to_python_converter( element_type = arrow_type.value_type # Check if this is a dict representation - if is_dict_arrow_type(arrow_type): - key_field = element_type.field("key") - value_field = element_type.field("value") - - key_converter = self.get_arrow_to_python_converter(key_field.type) - value_converter = self.get_arrow_to_python_converter(value_field.type) + if pa.types.is_struct(element_type): + field_names = [field.name for field in element_type] + if set(field_names) == {"key", "value"}: + # Dict representation + key_field = next(f for f in element_type if f.name == "key") + value_field = next(f for f in element_type if f.name == "value") + + key_converter = self.get_arrow_to_python_converter(key_field.type) + value_converter = self.get_arrow_to_python_converter( + value_field.type + ) - return lambda value: ( - { - key_converter(item["key"]): value_converter(item["value"]) - for item in value - if item is not None - } - if value - else {} - ) + return lambda value: ( + { + key_converter(item["key"]): value_converter(item["value"]) + for item in value + if item is not None + } + if value + else {} + ) # Regular list element_converter = self.get_arrow_to_python_converter(element_type) diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index ab1457b68..f536a0f84 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -552,8 +552,9 @@ def arrow_schema_hashes() -> dict[str, Any]: @case("arrow", "dict_order") def arrow_dict_order_hashes() -> dict[str, Any]: - # ITL-905: a dict-encoded list hashes in canonical key order, so the sorted and - # unsorted tables pin the same value; a list of other structs keeps its order. + # ITL-905: an Arrow table is hashed exactly as stored. A key/value list that + # arrives in Arrow form keeps its order as identity (sorted and unsorted pin + # different values), like any other list of structs. hasher = _ctx().arrow_hasher kv = pa.large_list( pa.struct([pa.field("key", pa.large_string()), pa.field("value", pa.int64())]) diff --git a/tests/test_hashing/golden/data/arrow.json b/tests/test_hashing/golden/data/arrow.json index 615d6ef9e..0b319b76c 100644 --- a/tests/test_hashing/golden/data/arrow.json +++ b/tests/test_hashing/golden/data/arrow.json @@ -1,6 +1,6 @@ { "dict_order.kv.sorted": "arrow_v0.1:000001a2e624080089c3ee324cdded473a7fb482ccd86ddb8a00cb736671aa8a30498f", - "dict_order.kv.unsorted": "arrow_v0.1:000001a2e624080089c3ee324cdded473a7fb482ccd86ddb8a00cb736671aa8a30498f", + "dict_order.kv.unsorted": "arrow_v0.1:000001b82f48804f770106a9ce0638d6602b48f87137d26bee24e6f2005e74b26ff5ab", "dict_order.pairs.ab": "arrow_v0.1:0000018236ae50c0879c0da86fa4640be236f101bb0487e88bc935a089952abef915f9", "dict_order.pairs.ba": "arrow_v0.1:00000158d0d63f1a04a1d6fa1f1ca72561a74ea0024933aea63948c0dc81a7c26b282c", "extension_table.extension.path": "arrow_v0.1:000001e32d9b28a94c4f47e33a647ed137a7316d4cf80d5bd216cc74e2ea4e46bc8d48", diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index ca5e57952..c5e4f1a6b 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -1,10 +1,10 @@ """A dict's identity does not depend on its key order (ITL-905). -A ``dict[K, V]`` column is stored as ``large_list>``. Equal dicts must -give equal row hashes whatever order their entries were inserted or stored in, on every -path: Python values through the converter, Arrow tables that already hold key/value lists, -nested dicts, function pod outputs (sync and async), and cached results read back from a -store. Ordered layouts (``list[tuple[K, V]]``) keep order as identity. +A ``dict[K, V]`` column is stored as ``large_list>``. Equal Python +dicts must give equal row hashes whatever order their entries were inserted in, on every +path that turns a Python dict into Arrow: sources, nested dicts, dataclass fields, +function pod outputs (sync and async), and cached results read back from a store. An +Arrow table that already holds a key/value list is hashed exactly as stored. """ from __future__ import annotations @@ -191,16 +191,23 @@ def test_dict_inside_dataclass_field_reordered(self): class TestArrowNativeKeyValueLists: - def test_unsorted_and_sorted_entries_hash_equal(self): + """An Arrow table is hashed exactly as stored (developer, November 11, 2026). + + A ``list>`` that arrives already in Arrow form is an ordered list + at the Arrow level; Python reads it as a dict, but its stored order is part of its + hash. Canonical order is established only where a Python dict becomes Arrow. + """ + + def test_unsorted_entries_hash_as_stored(self): sorted_entries = [{"key": "a", "value": 1}, {"key": "b", "value": 2}] unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] - assert _arrow_row_hash(sorted_entries) == _arrow_row_hash(unsorted_entries) + assert _arrow_row_hash(sorted_entries) != _arrow_row_hash(unsorted_entries) - def test_arrow_native_matches_python_dict_hash(self): - unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] - assert _arrow_row_hash(unsorted_entries) == _row_hash( - {"a": 1, "b": 2}, dict[str, int] - ) + def test_key_sorted_entries_match_python_dicts_in_any_order(self): + sorted_entries = [{"key": "a", "value": 1}, {"key": "b", "value": 2}] + t = dict[str, int] + assert _arrow_row_hash(sorted_entries) == _row_hash({"b": 2, "a": 1}, t) + assert _arrow_row_hash(sorted_entries) == _row_hash({"a": 1, "b": 2}, t) def test_user_table_is_not_rewritten(self): unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] @@ -209,38 +216,6 @@ def test_user_table_is_not_rewritten(self): data.content_hash() assert source.as_table().column("m").to_pylist()[0] == unsorted_entries - def test_duplicate_keys_are_rejected_naming_the_column(self): - entries = [{"key": "a", "value": 1}, {"key": "a", "value": 2}] - with pytest.raises(Exception, match="(?s)'m'.*duplicate key"): - _arrow_row_hash(entries) - - def test_struct_keys_sort_field_by_field(self): - key_type = pa.struct( - [pa.field("f0", pa.int64()), pa.field("f1", pa.large_string())] - ) - kv_type = pa.large_list( - pa.struct([pa.field("key", key_type), pa.field("value", pa.int64())]) - ) - ordered = [ - {"key": {"f0": 0, "f1": "z"}, "value": 3}, - {"key": {"f0": 1, "f1": "a"}, "value": 2}, - {"key": {"f0": 1, "f1": "b"}, "value": 1}, - ] - - def kv_hash(entries: list[dict[str, Any]]) -> str: - table = pa.table( - { - "k": pa.array(["r"], pa.large_string()), - "m": pa.array([entries], kv_type), - } - ) - src = ArrowTableSource(table, tag_columns=["k"], source_id="struct_keys") - _, data = next(iter(src.iter_data())) - return data.content_hash().to_string() - - assert kv_hash(ordered) == kv_hash(ordered[::-1]) - assert kv_hash(ordered) == kv_hash([ordered[1], ordered[2], ordered[0]]) - def test_ordered_struct_pairs_are_not_sorted(self): pair_type = pa.large_list( pa.struct([pa.field("f0", pa.large_string()), pa.field("f1", pa.int64())]) From 984c524df26d9a80790e0f726cdcff810ccfca3f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:50:20 -0800 Subject: [PATCH 10/22] test(hashing): pin how a v0.2.0 cached dict output is served (ITL-905) The v0.2.0 fixture store gains a pod that returns {'z': n, 'a': n}. A cache hit serves that row as stored (insertion order, old hash); a fresh run gives the key-sorted form and the new hash. Co-Authored-By: Claude Opus 5.5 --- .../data/generate_v0_2_0_dict_store.py | 50 ++++++++++++------ .../data/v0_2_0_dict_store.tar.gz | Bin 4870 -> 6364 bytes tests/test_hashing/test_dict_order_hash.py | 29 +++++++++- 3 files changed, 63 insertions(+), 16 deletions(-) diff --git a/tests/test_hashing/data/generate_v0_2_0_dict_store.py b/tests/test_hashing/data/generate_v0_2_0_dict_store.py index d64fb77f3..e662cc684 100644 --- a/tests/test_hashing/data/generate_v0_2_0_dict_store.py +++ b/tests/test_hashing/data/generate_v0_2_0_dict_store.py @@ -6,14 +6,21 @@ tar -czf v0_2_0_dict_store.tar.gz -C . The process can hang at interpreter exit inside deltalake after printing; the store is -complete once the record count is printed. +complete once the record counts are printed. + +Two pods write to the store: + +* ``total`` takes a dict: ``k=unsorted`` holds ``{"b": 2, "a": 1}`` (stored in that order, + so its v0.2.0 input hash depends on insertion order), ``k=sorted`` holds + ``{"a": 1, "c": 3}`` (already in key order). +* ``tally`` takes an int and returns ``{"z": n, "a": n}``, stored in that order. """ import sys -import orcapod + from orcapod.core.cached_function_pod import CachedFunctionPod -from orcapod.core.function_pod import FunctionPod from orcapod.core.data_function import PythonDataFunction +from orcapod.core.function_pod import FunctionPod from orcapod.databases import DeltaTableDatabase from orcapod.sources import DictSource @@ -22,12 +29,16 @@ def total(m: dict[str, int]) -> int: return sum(m.values()) -out = sys.argv[1] -db = DeltaTableDatabase(out) -pod = CachedFunctionPod( +def tally(n: int) -> dict[str, int]: + return {"z": n, "a": n} + + +db = DeltaTableDatabase(sys.argv[1]) + +total_pod = CachedFunctionPod( FunctionPod(PythonDataFunction(total, output_keys="total")), result_database=db ) -src = DictSource( +dict_inputs = DictSource( data=[ {"k": "unsorted", "m": {"b": 2, "a": 1}}, {"k": "sorted", "m": {"a": 1, "c": 3}}, @@ -36,12 +47,21 @@ def total(m: dict[str, int]) -> int: data_schema={"k": str, "m": dict[str, int]}, source_id="itl905_v020", ) -res = list(pod.process(src).iter_data()) +list(total_pod.process(dict_inputs).iter_data()) + +tally_pod = CachedFunctionPod( + FunctionPod(PythonDataFunction(tally, output_keys="counts")), result_database=db +) +int_inputs = DictSource( + data=[{"k": "r", "n": 5}], + tag_columns=["k"], + data_schema={"k": str, "n": int}, + source_id="itl905_v020_int", +) +for _, out in tally_pod.process(int_inputs).iter_data(): + print("tally output", out.as_dict(), out.content_hash().to_string()) db.flush() -for t, d in res: - print(t.as_dict(), d.as_dict(), d.content_hash().to_string()) -for t, d in src.iter_data(): - print("input", t.as_dict(), d.content_hash().to_string()) -print(pod.record_path) -print(db.get_all_records(pod.record_path).num_rows) -print("version", orcapod.__version__ if hasattr(orcapod, "__version__") else "?") + +for pod in (total_pod, tally_pod): + records = db.get_all_records(pod.record_path) + print(pod.record_path, records.num_rows if records is not None else 0) diff --git a/tests/test_hashing/data/v0_2_0_dict_store.tar.gz b/tests/test_hashing/data/v0_2_0_dict_store.tar.gz index 5a59fd95160a4a5cadf5a3f49cb540a4f1d805f5..e74ae1cad11261d1bbed4a9b07e89b8e65c7e2ea 100644 GIT binary patch literal 6364 zcmV<27$fH&iwFQ@Nc3s|1MOXVY#Ya!Cu#YS#C9x4yQXroqcAaTC!)B!+~q@Qi>%(Z zC0SyuXPmIh-K9i}50wwmQqEVWJp^r<3k=(rxPj|y4sI?Day}<04!$BNazJ3@-2CC| z1wro`7)_ex~{V&A@^dAK+VEO5vuIYa%E=B)wG0a60 zLEbk@w32NMLAc$Fhhf}q(&?rdLL@kncZ&`_5TN}$AqY+n&yytWVEi5@L-QV|Cm{Ms z%E^Hz51rkd%GN*Sq#UlA{+HsalKy82RTEf%{uyWO{a=y`C~9UdF%^xp29gmz5d=3` zx~s%hw*DOy>00Uie|280idSD>|1rVun|j#^SF7x zKj3C;6ybN;Xr2~q9zt;0L=o`m^iYJu?X>YAX=^;f&CJZTg1p~IiV0aXPMXfAf70n7 z$!z`83=n{!nW%Veb*?{g{SUHU605^@Vl91`2{UXq7#jLEfpFuC7u&uUn)}TU-gvVA z?q9q$fEln&SVId)vk6;+t!Z)^c5cc|W7dS!*8@`PnoQuY$!IWvf&^?zDTsQj6@+uK zSTrRG2b9|U(f*^JUq1iv2h(pqwC<1h{%H8amizBJ{^R4{d^!5iLv$#i!$xa)6wGC& z*H&ROhAUOLt$8&FM~c?rY&D87OFt!uBD<+1?%U${?K2;r`QU?dr+&Qa-Q>le{dezA zhOGZWt{u5FHM;fR^h8k}1#_9{%dWx`L+gHba`)G#w{8CJ$K8i7o%{4N>ec^w^53ug z!yZrQ%H2QwUGXXuqmWyLV(GP2z_GiPD%`PiEeKCWrX$f*M2EMva+F!QDWPqIqP%|F zKoMr?M@z?mnfkyVNC*7I?#8yIa~ljB2$a$wr!PObrp}I5l4>Oxy*kTLX60t7c?Aq9 zysm4ildYBxqwAD7bNWups>aO46G9tmID#19lDhU~Q*IiwCPZ7%Srhh;CZ(w7jn|$# zuGopT%615Vk2Y-w-quEtfDGs0+ocsKLeQR`Es}g5U-{lZ%tT#1kJ5-$@%+!b*c3HtIgR+#_y_KCIr#Pu| zQ+{)KDX)JiJ44~(iuDC$XUHi|=>;-(xw1jVUHMf<8XL_9@DB!=8m%B?1X;X-F|b70 z250UnKs0SGRkt}6(HK0PpqwlZFZOj^JPAIBrSzMox>Y4EAq?RvL0Vwj|I@vM)-7bgF5Ghjo8EJ$pD|u+^K$gVUs%F z>3VhC^{l5)He&~I1G+ukkj3_mE!YdI(c?>~Qg3gSQi7UZ-_pBYdfDUr6p$*`WWd&M zH<|D7`C=j;jS0S>AcfTXe8I>}GT{@r1m~OL;!{#M?(@wAe6w84rfDqz*LIHxXf^(+@kW>walDM0E* ze6wOK4l0x3p+U_ zhQ4R&n_AGz4{StJeZFy}2SO z@;0lu{_-~K7weX0qL<<>3Uw_|kqWk}X2UCNm|txw&Vr<6a!QchX3OUZ9NRcd%)ll7xJ z^G!rEYBQ>-E;l`HFZ47I8r- zqD~f)GOz{&jv4V6jMh&!Ti@8+WY(GQ;`wI5eIEJkZN+f>%Nt6})b5?#!|8*}V`d=cBdwwYAn8TdXf*wV7wRGtWLSz>@`j%8oVTFYUs> zh)>;g(!IN?^}M~+dU9)%S!e!<=bM!(t>+CpXg%*d2wD$*b}MeOuESs1)&Pf~=Jl-C zyKl5j8daK4t+&IAPIa*?JH$%YUY1R<9h2-H7zBP**c;upygsgQF~J$X=e&%O29u7rT#R%_IbD6?&jfbC##el zoo3V5Bn;n+_>-TdYd2(_WxGd)hf)&NG18OrQ{I5ki^#L%gY$O#Ayn0VIa7Q4P%2G% z7{;dv7#!d!56OoILSqi^tkBaPhw;-z@=v1tDU^Sl@+5_x5nAZ>#3sfh_ERYTq?+k= zb}E%-9@M{2ifT{Olr_^0{|G%EVtf>)@1*U6c6*ORRZiH|e^B~OOVx$SJLu8MlblS8 zu5&1zzH887x6jYR^uyWdhn4yQRib!_pKy0v7==mOr|ki|y*rzq?sR^(eHg~0sGdYf z90E-PToa>*LjKW_qyR(@pP{}cLY|P=Gjec@0QSGQrwIg|s z#3x3{kUuiScBA(13Gh7=fr+uH&^Rf{QW*2j@*&TRKRhwZ?nV6N*Y6$_Kc=S-AWIUo z!8!0wFnrH2>nOtSJc{qi=)*BF1^9??W8MJQ(>>2g61Qvl8Ord(qIglo4>dlvComBi zNsc?bK$nd2t|h^{79AfOhytQR(6r^qzf08~Ss0HrC(8@8<+(9xg7k+62yPU%m4TTF zikT96N9HC*6uL-0I{d2Sw;RQe7UI_*?oLb$LHXFV@?$gdn?>=m{J?v__l^)_ly?f) zBbEG+aTtPj7+6*G@^{EqBgZ~;b!CLw1(QftCNAq|PYB3yZlc>WGr_W7HJBUaPs6rY zyjdRbH8V^r(Mo_>CzzU+_0RCjP0~k9L+x6~Rq*@2PY^?3E*^?bE(!`#KK_HD>6MQE zS|Q*6OT~YXyhC*RMW>CX0|6WOAF#PanzPX^0#t(&-9ms|DE@3puzJ2_ef4XD3{%=3chyPHgF>6A~#((U%=jo{z_r3qXlMSzY?KiK!bN;RG8`uB! zhmXi^c`Z= zR971hkAUsIv|ge5E}7Z zSs)K&K1xP-`~((osVsyCDxeH_$0bzX zL+JA-qLhJiu+@We4C}x#x>cV~hl;>C2<_)rR-KjbRl$Ar9kn z;RzEu@C&uPMqs=eo&W=TLd3)|-%bQqNM+Tp6Me{+L^r@H>Dev_ z-{{wYKd9xg8^JXYJcb${Jw(D!{%rR_E&M`Vzb+JyqBv@Vk(g|s0c|fKl>nPkHoOJN z(lJ8#9eV=YC=>RlI;M5tCo27R45RqMa1lZrYY(vl^M(2CNAX^jJ|wsU6(Yu{sVSa9 z^3sE2>{qon1jp!|Wb?x@sDW93#2W(t)6j=u9r#37O8x$W_&Ju%fuDqhQRbk~6M}68 zv=bpYKIR3sVI%{ofb!AutCn9EiZ947Gv$ZSsQ!5^T*TX-kzXH*SL8S2kF=A(#!LgA zrquF-#sOG~eZVx!CT3Ey8tvKkq3cjartLD!gGI11M1KynmsvgxWH>s+RvO}QsRr?= z+Ept4gV0`N>%n+5QbHMpzyA{s7oF?>fk6JhYTy5>|NZ|73nvJcwkIqzTw==7W~qS7 z4@(Q85`>Lu1WuCTfd1hYOFTF)TH0I=r>Dgdjtap*kmq2U{lPE@CtV)e;h{;7)9rD! zSm63l#}pTt1W}1tQUuu(Tmn$CJaNDhNrt^>r7C^^WNbNL2?ynpz%z)?nwo&11D0h7 z;XYsij4MNW#3V&fP8&&rfBQ+IjrO!L&Q>=K7$YW>#5zk-nfjqbN}XwS24(33rIlAW zG9%p#1D3;3UEHB8b(O(pA=3G3Ur1afR_zvU zP=&P*)fGvGLLJd$B%xB5ZZ1MAP$HWrB2TL%3M-k=(E*`SH&IC4#NorhBqg{>U_D?} zS}cApE_$P>-GUMz>l%c0M8n}=VnixgiRMF~mk?vY$mC!wIvW(km=tsP@a<1nc$8yz zBoKvWGy#kmuvFn0jYW&jh%sdEfDP;K^7eFDRAGZ$3@{7IfVR>Si6;8F1eksc?Q@Ty zS`WrV0h&u_OD&H+2C|05*kmRcgo18tQcSd}7V(?E+APUT9SE~Zw6!!diuFO7UWUA zrS=3>2gT?MgFpr(1?JutO(x)HgIF{bhbA9|$qgbaOp<~x5DbZNAL!adK|d-chNH7E zlF*5a^IRm7FK2u@I5Q&(J|PJk1WKDr1#$%5gPEYV02Bq}19Yt)CrMUYIEqZpfPPWV z7!y;mU_#7@igbWZ5^$3V0WJSI77EMI?Xb8T5 zU>xNI@2MC|zp0=Rg1}MO(lUV*GJL>Zah+h81~US?(q;*GIL;&RU{L5Hc^e&Y_-!6h z6l@NGBSfd@Al(GC1kfyJ3~!u|1!oc}V;hLZfPq3o(zLO)WQLYelo$;u;g}MwQ9hD@ zuc9$Fl^+d?mG|^LujlBEm?XHpfZZm*g?tZ4@S7y914RhasChtN2AaKsMc_B zdO+eJn_$kj%}SBg@7P+hD+?r03EJ78BCAyc^@qa>mysD&R7^m6gPkf7haZ{L#MLdD zi^VueGVBz4mtpRxvCUawL}W=|<9)L`Vfj+$C}R#2kw)k`QH(NHoBJG@69 z10!uPj0OY1+$9qiLA5}21%i`FWs(61!CpG=^3Fnc^b}j(p^3uFJ1!@)*yWva)3ol)=Z{RNhHfOUimHv52_&=ejnp_I5B;-aKQ0kpCq zEl8vn3Z$^C)sVEIj;Gfeh4}P$K6dJ-`gQh=9}O00Rma?OOc#FEq8n_kXXDt4jJ` z{P{1n^M97)TBhf}to-x;s`Fx1y!y-b{Fhtc{GS!Ce>E5+KrtQ{>9kQ|z-6Na#$n?e zjA#phnGQ)X4Cx@K1^WMRQIs?1`w#896~6zh1pOC0|K&%Ce}47#*T&Xf@`m%C|DsA` z)`XPp|Ks`huU-B5tH1yCufB10)9<&w{^G;mU-yF#_I^3|`PH6P_=zifi~Ij5Mj>~z zSh4h_`2T>4*7g4ZBh1oI+0HYLefRIa^1VGf{wq4$`E}c~_x|}Go*at(mcJ+be|Oto z_|=VLuM~HlQH(zYzq_!#^Czxec?lAomM0tHyb=kwfg= z>4y1Nnc&~LRmN2sFAb|@>j$VbVa-`o)+ur3^c|wPkyg_*)Gduqk#i>OeA=hTN#nKW zPbl`Lt+E|UaqJn#xM-f_ct7PQ88_h)2!?VBZEZYUI*jWyofF38$f$XFvQ@;bi1o;8 zdEJUAapv@0{-b16eX~wbVYed1ddjjhXxxfGy(qth{d^Sbs>;q#xVU27O4%81O}8RO z4*Z`-gg~6Bu^ZY-*&&BwOjWnK3Cs39vdEWt@gy9%zDJbTP<6XfeUDt*j2%EvbtvvG zY*zL}@>5LhiR5S_c3Sd8@-nL7HB!5=(L~<*5B4G^XHF8&9CaJnKsUj{+HY%SA zrE*I2VNzf2K{dr@tQj9BA40bSZ{Jf1A0}0pmS$XE+<@nsoNZo|50mVvYQ{gsDY*Op z8A}y>m{ehUni0u)cm~A|R`!K1(j5Ndbdywlh_gBM*$>&gy-?CQ4hsjYpp#7G$ zdOl2Eg$3Nm_0UmKA0}_ZB3|E7Js&0?!2+)AsFDwpBLtxS^3JOHFgXS3zn@EA?h!Qn z*2!(>8ggl(nw$6Gb*cd$cugVMqq{Jt+S7%WY?{$J{In5&$!NWPm-U^yn#?+rbv)k` zzHgVh;CBfO$1m){ye0Wf`6*`otf5@LDOX|LPCk(1HzlW&Lxo&ji+{1!dbH7c9UIh5*V7wtNw1G!ESAeqj|E_BjOie+PPhkox+CrSjK129`>szW)eLDzN4 z?s-?WT&EC&vjtsOq+oms)<@+!g&LR@Mj3)n6?dKDM)?=+Iwf7-_EBD{ce;KAJo!dC z$tTBE2RgGv5`{M{kM?nxermBN@1#$<>2O1O1zbbKVZ7?38|0;d8}o+z5h@EKlYF$Uaw>L`&+tM{@=@1MbCM4X zRJ+Pq=t(|m{Vwt(A2q)VKFLSz>ZEcf`RuRSNj_@%E%GEEwJVwmpX9T@awqxJm+R&Q z&h?4SHBT=~doA1Ze^xsFZ-wALmc#yUhg0kS$5F%oEQ$Urx&PZEQk>xCoi;xs@HW~R zAZ&iZ#o2hl9}sCVzyw@mZU1*||95TwcWwW7ZU6V5hI*t$x)m!jy}jy-5!lV$=dM-8 zO55n}S-?j3g+kYhY;<=!ifnZE5ROW1bkD!VH%GHj14k%BD1yMrQZF!q2J?b6=sd3|hW=3$8CGPO;XjP6 zD7vjMnvVt9kZoyE1a07Xd*Ao&$UENg%b{#*<3T3x-sgRv=Xsy!d7t;=cc#;2+V%^_ zaUVquqi?SleS=H+mL3R-^ijBn^pMmrPLL$-HVsqTubRnJDvFE%RN*(uOlFN2>eVZ! zcs|pBkG508omX^wE~iV(i%e!)qo95NyWOO>>;KzvDgH0=Stci@+1NVnBt|J5_xq^; zMfv@N*YBZl9%l&F&%4<~!V_n4j`IdsmLNQCDjx7s9yZ_&B=|T%dKvH(@My!SZU2*A z(nECpe><*i^8Y%n3V~bjKjrO?|J!l_M-?{2R6gfSlyaeE<|86k<<2&5{ z+cU5_-hE^L3tT+5PIM{^+V{WPO;KI{-*NvJ7(sNv?hpwgbLkYN7DJKkG^ z>;Jj`7w8ay^P(!d-4^WIhb>%xCwCafPD907Iu zvEA72A+PCJe{CJBu4G^Uu<9E!gHLmh$qWQ>SV|E@{@4l189~UGCF208!|$JZKJc?| zJ@L-k+mHAC=Al1ddUxdW55Mr87yjt={Ns;%GNOSR&h|K1+pMqe!U4>pbm89NU7#E} z+CZ~ih+vhzS}=t+RK$l5y8rE)A6|Utoy!-#^XN}XSAO&#)8AjT|2?s1`C2M`_#cf- z(H;kDoAul7!r8^XU!I@&{k0@&4G|BbbudeH}+ewg4f->r}OtLCBw_8`BkfLkfP_LY|3>58CXx7S| zUR7P1D~jAG3LK74FeStM(qCK0sw>e8x@5-w+N_BBR?o*Ty`bcY(auf?K#x)V1U@>7 z6Mzjva$KDC`6z;5m_!0FW!<#u*A%mKE(WPyH^I(v>TeIj}Bm`EhcpP z%AiK=n+LI%ccI7EP^aELEY$=(y>W1QuQb?;YzkNv8!};gkD9Ib$6^A{<^?X6=A@E= zSS+0@l*Aavh)gWS6jM^UB^E0rV(W|$Q_^KD$7K1Vrr9AV0#+H;S1G2GIYulAyo%Ei zeKj_p6ZxE|sx(lkluS$IYDotXl|RF?=%Xa&g_yWeP_&_pYov3DJjlemXMYdY>9zJ| z(nS!wu=*Z8!)Jj8Sg$1Do;7|$EpXR5lPPIBYpn{9rjgh>FBCy%syYm+br1!na|-mr z5;R66>R!o$XT(@dvwg8*z9g`G3{_S)?*q!^xtLfyIyn2yfuRvJ_(6^sI&`DoZW_Y& z9->UU`wwX%1T$OAmMcA$f9SC<*!x=F2+mrD(Lgg<&W&JTj2yudCbbNG%pplP&6a2H zv0pda|GDS(?Dl3YtFoS#MzAB6^Pj`6_h=NYGYF8jU+b|v-(x?w%l?aAqaEO^!yQ8Ws+a-wtu|q_U!tqCixJW~ulno@Cd<#Ej1P}u zp7v!(*{CAr;sf?)AE+bcH&Dt2yH-m5?5T3Y%l-EEAE>txt-9ewClGMWu64t-VeYMO zr&k8-&kWS*!DT4m+#@x1YA&G0NH+%T*XxP63Prr7i0GA#q%5ofgL6HWmwW6V9kBoK zz>w8oziX+t3!V+gqt}U9ET2AtO={ernKx9~{gpxcPk*QN9dI?j-*kfE_U5sg{H~pa zkev7TSU%okzj@I9I@aBJw!8D31q(dc;1^uju;sN!F|guOhfZa7S4Gd;hwbMN4_OWN zpO$*NQYCtR=mODm^)V1VmP>~%W_zFIjU$6_32NO-he9XAqtdFB;4# z#j?aTLIlcjGL+z^5qo-NVbkS0gSt8=Hwp$9%N5Q`kY2`Nae^fS1e=}DMBJfuZfdd! z>1$2eCsF${YQI7TO5D`4hnoxt(TJpe8MRNUl@8LWaz%O2|5=F?tnidQGYLPqsQ|tb zhV{oit_7ECO5!S4oR~W;eOIjNL-I}s^!6l@71NC`R_aeIxLvNzO;~?PTYpLEFK{KI zOZsFdi(D914X(KoF4v@1pUFylI=BSsh%_Z;_(c#J;EIONWa8oF5(i8VpP|2^nLvi0 zT0R{if$gS`ry^vg6wUe(mm<`!f&KG{j_j9+xPw{9BO0Dhu+tfwnF^F5?r4V1%@^V% zidkSErGAy)_)M8r9Dw5M4zJXwPdx_emtp<-_MnrH9)w+) zj;7+%%NbZ0Uhp;S-;wgrPpV%st=9myhKu& z1MIjFoeUJBH2npW^^gn=JA~q}yx<$AC@1d3!LAc*P22i2{j6UCF-_g;R<2g@f09kG zZpIyO_~W?G;b8)9N5BW}eDOHv^ZDa!g7Due{!db#TKu1c?|*jj|2vHT|C#uU^Y8Jm zW=ntbP+k0ARmZ9;X&e84SF#1&04vukNK-6 zQG=q!n18b=t?dR~%>Vxy=C7nk)h%M%CfpwS${b5z>4{D(P z%U!6yRj9u=E%MnrhV!=}z4xd|1@fEjG|1m(x+ zDzLBL$Xe=dZ!K>^NdMhOurD@&^ncW2c?;yMLHm{VAblX>r%?TGAH$w*4bK1Ip#Awf z3g_?4O{EIX|CJLPw%k0{3Y>rWu>Bbb=j*qYmU0q*?g9fZ%)&gpOShMVc*3&49!+2>wExs&GENeQ^Gg0UST6 zg3lJ4!1=fyexudx*(PxQ1ca@nPAFLnM%CZbi!)tb2|N22l1qYu%bksEsJnarr+*E2ko(YiA$ni}G;nI?QD)p=U z!V*8!74!~7683i-!PR9=|HcuJE17PdR_fCe5Wb%?fWOuDm_cwodOd<%37;V#jF3*31;Z|&Bpr9A$myV4+-ujGkAndrC1W#%LtA?rHVHM$4@8e`fxlszaGzpGT>hY z`d%`C&nL?2@ek-5G+p!hL6!?sr@5&Nj1>?kTxlf|0=Z$i3aLZ;==D|GZvxR9>Pw~K z@YR^PP5tWvp}DI4W)WSnUm>0g5+KK{0i9B6`@!S@vcyS{n&lLel(I%pn;$05RBalR zVSXCDR)qL(fOuJFv%rSoMY_`v|Ftf}zin4T{6EI=8L&;yWrECbDR2El3!;w{6HZ4m6&w#Ew@&U!GHu+>{ z)ixs6FgTh(73C9%YDL{KAgYG^D^*hMwh^gzFO6!+^+!D#`I@66)EJ!+DZ2(kU_&Y`5qi*jgPB}dukXWgxR9U?ruPS(Vy|=z<+k)<5&S>0qE9Ps%FfIFfXIQYE z@M$4-8;=#}u3RaT8OxV)qH1&*3di+h0oQs9S2+?lFge~xZM?BF@jAW1x3)k^5}72( z92Fk|r*=$-|mY6DJHUnk}UP$MX3qpQ9&GCXn+1k4Atc^u2 zW^#!R*io5`Pw*{MJjoKgrFC?pCKq;6Olf39u zfkMMy7?#|nE(8%`6L^NJm<)&ZO7pp@5p8H00rmpU!iIwYy8u$%^*LS?(k$w?6i?9O z(2TJxC?v=cnC@fwk_cfVAzy}tQkfh(nlef2SR$R_i!m^@@tiS9;IsL4NW=|@MV86s z>a{Gcr3(d~i*Y3w5U6e~7pM^!2Zgj=03rc<0ACx^ByZ`3BVwrl=0&-oz?X%z$X87T zECrRR7J$ItYBv9WhV>ubq4l2+_xN<{KR4c8|J|PTUqQ%=c{ZPc0}!mELSQWOLS^iN zAOwYB7-`i)5+y#6ub6SLOoI(U9vHPH{5WiKA)eb0LjM@ zFmo|*y{9DJA}gc|qAJ-Y@&ZUwXh~X0wzlfhGLI^wB_&)_qBH6zMK~0twP9r*EP`bt zD=i%oX+DG2iZDA?=0H^vlNK0C%Xm7+2pgyjs>H-IupF&IS5dFw;&fHgK+CWO+wK%w z>}(Tr2r{aft0S5b1V*xq{GW?Y>FYF& zdbBL-gz^agyk%BT!G$G^0tQTY^PsX^aqHG?Si2zXw1*T7xa*!a)TnMWR2vk%5(y?# zY~~8h#-e$@T4AZ(eRL>bW;FHjPB=7ks=@IN$d2%Uxt=UfrvE?Xc^A0o-Kp1ORRT0At0gxc~qF diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index c5e4f1a6b..ae68b7ab1 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -78,6 +78,10 @@ def total(m: dict[str, int]) -> int: return sum(m.values()) +def tally(n: int) -> dict[str, int]: + return {"z": n, "a": n} + + def echo(m: dict[str, int]) -> dict[str, int]: return m @@ -343,7 +347,8 @@ class TestV020Store: ``data/v0_2_0_dict_store.tar.gz`` was produced by v0.2.0 running ``total`` over two inputs: ``k=unsorted`` holding ``{"b": 2, "a": 1}`` (stored in that order, so its v0.2.0 input hash is order dependent) and ``k=sorted`` holding ``{"a": 1, "c": 3}`` - (already in key order, so its hash does not move). ``data/generate_v0_2_0_dict_store.py`` + (already in key order, so its hash does not move); and ``tally`` over ``n=5``, + whose output ``{"z": 5, "a": 5}`` is stored in insertion order. ``data/generate_v0_2_0_dict_store.py`` regenerates it from a checkout of the ``v0.2.0`` tag. """ @@ -393,3 +398,25 @@ def test_unsorted_row_recomputes_once_and_old_rows_stay(self, store): out.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False for out in again.values() ) + + def test_cached_unsorted_dict_output_is_served_as_stored(self, store): + # A v0.2.0 row is read back as stored: an Arrow table is never re-sorted, so a + # cache hit returns the old insertion order and hash until it is recomputed. + inputs = DictSource( + data=[{"k": "r", "n": 5}], + tag_columns=["k"], + data_schema={"k": str, "n": int}, + source_id="itl905_v020_int", + ) + pod = CachedFunctionPod(_pod(tally, "counts"), result_database=store) + [(_, cached)] = list(pod.process(inputs).iter_data()) + fresh_pod = CachedFunctionPod( + _pod(tally, "counts"), result_database=InMemoryArrowDatabase() + ) + [(_, fresh)] = list(fresh_pod.process(inputs).iter_data()) + + assert cached.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False + assert list(cached.as_dict()["counts"]) == ["z", "a"] + assert list(fresh.as_dict()["counts"]) == ["a", "z"] + assert cached.as_dict() == fresh.as_dict() + assert cached.content_hash() != fresh.content_hash() From 44e7aa87ddc0b12cdfcb1cbfb9bd4f54e2ea3180 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:51:37 -0800 Subject: [PATCH 11/22] docs(hashing): dict and set canonical order, changelog and design issue (ITL-905) Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 37 ++++++++++++++++++++++++ DESIGN_ISSUES.md | 34 ++++++++++++++++++++++ docs/reference/hashing.md | 61 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8053fbb77..8df62f61e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,42 @@ # Changelog +## [0.3.0] — unreleased + +### Breaking changes and migration + +#### A `dict` or `set` value hashes the same whatever order it was built in (ITL-905) + +A `dict[K, V]` value is stored as `large_list>` and a plain `set[T]` +(element type without a logical type) as `large_list`. Both were stored in Python +iteration order, so `{"a": 1, "b": 2}` and `{"b": 2, "a": 1}` had different row hashes, +and the same `set[str]` had a different row hash in every process (`PYTHONHASHSEED`). +Orcapod now stores dict entries sorted by key and set elements sorted, wherever a Python +value becomes Arrow; see the hashing reference, section 1c, for the order. A NaN key or +element, values that cannot be compared, and two keys that convert to the same stored +value raise `CanonicalOrderError` (a `ValueError`) naming the column. A datagram built from +Python values holds its dicts in the same order, so fresh and cached pod outputs iterate +alike. An Arrow table is still hashed exactly as stored: a key/value list that arrives in +Arrow form is never re-sorted. + +Which released hashes move: only those of values holding a dict whose insertion order was +not key order, or a `set[T]` of two or more elements whose iteration order was not sorted +(for `str` and `bytes` that order changed with every process, so it had no stable value). +For such a row, its data hash moves, and with it the function pod cache key, the pipeline +record and entry ids, and the default `source_id` (with its per-row record ids) of a +source built without an explicit `source_id`. Schema hashes, `pipeline_hash` values and +explicit `source_id`s do not move, and a dict already built in key order is stored +byte-for-byte as before. In the golden hash suite exactly two entries moved: +`pipeline:index_pick.index.row[{"subject":"s1"}].data_hash` and `...s2...`, whose rows hold +`meta={"k": ..., "j": ...}`. + +What a `0.2.0` store sees (checked against a Delta store written by `0.2.0`): table paths +are unchanged, so every stored row is still found. An input holding an out-of-order dict +gets a new hash, misses the cache once, and is recomputed; the new row is written next to +the old one, which stays (a store-wide read shows both, nothing is dropped). A cache hit +whose stored output holds an out-of-order dict is served as stored, with its `0.2.0` order +and hash, until that entry is recomputed. To start clean, drop the old rows or use a new +store; Orcapod does not rewrite stored rows. + ## [0.2.0] — 2026-11-10 **Distribution:** Orcapod is MIT licensed starting with `0.2.0` (it was proprietary on diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 4be0e7e34..bdfabb8f1 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2245,3 +2245,37 @@ xfail. The async path checks key types on the first row of each input, because So when either input is empty (left or right), a key type mismatch that the sync path reports yields an empty result on the async path instead of an error. `Join`'s async key check (`_check_shared_key_types`) has the same limit. + +--- + +## `src/orcapod/semantic_types/canonical_order.py` — dict and set values in canonical order + +### DO1 — A `dict` value's insertion order, or a `set`'s iteration order, changed its row hash + +**Status:** resolved +**Severity:** high +**Issue:** ITL-905 + +A `dict[K, V]` value is stored as `large_list>` and a plain `set[T]` as +`large_list`. The converter wrote both in Python iteration order, so `{"a": 1, "b": 2}` +and `{"b": 2, "a": 1}` had different row hashes (and missed each other's function pod cache +entries), while the semantic hasher, which sorts dict keys, gave them the same hash. A +`set[str]` was stored in an order that depends on `PYTHONHASHSEED`, so its row hash changed +from process to process. `set[]` already sorted (`ListLogicalType`). + +**Fix:** the developer ruled (November 11, 2026) that a dict and a set are order independent, +that Orcapod sorts them before storing them into Arrow, and that hashing an Arrow table never +sorts anything (starfix ignores only column order and struct field order). The converter's +`dict` branch sorts entries by storage key and a new `set` branch sorts elements, both with +the one ordering rule in `semantic_types/canonical_order.py` (None first; numeric; `str` by +code point; `bytes` bytewise; dates and times naturally). NaN, incomparable values and two +keys or elements with the same storage value raise `CanonicalOrderError`, wrapped with the +column name by the caller. `Datagram` built from Python values holds key-ordered copies of +its dict values (inside `list`, `dict`, `Optional`), so fresh and cached outputs iterate +alike; the caller's dict is never mutated. Released hashes move only for values not already +in canonical order (two golden entries, `pipeline:index_pick.index` row data hashes); see +the `0.3.0` changelog. Regressions: `tests/test_hashing/test_dict_order_hash.py` (including a +Delta store written by `0.2.0`) and `tests/test_hashing/test_set_order_hash.py`. Open: a +key/value list that arrives already in Arrow form keeps its stored order and hash, by the +same ruling; dicts inside dataclass or pydantic objects keep their own order in memory +(their stored and hashed form is canonical). diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 886b1580a..327a62b5a 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -242,6 +242,67 @@ The pipeline hash uses the same `SemanticAwarePythonHasher` with the **`pipeline --- +### 1c. `dict` and `set` values: canonical order + +A `dict` and a `set` have no order, so equal values must hash equally whatever order they +were built in. Arrow stores both as lists: `dict[K, V]` as `large_list>` +and a plain `set[T]` as `large_list`. Orcapod puts them in canonical order at the point +where a Python value becomes Arrow, so the stored list, and therefore the row hash, does not +depend on insertion order (dicts) or on `PYTHONHASHSEED` (sets of `str` or `bytes`). + +```python +from orcapod.sources import DictSource + + +def dict_row_hash(value): + source = DictSource( + data=[{"k": "r", "m": value}], + tag_columns=["k"], + data_schema={"k": str, "m": dict[str, int]}, + ) + _, data = next(iter(source.iter_data())) + return data.content_hash() + + +print(dict_row_hash({"a": 1, "b": 2}) == dict_row_hash({"b": 2, "a": 1})) +# True +``` + +**Where the order is applied.** In the type converter (`get_python_to_arrow_converter`), +which every Python-to-Arrow path uses: sources built from Python values, datagrams, function +pod outputs (sync and async), and dicts nested in lists, dicts or dataclass and pydantic +fields. A datagram built from Python values also holds its dict values (including dicts +inside lists, dicts and `Optional`) as key-ordered copies, so a fresh pod output and the +same output read back from a cache iterate in the same order. The caller's dict is never +modified; dicts inside dataclass or pydantic objects keep their own order in memory. + +**The order** is defined on the converted (storage) key or element, in +`semantic_types/canonical_order.py`: + +| Key or element type | Order | +|---|---| +| `None` (for `K \| None`) | first | +| `bool` | `False` before `True` | +| `int`, `float`, `Decimal` | numeric | +| `str` | Unicode code point, never locale | +| `bytes` | unsigned bytewise | +| `date`, `datetime`, `time`, `timedelta` | natural order; datetimes by instant (they are timezone aware after conversion) | + +A NaN key or element has no order and raises `CanonicalOrderError` (a `ValueError`) naming +the column, as do keys or elements that cannot be compared and two keys or elements that +convert to the same stored value. A dict or set already built in this order is stored +byte-for-byte as before this rule existed, so its hash did not change. + +**Arrow tables are hashed exactly as stored.** `StarfixArrowHasher` never sorts list +contents. A `list>` column that arrives already in Arrow form (an +`ArrowTableSource`, a Delta table, rows read back from a store) is read by Python as a +`dict`, but at the Arrow level it is an ordered list and its stored order is part of its +hash. Rows written by `0.2.0` from dicts that were not built in key order therefore keep +their old hash when read back; see the `0.3.0` changelog for what that means for an +existing store. A `list[T]` value is never sorted: its order is part of its identity. + +--- + ## 2. Source Provenance & System Tags When a source is built by `SourceStreamBuilder`, four hash-related operations happen in From f63602cc925faeb61e595d3e72d9583361899a8a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:51:54 -0800 Subject: [PATCH 12/22] docs(plan): record ITL-905 rulings and outcome Co-Authored-By: Claude Opus 5.5 --- .../2026-11-11-itl-905-dict-order-hash.md | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md index 155481b9f..496a69aff 100644 --- a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md +++ b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md @@ -8,7 +8,16 @@ **Tech Stack:** Python, PyArrow, starfix 0.4 (unchanged), pytest via `uv run`. -**Status:** STEP 1 (plan + migration assessment) — awaiting coordinator approval. No library code yet. +**Status (November 11, 2026, final design):** implemented on the branch; see "Outcome" at +the end, which supersedes sections 2, 4 and 5 where they differ. + +> **Developer rulings (November 11, 2026):** Orcapod sorts dict and set values before storing +> them into Arrow; this is not starfix's concern. Hashing an Arrow table NEVER sorts anything +> (starfix ignores only column order and struct field order): a `list>` +> that arrives in Arrow form is an ordered list and its order is part of its hash. So fix 2 +> (hash-time sorting) was implemented and then REMOVED. Plain `set[T]` was folded in (separate +> commit). In-memory canonical copy (fix 3) and raising on NaN/duplicate keys are kept, +> provisional pending the developer. --- @@ -195,3 +204,27 @@ def sort_dict_entries(entries: list[dict], column: str) -> list[dict]: 2. Approve fix 3 (in-memory canonical copy of dict values in plain containers; dataclass/pydantic internals left as given)? 3. NaN keys and duplicate keys raise (including when hashing a v0.2.0 stored row that has duplicates, only possible via Arrow-native input) — OK? 4. The `set[T]` per-process hash bug: separate issue (recommended), or fold into this PR (same mechanism: sort plain `set[T]` on write)? + +--- + +## Outcome (as implemented) + +Commits on `eywalker/itl-905-a-dict-values-insertion-order-changes-its`: +1. tests first (failing) incl. a Delta store written by v0.2.0 (`tests/test_hashing/data/`). +2. converter sorts dict entries (`semantic_types/canonical_order.py`, `CanonicalOrderError`). +3. hash-time sort (later reverted by ruling). +4. `Datagram` in-memory canonical copy. +5. golden: re-pin `pipeline:index_pick.index.row[s1|s2].data_hash` (approved), then additions only + (`arrow:dict_order`, `datagram:dict_order`). +6. plain `set[T]` sorted at conversion (+ module/error rename). +7. revert hash-time sort; Arrow-native unsorted vs sorted pinned as DIFFERENT. +8. v0.2.0 fixture extended with a dict-returning pod: cache hit serves the stored order/hash. +9. docs (hashing reference 1c), CHANGELOG `[0.3.0]`, DESIGN_ISSUES DO1. + +Correction to section 5: the golden suite DID hold one out-of-order dict +(`pipeline:index_pick`, `meta={"k":..,"j":..}`); my first survey truncated grep output. An AST +walk over every dict literal in `cases.py` found no other dict reaching Arrow out of key order +(`datagram:nested` is key-sorted; `semantic:*` dicts go to the semantic hasher only). + +Performance: the Arrow hasher is byte-for-byte unchanged from `main`, so hashing tables with no +dict columns costs nothing extra. Conversion adds one sort per dict or set value. From 2994073e7736e3fe5c9fa5145c2939895c86741f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 02:58:21 -0800 Subject: [PATCH 13/22] refactor(semantic_types): type the canonical order helpers; fix new type errors in tests (ITL-905) Use object and a TypeVar instead of Any in canonical_order.py, build test pods through the public function_pod decorator, and narrow DataValue in test assertions. basedpyright reports no errors. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/semantic_types/canonical_order.py | 47 +++++++++++-------- .../data/generate_v0_2_0_dict_store.py | 8 +++- tests/test_hashing/test_dict_order_hash.py | 38 +++++++++------ 3 files changed, 58 insertions(+), 35 deletions(-) diff --git a/src/orcapod/semantic_types/canonical_order.py b/src/orcapod/semantic_types/canonical_order.py index 9525fbe3d..2a6ed2f35 100644 --- a/src/orcapod/semantic_types/canonical_order.py +++ b/src/orcapod/semantic_types/canonical_order.py @@ -26,12 +26,17 @@ import types import typing from collections.abc import Callable -from typing import Any, get_args, get_origin +from typing import Any, TypeVar, cast, get_args, get_origin from orcapod.errors import CanonicalOrderError +_T = TypeVar("_T") -def _key_rank(key: Any) -> tuple[Any, ...]: +# A sort key: ``(0,)`` for None, ``(1, value)`` otherwise. +_Rank = tuple[Any, ...] + + +def _key_rank(key: object) -> _Rank: """Return the sort key of one storage key or set element. Raises: @@ -44,7 +49,7 @@ def _key_rank(key: Any) -> tuple[Any, ...]: return (1, key) -def _sorted_by_rank(items: list[tuple[tuple[Any, ...], Any]]) -> list[Any]: +def _sorted_by_rank(items: list[tuple[_Rank, _T]]) -> list[_T]: """Sort ``(rank, payload)`` pairs by rank and return the payloads. Raises: @@ -60,7 +65,7 @@ def _sorted_by_rank(items: list[tuple[tuple[Any, ...], Any]]) -> list[Any]: return [payload for _, payload in items] -def sort_dict_entries(entries: list[Any]) -> list[Any]: +def sort_dict_entries(entries: list[dict[str, _T]]) -> list[dict[str, _T]]: """Return converted entries ``{"key": ..., "value": ...}`` in canonical key order. Args: @@ -76,7 +81,7 @@ def sort_dict_entries(entries: list[Any]) -> list[Any]: return _sorted_by_rank([(_key_rank(entry["key"]), entry) for entry in entries]) -def sort_set_elements(elements: list[Any]) -> list[Any]: +def sort_set_elements(elements: list[_T]) -> list[_T]: """Return the storage elements of one set value in canonical order. Args: @@ -93,10 +98,10 @@ def sort_set_elements(elements: list[Any]) -> list[Any]: def canonical_python_value( - value: Any, - python_type: Any, - key_converter: Callable[[Any], Callable[[Any], Any]], -) -> Any: + value: _T, + python_type: object, + key_converter: Callable[[Any], Callable[[Any], object]], +) -> _T: """Return ``value`` with every dict in it in canonical key order. Walks ``python_type`` through ``dict``, ``list`` and ``Optional``. A dict whose order @@ -118,32 +123,36 @@ def canonical_python_value( cannot be ordered, or two keys that convert to the same storage key. """ if value is None: - return None - origin = get_origin(python_type) - args = get_args(python_type) - if origin is typing.Union or origin is types.UnionType: + return value + origin = cast(object, get_origin(python_type)) + args = cast(tuple[object, ...], get_args(python_type)) + if origin is typing.Union or origin is types.UnionType: # pyright: ignore[reportDeprecated] -- detects Optional[T] non_none = [a for a in args if a is not type(None)] if len(non_none) != 1: return value return canonical_python_value(value, non_none[0], key_converter) if origin is dict and len(args) == 2 and isinstance(value, dict): + original = cast(dict[object, object], value) to_storage = key_converter(args[0]) items = [ ( _key_rank(to_storage(k)), (k, canonical_python_value(v, args[1], key_converter)), ) - for k, v in value.items() + for k, v in original.items() ] ordered = _sorted_by_rank(items) unchanged = all( new_k is old_k and new_v is old_v - for (new_k, new_v), (old_k, old_v) in zip(ordered, value.items()) + for (new_k, new_v), (old_k, old_v) in zip(ordered, original.items()) ) - return value if unchanged else dict(ordered) + return value if unchanged else cast(_T, dict(ordered)) if origin is list and len(args) == 1 and isinstance(value, list): - new_items = [canonical_python_value(v, args[0], key_converter) for v in value] - if all(new is old for new, old in zip(new_items, value)): + original_items = cast(list[object], value) + new_items = [ + canonical_python_value(v, args[0], key_converter) for v in original_items + ] + if all(new is old for new, old in zip(new_items, original_items)): return value - return new_items + return cast(_T, new_items) return value diff --git a/tests/test_hashing/data/generate_v0_2_0_dict_store.py b/tests/test_hashing/data/generate_v0_2_0_dict_store.py index e662cc684..d8b3bef28 100644 --- a/tests/test_hashing/data/generate_v0_2_0_dict_store.py +++ b/tests/test_hashing/data/generate_v0_2_0_dict_store.py @@ -36,7 +36,9 @@ def tally(n: int) -> dict[str, int]: db = DeltaTableDatabase(sys.argv[1]) total_pod = CachedFunctionPod( - FunctionPod(PythonDataFunction(total, output_keys="total")), result_database=db + # The fixture is built with the released v0.2.0 API, unchanged. + FunctionPod(PythonDataFunction(total, output_keys="total")), # pyright: ignore[reportArgumentType] + result_database=db, ) dict_inputs = DictSource( data=[ @@ -50,7 +52,9 @@ def tally(n: int) -> dict[str, int]: list(total_pod.process(dict_inputs).iter_data()) tally_pod = CachedFunctionPod( - FunctionPod(PythonDataFunction(tally, output_keys="counts")), result_database=db + # The fixture is built with the released v0.2.0 API, unchanged. + FunctionPod(PythonDataFunction(tally, output_keys="counts")), # pyright: ignore[reportArgumentType] + result_database=db, ) int_inputs = DictSource( data=[{"k": "r", "n": 5}], diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index ae68b7ab1..f3df05bd3 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -19,8 +19,7 @@ import pytest from orcapod.core.cached_function_pod import CachedFunctionPod -from orcapod.core.data_function import PythonDataFunction -from orcapod.core.function_pod import FunctionPod +from orcapod.core.function_pod import function_pod from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase from orcapod.sources import ArrowTableSource, DictSource @@ -91,8 +90,19 @@ def reorder(m: dict[str, int]) -> dict[str, int]: return dict(reversed(list(m.items()))) -def _pod(fn: Any, output_key: str) -> FunctionPod: - return FunctionPod(PythonDataFunction(fn, output_keys=output_key)) +def _pod(fn: Any, output_key: str) -> Any: + return function_pod(output_keys=output_key)(fn).pod + + +def _keys(value: Any) -> list[Any]: + """Return a dict value's keys in iteration order.""" + assert isinstance(value, dict) + return list(value) + + +def _stored(source: Any, column: str) -> Any: + """Return the first row's stored (Arrow) value of ``column``.""" + return source.as_table().column(column).to_pylist()[0] @dataclass @@ -117,12 +127,12 @@ def test_declared_schema_reordered_dicts_hash_equal(self): assert _row_hash({"a": 1, "b": 2}, t) == _row_hash({"b": 2, "a": 1}, t) def test_stored_entries_are_in_key_order(self): - table = _source({"b": 2, "a": 1}).as_table() - assert [e["key"] for e in table.column("m").to_pylist()[0]] == ["a", "b"] + entries = _stored(_source({"b": 2, "a": 1}), "m") + assert [e["key"] for e in entries] == ["a", "b"] def test_str_keys_order_by_code_point_not_locale(self): - table = _source({"é": 1, "z": 2, "A": 3}).as_table() - assert [e["key"] for e in table.column("m").to_pylist()[0]] == ["A", "z", "é"] + entries = _stored(_source({"é": 1, "z": 2, "A": 3}), "m") + assert [e["key"] for e in entries] == ["A", "z", "é"] def test_different_values_still_hash_differently(self): assert _row_hash({"a": 1, "b": 2}) != _row_hash({"a": 2, "b": 1}) @@ -256,7 +266,7 @@ def test_pod_output_reordered_dicts_hash_equal(self): def test_fresh_pod_output_iterates_in_key_order(self): source = _source({"b": 2, "a": 1}, dict[str, int]) [(_, out)] = list(_pod(echo, "out").process(source).iter_data()) - assert list(out.as_dict()["out"]) == ["a", "b"] + assert _keys(out.as_dict()["out"]) == ["a", "b"] def test_pod_does_not_mutate_the_returned_dict(self): returned: list[dict[str, int]] = [] @@ -282,7 +292,7 @@ async def test_async_output_matches_sync_and_is_order_independent(self): assert async_flipped is not None assert sync_out.content_hash() == async_plain.content_hash() assert async_plain.content_hash() == async_flipped.content_hash() - assert list(async_flipped.as_dict()["out"]) == ["a", "b"] + assert _keys(async_flipped.as_dict()["out"]) == ["a", "b"] def _delta_db(tmp_path: Path) -> DeltaTableDatabase: @@ -336,8 +346,8 @@ def test_fresh_and_cached_dict_outputs_iterate_identically(self, result_db): [(_, cached)] = list(pod.process(source).iter_data()) assert cached.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False - assert list(fresh.as_dict()["out"]) == list(cached.as_dict()["out"]) - assert list(cached.as_dict()["out"]) == ["a", "b", "c"] + assert _keys(fresh.as_dict()["out"]) == _keys(cached.as_dict()["out"]) + assert _keys(cached.as_dict()["out"]) == ["a", "b", "c"] assert fresh.content_hash() == cached.content_hash() @@ -416,7 +426,7 @@ def test_cached_unsorted_dict_output_is_served_as_stored(self, store): [(_, fresh)] = list(fresh_pod.process(inputs).iter_data()) assert cached.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False - assert list(cached.as_dict()["counts"]) == ["z", "a"] - assert list(fresh.as_dict()["counts"]) == ["a", "z"] + assert _keys(cached.as_dict()["counts"]) == ["z", "a"] + assert _keys(fresh.as_dict()["counts"]) == ["a", "z"] assert cached.as_dict() == fresh.as_dict() assert cached.content_hash() != fresh.content_hash() From cb8eb1dd9f032a882657fb6339b33c76558bcf2b Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 11:02:14 -0800 Subject: [PATCH 14/22] docs(plan): mark hash-time sorting superseded; ITL-905 review round plan Co-Authored-By: Claude Opus 5.5 --- .../2026-11-11-itl-905-dict-order-hash.md | 36 ++++++++++++++++--- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md index 496a69aff..dc98459d0 100644 --- a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md +++ b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md @@ -2,9 +2,9 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Equal dicts give equal row hashes on every path (Python and Arrow inputs, nested, sync and async, fresh and cached), because a `dict` is order independent (developer, November 11, 2026). +**Goal:** Equal dicts and sets built from Python values give equal row hashes on every path (nested, sync and async, fresh and cached), because a `dict` and a `set` are order independent (developer, November 11, 2026). A key/value list that arrives already in Arrow form is an ordered list and hashes as stored (ruling below). -**Architecture:** One ordering rule, defined once on Arrow *storage* key values (`semantic_types/dict_order.py`). It is applied in three places: (1) the converter sorts entries when a Python dict becomes `large_list>`; (2) the Arrow hasher sorts the entries of every dict-encoded list before starfix sees them, so tables that arrive already holding unsorted entries (Arrow/Delta sources, v0.2.0 stores, operator rebuilds) hash canonically; (3) a datagram built from Python values holds its dict values in the same canonical order, so a fresh value and a cached read-back value iterate identically. +**Architecture (final):** One ordering rule, defined once on Arrow *storage* values (`semantic_types/canonical_order.py`). It is applied in two places: (1) the converter sorts dict entries and plain `set[T]` elements when a Python value becomes Arrow; (3) a datagram built from Python values holds its dict values in the same canonical order, so a fresh value and a cached read-back value iterate identically. Fix (2), sorting dict-encoded lists at hash time, was implemented and then REMOVED by the developer's ruling: hashing an Arrow table never sorts anything. Sections 2, 4 and 5 below are the original proposal; "Outcome" at the end is authoritative. **Tech Stack:** Python, PyArrow, starfix 0.4 (unchanged), pytest via `uv run`. @@ -64,7 +64,7 @@ Sort key `dict_entry_sort_key(storage_key)`, total and independent of process, l | Key type (storage) | Order | |---|---| -| `None` (key type `K | None`) | first | +| `None` (key type `K \| None`) | first | | `bool` | `False < True` | | `int` (any width) | numeric | | `float` | numeric; `-0.0` and `0.0` compare equal (they are one Python dict key); **NaN rejected** | @@ -175,11 +175,13 @@ def sort_dict_entries(entries: list[dict], column: str) -> list[dict]: - [ ] In the `origin is dict` converter: build `[{"key": key_converter(k), "value": value_converter(v)} ...]`, then return `sort_dict_entries(entries, column=...)`. Replace the two `set(field_names) == {"key", "value"}` checks with `is_dict_arrow_type`. - [ ] Run Task 1 tests (Python-origin cases now pass; Arrow-native still fail) and `uv run pytest tests/test_semantic_types -q`. Commit. -### Task 4: hasher sorts dict entries (fix 2) +### Task 4: hasher sorts dict entries (fix 2) — SUPERSEDED, do not implement + +> Rejected by the developer (November 11, 2026) and reverted: hashing an Arrow table never sorts. `SemanticHashingVisitor.visit_list` stays unchanged. **Files:** Modify `src/orcapod/hashing/visitors.py` (`SemanticHashingVisitor.visit_list`). -- [ ] After `_visit_list_elements`, if `is_dict_arrow_type(list_type)`, return `sort_dict_entries(processed, column=".".join(self._current_field_path))`. Run Task 1 tests (Arrow-native, nested, store cases pass) and `uv run pytest tests/test_hashing -q` (golden must pass unchanged). Commit. +- [x] ~~After `_visit_list_elements`, if `is_dict_arrow_type(list_type)`, return `sort_dict_entries(processed, column=".".join(self._current_field_path))`. Run Task 1 tests (Arrow-native, nested, store cases pass) and `uv run pytest tests/test_hashing -q` (golden must pass unchanged). Commit.~~ (reverted) ### Task 5: canonical in-memory values (fix 3) @@ -228,3 +230,27 @@ walk over every dict literal in `cases.py` found no other dict reaching Arrow ou Performance: the Arrow hasher is byte-for-byte unchanged from `main`, so hashing tables with no dict columns costs nothing extra. Conversion adds one sort per dict or set value. + +--- + +## Review round 1 (CodeRabbit on `2994073e`) and merge of `main` `c8efc092` + +Worker 2, November 11, 2026. + +* Merge `origin/main` (`c8efc092`): conflicts only in `CHANGELOG.md` and `DESIGN_ISSUES.md`, + resolved additively (one `[0.3.0]` section with both sides; DO1 and NP1 both kept). + Golden diff vs main: 2 changed lines (the approved `index_pick` s1/s2 data hashes), 40 added. +* Findings and actions: + 1. `hashing.md` signed zero: asked coordinator (proposal: document, no normalisation). + 2. `hashing.md` fresh vs cached: qualify for `0.2.0` cache hits; same in CHANGELOG and DO1. + 3. `canonical_order._key_rank` NaN: reject every NaN-capable non-integral real + (`numpy.float32`) and Decimal NaN, not only `float`. Failing test first. + 4. `ListLogicalType` set NaN (pre-existing code): asked coordinator (proposal: decline, report). + 5. Plan Task 4: marked superseded (this commit). + 6-9. Docstrings on new test helpers/classes/methods and golden case functions. + 10. `tarfile.extractall(filter="data")` needs 3.11.4: use the filter when available, else a + member check (regular files and dirs only, no absolute or `..` paths). + 11. (body) `Datagram.update` / `with_columns` dict-only branches: apply the canonical copy. + Failing test first. + 12. (body nits) `itertools.pairwise`, `datetime.UTC`: apply. ast-grep CWE-78 on a constant + subprocess script and basedpyright "partially unknown" warnings from `DataValue`: decline. From 9ee3854779dd4cbb51753497980fc122827159c6 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 11:03:16 -0800 Subject: [PATCH 15/22] test(hashing): failing tests for numpy NaN keys and Datagram update order (ITL-905) Co-Authored-By: Claude Opus 5.5 --- tests/test_hashing/test_dict_order_hash.py | 39 ++++++++++++++++++++++ tests/test_hashing/test_set_order_hash.py | 12 +++++++ 2 files changed, 51 insertions(+) diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index f3df05bd3..e0e85f8ee 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -15,10 +15,12 @@ from pathlib import Path from typing import Any +import numpy as np import pyarrow as pa import pytest from orcapod.core.cached_function_pod import CachedFunctionPod +from orcapod.core.datagrams import Datagram from orcapod.core.function_pod import function_pod from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase from orcapod.sources import ArrowTableSource, DictSource @@ -169,6 +171,11 @@ def test_nan_key_is_rejected_naming_the_column(self): with pytest.raises(ValueError, match="'m'.*NaN|NaN.*'m'"): _row_hash({float("nan"): 1, 1.0: 2}, dict[float, int]) + def test_numpy_nan_key_is_rejected(self): + """A NaN that is not a Python ``float`` (``numpy.float32``) is rejected too.""" + with pytest.raises(ValueError, match="NaN"): + _row_hash({np.float32("nan"): 1, np.float32(1.0): 2}, dict[float, int]) + class TestNestedDicts: def test_dict_of_dicts_inner_and_outer_reordered(self): @@ -295,6 +302,38 @@ async def test_async_output_matches_sync_and_is_order_independent(self): assert _keys(async_flipped.as_dict()["out"]) == ["a", "b"] +# --------------------------------------------------------------------------- +# Datagram column updates +# --------------------------------------------------------------------------- + + +class TestDatagramColumnUpdates: + """``update`` and ``with_columns`` hold dict values in canonical key order.""" + + def test_update_holds_dict_in_key_order(self): + """A replaced dict value iterates in key order, like a constructed one.""" + datagram = Datagram({"m": {"a": 1}}, python_schema={"m": dict[str, int]}) + updated = datagram.update(m={"b": 2, "a": 1}) + assert _keys(updated.as_dict()["m"]) == ["a", "b"] + + def test_with_columns_holds_dict_in_key_order(self): + """A new dict column iterates in key order, like a constructed one.""" + datagram = Datagram({"x": 1}, python_schema={"x": int}) + extended = datagram.with_columns( + column_types={"m": dict[str, int]}, m={"b": 2, "a": 1} + ) + assert _keys(extended.as_dict()["m"]) == ["a", "b"] + + def test_update_and_with_columns_do_not_mutate_the_given_dict(self): + """The caller's dict keeps its own order.""" + given = {"b": 2, "a": 1} + Datagram({"m": {"a": 1}}, python_schema={"m": dict[str, int]}).update(m=given) + Datagram({"x": 1}, python_schema={"x": int}).with_columns( + column_types={"m": dict[str, int]}, m=given + ) + assert list(given) == ["b", "a"] + + def _delta_db(tmp_path: Path) -> DeltaTableDatabase: return DeltaTableDatabase(tmp_path / "store") diff --git a/tests/test_hashing/test_set_order_hash.py b/tests/test_hashing/test_set_order_hash.py index dc4c7253d..2b4b6fb2d 100644 --- a/tests/test_hashing/test_set_order_hash.py +++ b/tests/test_hashing/test_set_order_hash.py @@ -12,6 +12,9 @@ import sys import textwrap +import numpy as np +import pytest + from orcapod.sources import DictSource _ROW_HASH_SCRIPT = textwrap.dedent( @@ -66,3 +69,12 @@ def test_int_set_elements_are_sorted(self): data_schema={"k": str, "s": set[int]}, ) assert source.as_table().column("s").to_pylist() == [[-1, 7, 300]] + + def test_numpy_nan_element_is_rejected(self): + """A NaN that is not a Python ``float`` (``numpy.float32``) has no order either.""" + with pytest.raises(ValueError, match="NaN"): + DictSource( + data=[{"k": "r", "s": {np.float32("nan"), np.float32(1.0)}}], + tag_columns=["k"], + data_schema={"k": str, "s": set[float]}, + ).as_table() From fcb9f75990cf1c02e404c4e4df3b14ef8d89ba50 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 11:03:16 -0800 Subject: [PATCH 16/22] fix(semantic_types): reject NaN of any numeric type; canonical dicts in Datagram.update/with_columns (ITL-905) Addresses CodeRabbit review on PR #301. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/datagrams/datagram.py | 11 ++++++-- src/orcapod/semantic_types/canonical_order.py | 28 +++++++++++++++---- 2 files changed, 32 insertions(+), 7 deletions(-) diff --git a/src/orcapod/core/datagrams/datagram.py b/src/orcapod/core/datagrams/datagram.py index 029bd0a4e..bcc3ee9fe 100644 --- a/src/orcapod/core/datagrams/datagram.py +++ b/src/orcapod/core/datagrams/datagram.py @@ -849,8 +849,13 @@ def update(self, **updates: DataValue) -> Self: return new_d else: assert self._data_dict is not None + # Updated dict values are held in canonical key order, as at + # construction (ITL-905); the caller's dicts are copied, not mutated. + canonical_updates = self._canonical_dict_values( + updates, self._ensure_python_schema() + ) new_d = self.copy(include_cache=False) - new_d._data_dict = {**self._data_dict, **updates} + new_d._data_dict = {**self._data_dict, **canonical_updates} new_d._data_table = None return new_d @@ -1009,7 +1014,9 @@ def with_columns( {k: schema.get(k, inferred.get(k, str)) for k in new_data} ) new_d = self.copy(include_cache=False) - new_d._data_dict = new_data + # New dict values are held in canonical key order, as at construction + # (ITL-905); the caller's dicts are copied, not mutated. + new_d._data_dict = self._canonical_dict_values(new_data, new_schema) new_d._data_python_schema = new_schema new_d._data_table = None new_d._data_arrow_schema = None diff --git a/src/orcapod/semantic_types/canonical_order.py b/src/orcapod/semantic_types/canonical_order.py index 2a6ed2f35..3ac9b7e0a 100644 --- a/src/orcapod/semantic_types/canonical_order.py +++ b/src/orcapod/semantic_types/canonical_order.py @@ -11,8 +11,10 @@ The order is defined on storage values (what the converter produces): * ``None`` sorts first. -* ``bool``, ``int``, ``float``, ``Decimal``: numeric. A NaN has no order and is - rejected. ``-0.0`` and ``0.0`` are one value. +* ``bool``, ``int``, ``float``, ``Decimal``: numeric. A NaN of any numeric type has + no order and is rejected. ``-0.0`` and ``0.0`` compare equal, so within one dict or + set they collide (and raise, see below); a dict keyed by ``-0.0`` and one keyed by + ``0.0`` store different bytes and hash differently, like any float value. * ``str``: Unicode code point (never locale). ``bytes``: unsigned bytewise. * ``date``, ``datetime`` (by instant), ``time``, ``timedelta``: natural order. @@ -23,9 +25,12 @@ from __future__ import annotations import math +import numbers import types import typing from collections.abc import Callable +from decimal import Decimal +from itertools import pairwise from typing import Any, TypeVar, cast, get_args, get_origin from orcapod.errors import CanonicalOrderError @@ -40,15 +45,28 @@ def _key_rank(key: object) -> _Rank: """Return the sort key of one storage key or set element. Raises: - CanonicalOrderError: If the key is a NaN float. + CanonicalOrderError: If the key is a NaN of any numeric type (Python ``float``, + NumPy floating scalars, ``Decimal``). """ if key is None: return (0,) - if isinstance(key, float) and math.isnan(key): + if _is_nan(key): raise CanonicalOrderError("a NaN key or set element has no canonical order") return (1, key) +def _is_nan(value: object) -> bool: + """Return whether ``value`` is a NaN of any numeric type. + + Integers are never NaN and are not converted (a large ``int`` does not fit a float). + """ + if isinstance(value, Decimal): + return value.is_nan() + if isinstance(value, numbers.Real) and not isinstance(value, numbers.Integral): + return math.isnan(value) + return False + + def _sorted_by_rank(items: list[tuple[_Rank, _T]]) -> list[_T]: """Sort ``(rank, payload)`` pairs by rank and return the payloads. @@ -59,7 +77,7 @@ def _sorted_by_rank(items: list[tuple[_Rank, _T]]) -> list[_T]: items = sorted(items, key=lambda item: item[0]) except TypeError as exc: raise CanonicalOrderError(f"values cannot be ordered: {exc}") from exc - for (previous, _), (current, payload) in zip(items, items[1:]): + for (previous, _), (current, payload) in pairwise(items): if previous == current: raise CanonicalOrderError(f"duplicate key or set element: {payload!r}") return [payload for _, payload in items] From fcd819b8379e02be88d4a17abd5bf5463ffd4d8c Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 11:05:16 -0800 Subject: [PATCH 17/22] docs(hashing): signed zero and 0.2.0 cache-hit order; test docstrings and 3.11.0 tarfile compat (ITL-905) Addresses CodeRabbit review on PR #301. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 5 +- DESIGN_ISSUES.md | 6 +- docs/reference/hashing.md | 25 ++++-- .../data/generate_v0_2_0_dict_store.py | 2 + tests/test_hashing/golden/cases.py | 10 ++- tests/test_hashing/test_dict_order_hash.py | 85 +++++++++++++++++-- tests/test_hashing/test_set_order_hash.py | 7 ++ 7 files changed, 117 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 011b4e3c8..6089d1cd1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,8 +14,9 @@ Orcapod now stores dict entries sorted by key and set elements sorted, wherever value becomes Arrow; see the hashing reference, section 1c, for the order. A NaN key or element, values that cannot be compared, and two keys that convert to the same stored value raise `CanonicalOrderError` (a `ValueError`) naming the column. A datagram built from -Python values holds its dicts in the same order, so fresh and cached pod outputs iterate -alike. An Arrow table is still hashed exactly as stored: a key/value list that arrives in +Python values (and values set with `update` or `with_columns`) holds its dicts in the same +order, so a newly computed pod output and one served from a `0.3.0` cache iterate alike (an +output served from a `0.2.0` cache keeps its stored order, see below). An Arrow table is still hashed exactly as stored: a key/value list that arrives in Arrow form is never re-sorted. Which released hashes move: only those of values holding a dict whose insertion order was diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index be03d1bbf..066004cd4 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2319,8 +2319,10 @@ the one ordering rule in `semantic_types/canonical_order.py` (None first; numeri code point; `bytes` bytewise; dates and times naturally). NaN, incomparable values and two keys or elements with the same storage value raise `CanonicalOrderError`, wrapped with the column name by the caller. `Datagram` built from Python values holds key-ordered copies of -its dict values (inside `list`, `dict`, `Optional`), so fresh and cached outputs iterate -alike; the caller's dict is never mutated. Released hashes move only for values not already +its dict values (inside `list`, `dict`, `Optional`; also after `update` and `with_columns`), +so a newly computed output and one served from a `0.3.0` cache iterate alike (one served +from a `0.2.0` cache keeps its stored order until recomputed); the caller's dict is never +mutated. Released hashes move only for values not already in canonical order (two golden entries, `pipeline:index_pick.index` row data hashes); see the `0.3.0` changelog. Regressions: `tests/test_hashing/test_dict_order_hash.py` (including a Delta store written by `0.2.0`) and `tests/test_hashing/test_set_order_hash.py`. Open: a diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 327a62b5a..c8b7560e6 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -272,9 +272,12 @@ print(dict_row_hash({"a": 1, "b": 2}) == dict_row_hash({"b": 2, "a": 1})) which every Python-to-Arrow path uses: sources built from Python values, datagrams, function pod outputs (sync and async), and dicts nested in lists, dicts or dataclass and pydantic fields. A datagram built from Python values also holds its dict values (including dicts -inside lists, dicts and `Optional`) as key-ordered copies, so a fresh pod output and the -same output read back from a cache iterate in the same order. The caller's dict is never -modified; dicts inside dataclass or pydantic objects keep their own order in memory. +inside lists, dicts and `Optional`) as key-ordered copies, including values set later with +`update` or `with_columns`, so a newly computed pod output and the same output read back +from a cache written by `0.3.0` iterate in the same order. An output served from a cache +written by `0.2.0` keeps the order it was stored in until it is recomputed (see the `0.3.0` +changelog). The caller's dict is never modified; dicts inside dataclass or pydantic objects +keep their own order in memory. **The order** is defined on the converted (storage) key or element, in `semantic_types/canonical_order.py`: @@ -288,10 +291,18 @@ modified; dicts inside dataclass or pydantic objects keep their own order in mem | `bytes` | unsigned bytewise | | `date`, `datetime`, `time`, `timedelta` | natural order; datetimes by instant (they are timezone aware after conversion) | -A NaN key or element has no order and raises `CanonicalOrderError` (a `ValueError`) naming -the column, as do keys or elements that cannot be compared and two keys or elements that -convert to the same stored value. A dict or set already built in this order is stored -byte-for-byte as before this rule existed, so its hash did not change. +A NaN key or element (a Python `float`, a NumPy floating scalar or a `Decimal`) has no order +and raises `CanonicalOrderError` (a `ValueError`) naming the column, as do keys or elements +that cannot be compared and two keys or elements that convert to the same stored value. A +dict or set already built in this order is stored byte-for-byte as before this rule existed, +so its hash did not change. + +**Signed zero.** A float is hashed by its stored bytes, and `0.0` and `-0.0` have different +bytes, so they hash differently wherever they appear: in a plain `float` column and as a +dict key or set element alike, although Python compares them equal (`{0.0: 1} == {-0.0: 1}`). +Orcapod does not rewrite `-0.0` to `0.0`: pod code can tell them apart (`math.copysign`, +`repr`, division), so the finer identity can cost a recompute but never serves a wrong +result. Within one dict or set the two compare equal, so they cannot both be present. **Arrow tables are hashed exactly as stored.** `StarfixArrowHasher` never sorts list contents. A `list>` column that arrives already in Arrow form (an diff --git a/tests/test_hashing/data/generate_v0_2_0_dict_store.py b/tests/test_hashing/data/generate_v0_2_0_dict_store.py index d8b3bef28..be6686441 100644 --- a/tests/test_hashing/data/generate_v0_2_0_dict_store.py +++ b/tests/test_hashing/data/generate_v0_2_0_dict_store.py @@ -26,10 +26,12 @@ def total(m: dict[str, int]) -> int: + """Return the sum of the dict's values.""" return sum(m.values()) def tally(n: int) -> dict[str, int]: + """Return a dict built out of key order.""" return {"z": n, "a": n} diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index f536a0f84..a7e09b766 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -552,9 +552,11 @@ def arrow_schema_hashes() -> dict[str, Any]: @case("arrow", "dict_order") def arrow_dict_order_hashes() -> dict[str, Any]: - # ITL-905: an Arrow table is hashed exactly as stored. A key/value list that - # arrives in Arrow form keeps its order as identity (sorted and unsorted pin - # different values), like any other list of structs. + """Pin that an Arrow table is hashed exactly as stored (ITL-905). + + A key/value list that arrives in Arrow form keeps its order as identity (sorted and + unsorted pin different values), like any other list of structs. + """ hasher = _ctx().arrow_hasher kv = pa.large_list( pa.struct([pa.field("key", pa.large_string()), pa.field("value", pa.int64())]) @@ -648,7 +650,7 @@ def _arrow_source(column: pa.Array[Any], nullable: bool = False) -> Any: @case("datagram", "dict_order") def datagram_dict_order() -> dict[str, Any]: - # ITL-905: rows holding equal dicts in different insertion orders pin equal hashes. + """Pin equal hashes for rows holding equal dicts in different insertion orders (ITL-905).""" from orcapod.sources import DictSource rows: dict[str, tuple[Any, list[Any]]] = { diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index e0e85f8ee..fea3b4e79 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -11,7 +11,7 @@ import tarfile from dataclasses import dataclass -from datetime import date, datetime, timezone +from datetime import UTC, date, datetime from pathlib import Path from typing import Any @@ -38,6 +38,7 @@ def _source(value: Any, value_type: Any = None) -> DictSource: + """Return a one-row ``DictSource`` whose ``m`` column holds ``value``.""" schema = None if value_type is None else {"k": str, "m": value_type} return DictSource( data=[{"k": "r", "m": value}], tag_columns=["k"], data_schema=schema @@ -45,15 +46,18 @@ def _source(value: Any, value_type: Any = None) -> DictSource: def _data(value: Any, value_type: Any = None) -> Any: + """Return the ``Data`` of the one row of ``_source(value, value_type)``.""" _, data = next(iter(_source(value, value_type).iter_data())) return data def _row_hash(value: Any, value_type: Any = None) -> str: + """Return the row data hash of ``value`` stored in a one-row source.""" return _data(value, value_type).content_hash().to_string() def _arrow_source(entries: list[dict[str, Any]]) -> ArrowTableSource: + """Return a one-row ``ArrowTableSource`` holding ``entries`` as a key/value list.""" # Non-nullable like a DictSource column declared ``dict[str, int]``. table = pa.table( { @@ -71,19 +75,23 @@ def _arrow_source(entries: list[dict[str, Any]]) -> ArrowTableSource: def _arrow_row_hash(entries: list[dict[str, Any]]) -> str: + """Return the row data hash of a key/value list given in Arrow form.""" _, data = next(iter(_arrow_source(entries).iter_data())) return data.content_hash().to_string() def total(m: dict[str, int]) -> int: + """Return the sum of the dict's values (the pod cached in the v0.2.0 store).""" return sum(m.values()) def tally(n: int) -> dict[str, int]: + """Return a dict built out of key order (the pod cached in the v0.2.0 store).""" return {"z": n, "a": n} def echo(m: dict[str, int]) -> dict[str, int]: + """Return the input dict unchanged.""" return m @@ -93,6 +101,7 @@ def reorder(m: dict[str, int]) -> dict[str, int]: def _pod(fn: Any, output_key: str) -> Any: + """Return the function pod of ``fn`` with one output key.""" return function_pod(output_keys=output_key)(fn).pod @@ -121,22 +130,29 @@ class Holder: class TestPythonDictValues: + """Python dicts reach Arrow in canonical key order.""" + def test_inferred_schema_reordered_dicts_hash_equal(self): + """Reordered dicts hash equal with an inferred schema.""" assert _row_hash({"a": 1, "b": 2}) == _row_hash({"b": 2, "a": 1}) def test_declared_schema_reordered_dicts_hash_equal(self): + """Reordered dicts hash equal with a declared ``dict[str, int]``.""" t = dict[str, int] assert _row_hash({"a": 1, "b": 2}, t) == _row_hash({"b": 2, "a": 1}, t) def test_stored_entries_are_in_key_order(self): + """The stored key/value list is sorted by key.""" entries = _stored(_source({"b": 2, "a": 1}), "m") assert [e["key"] for e in entries] == ["a", "b"] def test_str_keys_order_by_code_point_not_locale(self): + """``str`` keys sort by code point.""" entries = _stored(_source({"é": 1, "z": 2, "A": 3}), "m") assert [e["key"] for e in entries] == ["A", "z", "é"] def test_different_values_still_hash_differently(self): + """Sorting does not merge dicts with different values.""" assert _row_hash({"a": 1, "b": 2}) != _row_hash({"a": 2, "b": 1}) @pytest.mark.parametrize( @@ -154,20 +170,22 @@ def test_different_values_still_hash_differently(self): ( dict[datetime, int], { - datetime(2026, 1, 2, tzinfo=timezone.utc): 1, - datetime(2025, 5, 1, tzinfo=timezone.utc): 2, + datetime(2026, 1, 2, tzinfo=UTC): 1, + datetime(2025, 5, 1, tzinfo=UTC): 2, }, { - datetime(2025, 5, 1, tzinfo=timezone.utc): 2, - datetime(2026, 1, 2, tzinfo=timezone.utc): 1, + datetime(2025, 5, 1, tzinfo=UTC): 2, + datetime(2026, 1, 2, tzinfo=UTC): 1, }, ), ], ) def test_key_types_reordered_hash_equal(self, value_type, first, second): + """Every supported key type orders deterministically.""" assert _row_hash(first, value_type) == _row_hash(second, value_type) def test_nan_key_is_rejected_naming_the_column(self): + """A NaN key has no order and raises, naming the column.""" with pytest.raises(ValueError, match="'m'.*NaN|NaN.*'m'"): _row_hash({float("nan"): 1, 1.0: 2}, dict[float, int]) @@ -178,29 +196,36 @@ def test_numpy_nan_key_is_rejected(self): class TestNestedDicts: + """Dicts nested in dicts, lists and dataclass fields are ordered at every depth.""" + def test_dict_of_dicts_inner_and_outer_reordered(self): + """Inner and outer dicts are both ordered.""" t = dict[str, dict[str, int]] first = {"x": {"a": 1, "b": 2}, "y": {"c": 3}} second = {"y": {"c": 3}, "x": {"b": 2, "a": 1}} assert _row_hash(first, t) == _row_hash(second, t) def test_list_of_dicts_reordered_inside(self): + """Dicts inside a list are ordered.""" t = list[dict[str, int]] assert _row_hash([{"a": 1, "b": 2}, {"c": 3}], t) == _row_hash( [{"b": 2, "a": 1}, {"c": 3}], t ) def test_list_of_dicts_keeps_list_order(self): + """The list itself keeps its order.""" t = list[dict[str, int]] assert _row_hash([{"a": 1}, {"c": 3}], t) != _row_hash([{"c": 3}, {"a": 1}], t) def test_dict_of_list_of_dicts_reordered(self): + """Dicts inside a list inside a dict are ordered.""" t = dict[str, list[dict[str, int]]] first = {"x": [{"a": 1, "b": 2}], "w": []} second = {"w": [], "x": [{"b": 2, "a": 1}]} assert _row_hash(first, t) == _row_hash(second, t) def test_dict_inside_dataclass_field_reordered(self): + """A dict in a dataclass field is ordered when stored.""" first = Holder("h", {"a": 1, "b": 2}) second = Holder("h", {"b": 2, "a": 1}) assert _row_hash(first, Holder) == _row_hash(second, Holder) @@ -220,17 +245,20 @@ class TestArrowNativeKeyValueLists: """ def test_unsorted_entries_hash_as_stored(self): + """An unsorted Arrow key/value list keeps its order in the hash.""" sorted_entries = [{"key": "a", "value": 1}, {"key": "b", "value": 2}] unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] assert _arrow_row_hash(sorted_entries) != _arrow_row_hash(unsorted_entries) def test_key_sorted_entries_match_python_dicts_in_any_order(self): + """A key-sorted Arrow list hashes like the Python dict.""" sorted_entries = [{"key": "a", "value": 1}, {"key": "b", "value": 2}] t = dict[str, int] assert _arrow_row_hash(sorted_entries) == _row_hash({"b": 2, "a": 1}, t) assert _arrow_row_hash(sorted_entries) == _row_hash({"a": 1, "b": 2}, t) def test_user_table_is_not_rewritten(self): + """Hashing leaves the user's Arrow table as given.""" unsorted_entries = [{"key": "b", "value": 2}, {"key": "a", "value": 1}] source = _arrow_source(unsorted_entries) _, data = next(iter(source.iter_data())) @@ -238,11 +266,13 @@ def test_user_table_is_not_rewritten(self): assert source.as_table().column("m").to_pylist()[0] == unsorted_entries def test_ordered_struct_pairs_are_not_sorted(self): + """A list of ``f0``/``f1`` pairs is an ordered list, never sorted.""" pair_type = pa.large_list( pa.struct([pa.field("f0", pa.large_string()), pa.field("f1", pa.int64())]) ) def pair_hash(entries: list[dict[str, Any]]) -> str: + """Return the row hash of ``entries`` stored as a list of ordered pairs.""" table = pa.table( { "k": pa.array(["r"], pa.large_string()), @@ -264,21 +294,27 @@ def pair_hash(entries: list[dict[str, Any]]) -> str: class TestFunctionPodDictOutputs: + """Dicts returned by a function pod, sync and async, are ordered.""" + def test_pod_output_reordered_dicts_hash_equal(self): + """Equal pod outputs built in different orders hash equal.""" source = _source({"a": 1, "b": 2}, dict[str, int]) [(_, plain)] = list(_pod(echo, "out").process(source).iter_data()) [(_, flipped)] = list(_pod(reorder, "out").process(source).iter_data()) assert plain.content_hash() == flipped.content_hash() def test_fresh_pod_output_iterates_in_key_order(self): + """A fresh pod output dict iterates in key order.""" source = _source({"b": 2, "a": 1}, dict[str, int]) [(_, out)] = list(_pod(echo, "out").process(source).iter_data()) assert _keys(out.as_dict()["out"]) == ["a", "b"] def test_pod_does_not_mutate_the_returned_dict(self): + """The dict the pod returned keeps its own order.""" returned: list[dict[str, int]] = [] def keep(m: dict[str, int]) -> dict[str, int]: + """Return a new dict out of key order and remember it.""" result = {"b": m["b"], "a": m["a"]} returned.append(result) return result @@ -289,6 +325,7 @@ def keep(m: dict[str, int]) -> dict[str, int]: @pytest.mark.asyncio async def test_async_output_matches_sync_and_is_order_independent(self): + """The async path stores and hashes like the sync path.""" source = _source({"a": 1, "b": 2}, dict[str, int]) tag, data = next(iter(source.iter_data())) _, sync_out = _pod(echo, "out").process_data(tag, data) @@ -334,19 +371,43 @@ def test_update_and_with_columns_do_not_mutate_the_given_dict(self): assert list(given) == ["b", "a"] +def _extract_data_only(archive: tarfile.TarFile, target: Path) -> None: + """Extract ``archive`` under ``target``, allowing only plain files and directories. + + ``extractall(filter="data")`` exists from Python 3.11.4; on older 3.11 releases the + same guarantee is checked by hand: no links or devices, no absolute or ``..`` paths. + """ + if hasattr(tarfile, "data_filter"): + archive.extractall(target, filter="data") + return + root = target.resolve() + for member in archive.getmembers(): + destination = (target / member.name).resolve() + if not (member.isfile() or member.isdir()) or not destination.is_relative_to( + root + ): + raise ValueError(f"unsafe archive member: {member.name!r}") + archive.extractall(target) # noqa: S202 -- every member was checked above + + def _delta_db(tmp_path: Path) -> DeltaTableDatabase: + """Return a Delta result store under ``tmp_path``.""" return DeltaTableDatabase(tmp_path / "store") @pytest.fixture(params=["in_memory", "delta"]) def result_db(request, tmp_path): + """Yield an in-memory or a Delta result store.""" if request.param == "in_memory": return InMemoryArrowDatabase() return _delta_db(tmp_path) class TestCachedPodHit: + """Equal dicts built in different orders share one cache entry.""" + def test_reordered_equal_dict_is_a_cache_hit(self, result_db): + """An equal input dict in another order is a cache hit.""" pod = CachedFunctionPod(_pod(total, "total"), result_database=result_db) first = _source({"b": 2, "a": 1}, dict[str, int]) second = _source({"a": 1, "b": 2}, dict[str, int]) @@ -363,6 +424,7 @@ def test_reordered_equal_dict_is_a_cache_hit(self, result_db): @pytest.mark.asyncio async def test_reordered_equal_dict_is_an_async_cache_hit(self, result_db): + """The same holds on the async path.""" pod = CachedFunctionPod(_pod(total, "total"), result_database=result_db) tag1, data1 = next(iter(_source({"b": 2, "a": 1}, dict[str, int]).iter_data())) tag2, data2 = next(iter(_source({"a": 1, "b": 2}, dict[str, int]).iter_data())) @@ -377,6 +439,7 @@ async def test_reordered_equal_dict_is_an_async_cache_hit(self, result_db): assert out2.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False def test_fresh_and_cached_dict_outputs_iterate_identically(self, result_db): + """Fresh and cached outputs iterate in the same order.""" pod = CachedFunctionPod(_pod(reorder, "out"), result_database=result_db) source = _source({"a": 1, "b": 2, "c": 3}, dict[str, int]) @@ -403,13 +466,15 @@ class TestV020Store: @pytest.fixture def store(self, tmp_path): + """Return the v0.2.0 Delta store, extracted under ``tmp_path``.""" # The store is shipped as a tarball: its directory names contain ":". target = tmp_path / "v020_store" with tarfile.open(V020_STORE) as archive: - archive.extractall(target, filter="data") + _extract_data_only(archive, target) return DeltaTableDatabase(target) def _inputs(self) -> DictSource: + """Return the two inputs the v0.2.0 store was built from.""" return DictSource( data=[ {"k": "unsorted", "m": {"b": 2, "a": 1}}, @@ -421,6 +486,7 @@ def _inputs(self) -> DictSource: ) def test_unsorted_row_recomputes_once_and_old_rows_stay(self, store): + """An out-of-order v0.2.0 input recomputes once; its old row stays.""" pod = CachedFunctionPod(_pod(total, "total"), result_database=store) before = store.get_all_records(pod.record_path) assert before is not None @@ -449,8 +515,11 @@ def test_unsorted_row_recomputes_once_and_old_rows_stay(self, store): ) def test_cached_unsorted_dict_output_is_served_as_stored(self, store): - # A v0.2.0 row is read back as stored: an Arrow table is never re-sorted, so a - # cache hit returns the old insertion order and hash until it is recomputed. + """A cached v0.2.0 dict output is served in its stored order. + + An Arrow table is never re-sorted, so a cache hit returns the old insertion order + and hash until the entry is recomputed. + """ inputs = DictSource( data=[{"k": "r", "n": 5}], tag_columns=["k"], diff --git a/tests/test_hashing/test_set_order_hash.py b/tests/test_hashing/test_set_order_hash.py index 2b4b6fb2d..461452d4d 100644 --- a/tests/test_hashing/test_set_order_hash.py +++ b/tests/test_hashing/test_set_order_hash.py @@ -34,6 +34,7 @@ def _row_hash_in_subprocess(seed: str) -> str: + """Return the row hash of a fixed ``set[str]`` computed under ``PYTHONHASHSEED=seed``.""" env = {**os.environ, "PYTHONHASHSEED": seed} result = subprocess.run( [sys.executable, "-c", _ROW_HASH_SCRIPT], @@ -46,6 +47,7 @@ def _row_hash_in_subprocess(seed: str) -> str: def _source(value: set[str]) -> DictSource: + """Return a one-row ``DictSource`` whose ``s`` column holds ``value``.""" return DictSource( data=[{"k": "r", "s": value}], tag_columns=["k"], @@ -55,14 +57,19 @@ def _source(value: set[str]) -> DictSource: class TestPlainSetOrder: + """A plain ``set[T]`` is stored sorted, never in iteration order.""" + def test_row_hash_is_independent_of_hash_seed(self): + """Two processes with different hash seeds give one row hash.""" assert _row_hash_in_subprocess("1") == _row_hash_in_subprocess("2") def test_stored_elements_are_sorted(self): + """``str`` elements are stored sorted.""" table = _source({"cherry", "apple", "banana"}).as_table() assert table.column("s").to_pylist() == [["apple", "banana", "cherry"]] def test_int_set_elements_are_sorted(self): + """``int`` elements are stored in numeric order.""" source = DictSource( data=[{"k": "r", "s": {300, -1, 7}}], tag_columns=["k"], From d0ff008b43483149e0862ef9d6d952979115d964 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 11:06:40 -0800 Subject: [PATCH 18/22] test(hashing): type the numpy-scalar row in the set NaN test (ITL-905) Co-Authored-By: Claude Opus 5.5 --- tests/test_hashing/test_set_order_hash.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/test_hashing/test_set_order_hash.py b/tests/test_hashing/test_set_order_hash.py index 461452d4d..b6590f841 100644 --- a/tests/test_hashing/test_set_order_hash.py +++ b/tests/test_hashing/test_set_order_hash.py @@ -11,6 +11,7 @@ import subprocess import sys import textwrap +from typing import Any import numpy as np import pytest @@ -79,9 +80,14 @@ def test_int_set_elements_are_sorted(self): def test_numpy_nan_element_is_rejected(self): """A NaN that is not a Python ``float`` (``numpy.float32``) has no order either.""" + # NumPy scalars are outside the declared ``DataValue`` type; DictSource accepts + # them at runtime, which is the case under test. + rows: list[dict[str, Any]] = [ + {"k": "r", "s": {np.float32("nan"), np.float32(1.0)}} + ] with pytest.raises(ValueError, match="NaN"): DictSource( - data=[{"k": "r", "s": {np.float32("nan"), np.float32(1.0)}}], + data=rows, tag_columns=["k"], data_schema={"k": str, "s": set[float]}, ).as_table() From 388bb602786e645f639c84b34bf78238e9653c25 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 11:25:56 -0800 Subject: [PATCH 19/22] docs(design_issues): reword DO1 in-memory order sentence (ITL-905) Co-Authored-By: Claude Opus 5.5 --- DESIGN_ISSUES.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 066004cd4..80ec7289e 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2319,8 +2319,8 @@ the one ordering rule in `semantic_types/canonical_order.py` (None first; numeri code point; `bytes` bytewise; dates and times naturally). NaN, incomparable values and two keys or elements with the same storage value raise `CanonicalOrderError`, wrapped with the column name by the caller. `Datagram` built from Python values holds key-ordered copies of -its dict values (inside `list`, `dict`, `Optional`; also after `update` and `with_columns`), -so a newly computed output and one served from a `0.3.0` cache iterate alike (one served +its dict values (inside `list`, `dict`, `Optional`), and `update` and `with_columns` do the +same, so a newly computed output and one served from a `0.3.0` cache iterate alike (one served from a `0.2.0` cache keeps its stored order until recomputed); the caller's dict is never mutated. Released hashes move only for values not already in canonical order (two golden entries, `pipeline:index_pick.index` row data hashes); see From a32387ce87e37857885d9ecc949b5e68c5ad5776 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 06:18:29 -0800 Subject: [PATCH 20/22] fix(hashing): retain canonical set order in Arrow-backed updates Refresh ITL-905 against the v0.3 hash layouts, document confirmed ordering decisions, and preserve the approved golden changes. Co-Authored-By: GPT-6 --- changelog.d/ITL-905.md | 33 +++-- design_issues/ITL-905-DO1.md | 23 +++- docs/reference/hashing.md | 33 +++-- .../semantic_types/universal_converter.py | 16 ++- .../2026-11-11-itl-905-dict-order-hash.md | 2 +- ...2026-11-12-itl-905-refresh-against-main.md | 63 +++++++++ tests/test_hashing/golden/data/arrow.json | 4 + tests/test_hashing/golden/data/datagram.json | 36 ++++++ tests/test_hashing/golden/data/pipeline.json | 4 +- tests/test_hashing/test_dict_order_hash.py | 120 ++++++++++++++---- tests/test_hashing/test_set_order_hash.py | 43 +++++++ 11 files changed, 312 insertions(+), 65 deletions(-) create mode 100644 superpowers/plans/2026-11-12-itl-905-refresh-against-main.md diff --git a/changelog.d/ITL-905.md b/changelog.d/ITL-905.md index 81592ce41..efe9f58c3 100644 --- a/changelog.d/ITL-905.md +++ b/changelog.d/ITL-905.md @@ -7,29 +7,34 @@ A `dict[K, V]` value is stored as `large_list>` and a plain ` iteration order, so `{"a": 1, "b": 2}` and `{"b": 2, "a": 1}` had different row hashes, and the same `set[str]` had a different row hash in every process (`PYTHONHASHSEED`). Orcapod now stores dict entries sorted by key and set elements sorted, wherever a Python -value becomes Arrow; see the hashing reference, section 1d, for the order. A NaN key or +value becomes Arrow; see the hashing reference, section 1e, for the order. A NaN key or element, values that cannot be compared, and two keys that convert to the same stored value raise `CanonicalOrderError` (a `ValueError`) naming the column. A datagram built from Python values (and values set with `update` or `with_columns`) holds its dicts in the same -order, so a newly computed pod output and one served from a `0.3.0` cache iterate alike (an -output served from a `0.2.0` cache keeps its stored order, see below). An Arrow table is still hashed exactly as stored: a key/value list that arrives in -Arrow form is never re-sorted. +order, so a newly computed pod output and one served from the cache iterate alike. An Arrow +table is still hashed exactly as stored: a key/value list that arrives in Arrow form is never +re-sorted. -Which released hashes move: only those of values holding a dict whose insertion order was +Set values passed to `update` after Arrow materialization, or to `with_columns` with an +explicit Arrow list field, are also stored in canonical order. The existing Arrow field +(width, nullability and metadata) is preserved; actual list values keep their supplied order. + +Which hashes move because of this change, on top of the v0.3 hash step (ITL-910, ITL-945), +which moves every stored hash: only those of values holding a dict whose insertion order was not key order, or a `set[T]` of two or more elements whose iteration order was not sorted (for `str` and `bytes` that order changed with every process, so it had no stable value). For such a row, its data hash moves, and with it the function pod cache key, the pipeline record and entry ids, and the default `source_id` (with its per-row record ids) of a source built without an explicit `source_id`. Schema hashes, `pipeline_hash` values and -explicit `source_id`s do not move, and a dict already built in key order is stored -byte-for-byte as before. In the golden hash suite exactly two entries moved: +explicit `source_id`s do not move for this reason, and a dict already built in key order is +stored byte-for-byte as before if every nested unordered container is also canonical. +In the golden hash suite exactly two entries moved: `pipeline:index_pick.index.row[{"subject":"s1"}].data_hash` and `...s2...`, whose rows hold `meta={"k": ..., "j": ...}`. -What a `0.2.0` store sees (checked against a Delta store written by `0.2.0`): table paths -are unchanged, so every stored row is still found. An input holding an out-of-order dict -gets a new hash, misses the cache once, and is recomputed; the new row is written next to -the old one, which stays (a store-wide read shows both, nothing is dropped). A cache hit -whose stored output holds an out-of-order dict is served as stored, with its `0.2.0` order -and hash, until that entry is recomputed. To start clean, drop the old rows or use a new -store; Orcapod does not rewrite stored rows. +What a `0.2.0` store sees (checked against a Delta store written by `0.2.0`): nothing +changes because of this fix. v0.3 reads no table `0.2.0` wrote (every table is at a new path, +see "Data context `std:v1:default`", ITL-944), so every input is computed again whether its +dict was in key order or not, and no row stored in `0.2.0`'s insertion order is ever served. +The `0.2.0` tables stay where they are, untouched. Every dict and plain set a v0.3 store +holds from a pod output or a source built from Python values is in canonical order. diff --git a/design_issues/ITL-905-DO1.md b/design_issues/ITL-905-DO1.md index 5687b20de..574de6f16 100644 --- a/design_issues/ITL-905-DO1.md +++ b/design_issues/ITL-905-DO1.md @@ -19,14 +19,23 @@ sorts anything (starfix ignores only column order and struct field order). The c the one ordering rule in `semantic_types/canonical_order.py` (None first; numeric; `str` by code point; `bytes` bytewise; dates and times naturally). NaN, incomparable values and two keys or elements with the same storage value raise `CanonicalOrderError`, wrapped with the -column name by the caller. `Datagram` built from Python values holds key-ordered copies of +column name by the caller (decided November 12, 2026). `Datagram` built from Python values +holds key-ordered copies (also decided November 12, 2026) of its dict values (inside `list`, `dict`, `Optional`), and `update` and `with_columns` do the -same, so a newly computed output and one served from a `0.3.0` cache iterate alike (one served -from a `0.2.0` cache keeps its stored order until recomputed); the caller's dict is never -mutated. Released hashes move only for values not already +same, so a newly computed output and one served from the cache iterate alike (v0.3 reads no +table `0.2.0` wrote, so no row in `0.2.0`'s insertion order is ever served); the caller's dict +is never mutated. Set inputs written through an existing Arrow list schema also use the +same ordering rule, preserving the exact field; list inputs retain their supplied order. +On top of the v0.3 hash step, hashes move only for values not already in canonical order (two golden entries, `pipeline:index_pick.index` row data hashes); see -the `0.3.0` changelog. Regressions: `tests/test_hashing/test_dict_order_hash.py` (including a -Delta store written by `0.2.0`) and `tests/test_hashing/test_set_order_hash.py`. Open: a +the `0.3.0` changelog. This storage order is a separate layer from the semantic hasher's +ordering of its hash input (ITL-942), which sorts by canonical JSON text and stores nothing. Regressions: `tests/test_hashing/test_dict_order_hash.py` (including a +Delta store written by `0.2.0`, which v0.3 does not read) and `tests/test_hashing/test_set_order_hash.py`. Open: a key/value list that arrives already in Arrow form keeps its stored order and hash, by the same ruling; dicts inside dataclass or pydantic objects keep their own order in memory -(their stored and hashed form is canonical). +(their stored and hashed form is canonical). A fresh user-class instance iterates its +nested dict in insertion order, while a read-back instance iterates it in key order; +an order-sensitive pod can observe that difference for equal hashed inputs. Reconstructing +user-class instances is outside this fix (coordinator decision, November 12, 2026). +`test_dataclass_dict_storage_is_canonical_but_fresh_object_is_unchanged` pins equal stored +values and hashes, the fresh/read-back iteration difference, and caller non-mutation. diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 8d84a2b4c..cc4a3d4c1 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -480,7 +480,7 @@ moves no path. --- -### 1d. `dict` and `set` values: canonical order +### 1e. `dict` and `set` values: canonical order A `dict` and a `set` have no order, so equal values must hash equally whatever order they were built in. Arrow stores both as lists: `dict[K, V]` as `large_list>` @@ -512,10 +512,12 @@ pod outputs (sync and async), and dicts nested in lists, dicts or dataclass and fields. A datagram built from Python values also holds its dict values (including dicts inside lists, dicts and `Optional`) as key-ordered copies, including values set later with `update` or `with_columns`, so a newly computed pod output and the same output read back -from a cache written by `0.3.0` iterate in the same order. An output served from a cache -written by `0.2.0` keeps the order it was stored in until it is recomputed (see the `0.3.0` -changelog). The caller's dict is never modified; dicts inside dataclass or pydantic objects -keep their own order in memory. +from the cache iterate in the same order (decided November 12, 2026). The caller's dict +is never modified; dicts inside dataclass or pydantic objects +keep their own order in memory. Their stored dict values and hashes are canonical, but +a fresh user-class instance can iterate a nested dict in insertion order while a read-back +instance iterates it in key order. An order-sensitive pod can therefore observe different +orders for equal hashed inputs; reconstructing user-class instances is outside this fix. **The order** is defined on the converted (storage) key or element, in `semantic_types/canonical_order.py`: @@ -531,8 +533,10 @@ keep their own order in memory. A NaN key or element (a Python `float`, a NumPy floating scalar or a `Decimal`) has no order and raises `CanonicalOrderError` (a `ValueError`) naming the column, as do keys or elements -that cannot be compared and two keys or elements that convert to the same stored value. A -dict or set already built in this order is stored byte-for-byte as before this rule existed, +that cannot be compared and two keys or elements that convert to the same stored value +(decided November 12, 2026). A +dict or set whose nested unordered containers are all already in this order is stored +byte-for-byte as before this rule existed, so its hash did not change. **Signed zero.** A float is hashed by its stored bytes, and `0.0` and `-0.0` have different @@ -546,9 +550,18 @@ result. Within one dict or set the two compare equal, so they cannot both be pre contents. A `list>` column that arrives already in Arrow form (an `ArrowTableSource`, a Delta table, rows read back from a store) is read by Python as a `dict`, but at the Arrow level it is an ordered list and its stored order is part of its -hash. Rows written by `0.2.0` from dicts that were not built in key order therefore keep -their old hash when read back; see the `0.3.0` changelog for what that means for an -existing store. A `list[T]` value is never sorted: its order is part of its identity. +hash. A `list[T]` value is never sorted: its order is part of its identity. (Tables written +by `0.2.0`, which stored dicts in insertion order, are not read by v0.3 at all.) + +**Two orderings, two layers.** This section is about the order a dict or set is *stored* in, +which is also the order it is read back in. The semantic hasher (section 1a) separately +orders the *hash input* it builds for a Python `dict` or `set` object, by each item's +canonical JSON text (ITL-942); that order is never stored and never seen. The two rules are +different on purpose (JSON text puts `10` before `2`; storage order is numeric) and neither +is defined in terms of the other: changing the hasher's order would move every semantic +hash, and changing the storage order would move row hashes and what a pod reads back. +`set[T]` of a logical type (`set[Path]`, `set[UUID]`) is ordered by `ListLogicalType`, not +by this rule. --- diff --git a/src/orcapod/semantic_types/universal_converter.py b/src/orcapod/semantic_types/universal_converter.py index 8bf384160..8af468924 100644 --- a/src/orcapod/semantic_types/universal_converter.py +++ b/src/orcapod/semantic_types/universal_converter.py @@ -1690,9 +1690,19 @@ def _convert_datetime(dt: datetime) -> datetime: elif origin is list: element_converter = self.get_python_to_arrow_converter(args[0]) - return lambda value: ( - [element_converter(item) for item in value] if value is not None else [] - ) + + def convert_list(value: Any) -> list[Any]: + """Convert list storage, preserving set order independence on updates.""" + if value is None: + return [] + elements = [element_converter(item) for item in value] + # An Arrow schema remembers a plain set only as a list. Python set + # inputs still need canonical storage order; actual lists keep theirs. + return ( + sort_set_elements(elements) if isinstance(value, set) else elements + ) + + return convert_list elif origin is dict: key_converter = self.get_python_to_arrow_converter(args[0]) diff --git a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md index dc98459d0..653b8b9d8 100644 --- a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md +++ b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md @@ -17,7 +17,7 @@ the end, which supersedes sections 2, 4 and 5 where they differ. > that arrives in Arrow form is an ordered list and its order is part of its hash. So fix 2 > (hash-time sorting) was implemented and then REMOVED. Plain `set[T]` was folded in (separate > commit). In-memory canonical copy (fix 3) and raising on NaN/duplicate keys are kept, -> provisional pending the developer. +> decided by the developer on November 12, 2026. --- diff --git a/superpowers/plans/2026-11-12-itl-905-refresh-against-main.md b/superpowers/plans/2026-11-12-itl-905-refresh-against-main.md new file mode 100644 index 000000000..0f4806101 --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-905-refresh-against-main.md @@ -0,0 +1,63 @@ +# ITL-905: refresh PR #301 against main (November 12, 2026) + +Goal: merge `origin/main` (`b46549fd`, the v0.3 hash step) into the branch so the PR can merge +before the v0.3 tag. No design change: both points were decided by the developer on +November 12, 2026 and stay as implemented. + +## Steps + +1. Merge `origin/main` (merge commit, no rebase). Done: `c0f40fb9`. + Hand-resolved: `src/orcapod/errors.py` (both `CanonicalOrderError` and + `UnsupportedValueError`), `src/orcapod/semantic_types/universal_converter.py` (both imports; + one `except` clause per error type in the per-column wrapper), + `tests/test_hashing/golden/cases.py` (both sides' cases), `CHANGELOG.md` (main's), + `DESIGN_ISSUES.md` (deleted, as on main), golden JSON (main's, regenerated in step 4). +2. `CHANGELOG.md` text -> `changelog.d/ITL-905.md`; `DESIGN_ISSUES.md` entry -> + `design_issues/ITL-905-DO1.md`. Both checks pass. +3. v0.2.0 store tests (`TestV020Store`): v0.3 reads no v0.2 table, so the tests now show that + every input is computed, nothing stored in insertion order is served, and the v0.2 tables + keep their rows. Claims in the changelog fragment, design entry, `docs/reference/hashing.md` + (section renamed 1e; main added its own 1d) and the PR body follow. +4. Golden: regenerate once; report every changed line against main by cause. Expected: two + moved entries (`pipeline:index_pick.index.row[...]` data hashes, dict not in key order) and + added `dict_order` entries. Anything else: stop and ask. +5. Suite on 3.12, 3.11, 3.13; ruff check, ruff format, basedpyright. One push. PR body. +6. CodeRabbit round on the new head; answer every thread and review-body finding. + +## Decisions from the coordinator (November 12) + +- Ordering layers: the semantic hasher's order (canonical JSON text of the hash input, ITL-942) + and `canonical_order.py` (Arrow storage order) are two layers; stated in the PR body and + `docs/reference/hashing.md`. Not unified. +- `ListLogicalType.python_to_storage` (`set[]`) keeps its own `sorted()` / + `repr` fallback (option A). Reported in `worker_done` with measurements; not changed here. + +## Decided November 12, 2026 (unchanged, one decision point each) + +- (a) `Datagram._canonical_dict_values` (`core/datagrams/datagram.py`): key-ordered copies. +- (b) `CanonicalOrderError` raised in `semantic_types/canonical_order.py`; column named by the + callers (`Datagram._canonical_dict_values`, the converter's per-column wrapper). + +## Codex continuation + +- Existing issue, branch, plan and approved design are retained; no Linear changes. +- Fetch confirms main remains `b46549fd`; no second merge is needed. +- Reviewed regenerated goldens against main: exactly two existing pipeline row data hashes + change; four Arrow keys and 27 Datagram keys are added for `dict_order`. +- Finish documentation of the dated decisions, measure logical sets without changing them, + run suites sequentially on 3.12/3.11/3.13, then lint/format/type and fragment checks. +- Update PR body to current v0.3 claims, commit and push once, then check all review threads, + review bodies and checks on the pushed head; route non-trivial findings to coordinator. + +### Approved review continuation (November 12) + +- Dataclass instances remain untouched: add a public characterization test and document + canonical stored values/hashes versus insertion-ordered fresh nested dicts. +- Fix confirmed plain-set updates after Arrow materialization (coordinator approved): + first add failing `Datagram.update` and `with_columns` tests; preserve exact Arrow schema + and list order, then route set inputs at the plain-list conversion boundary through + `sort_set_elements`. Check dict updates at the same boundary. No golden movement expected. +- Initial Python 3.12 suite: 9437 passed, 101 skipped, 19 xfailed, one failure because the + untracked deleted `migrations/` directory retained `__pycache__` and imported as a namespace + package. Preserved that directory under `/tmp/itl905-stale-migrations-20261112`; the + failing old-layout test now passes. Full suite will rerun after the set-update fix. diff --git a/tests/test_hashing/golden/data/arrow.json b/tests/test_hashing/golden/data/arrow.json index 4d704f3c4..f4dad29d8 100644 --- a/tests/test_hashing/golden/data/arrow.json +++ b/tests/test_hashing/golden/data/arrow.json @@ -1,4 +1,8 @@ { + "dict_order.kv.sorted": "arrow_v1:0000025649944353e5f5914b62967bf9d5c31062f149408ccdedc4337265a67561ebe2", + "dict_order.kv.unsorted": "arrow_v1:000002d4f20c961f702578b0f64808c20be5e461fedcdce533ef37aa4bd81b29b6fca7", + "dict_order.pairs.ab": "arrow_v1:00000219aa4c4bd32e76ef40f4e589adcd452740daf41aacd46dc970a97bdd54310d03", + "dict_order.pairs.ba": "arrow_v1:0000022a3837a05e7440f8093807593ef22de9d217dcb6cfb431458789ce7587325d6e", "extension_no_values.all_null.dataframe": "arrow_v1:00000264edcffc26588d0679426c4f16273a466f883933ed7f3834da20d9c5773e4f9a", "extension_no_values.all_null.directory": "arrow_v1:000002453096b2e240ea4cef0a5aea8993c2ee21acc8abb5cb7cd2f5bf2c042698e864", "extension_no_values.all_null.file": "arrow_v1:000002891dfab8553a43b65eb4cd139f5faba11b55d68fa51797a2ebfdaf75f00343f3", diff --git a/tests/test_hashing/golden/data/datagram.json b/tests/test_hashing/golden/data/datagram.json index 961d8181f..cb5da29f6 100644 --- a/tests/test_hashing/golden/data/datagram.json +++ b/tests/test_hashing/golden/data/datagram.json @@ -137,6 +137,42 @@ "_tag_source_id::09e21beb4bc98046eb825d92f94de9a313d22fc6d43d2309ea042381fc78ebb5", "_tag_record_id::09e21beb4bc98046eb825d92f94de9a313d22fc6d43d2309ea042381fc78ebb5" ], + "dict_order.int_keys.content_hash": "semantic_v1:442c4ae4b8fe16767c99b7ef521f7c9919d96ebba255bc32b154c8cd0bc6eb4d", + "dict_order.int_keys.pipeline_hash": "semantic_v1:a0b3c7a7c610bfca1b6b4a0e10cb45cf0f50076df1dca06166ea71c576d0ff42", + "dict_order.int_keys.row[{\"k\":\"r0\"}]._tag_record_id::a0b3c7a7c610bfca1b6b4a0e10cb45cf0f50076df1dca06166ea71c576d0ff42": "a7a545a63e0d5cfdbbb7444d2ff924c4", + "dict_order.int_keys.row[{\"k\":\"r0\"}]._tag_source_id::a0b3c7a7c610bfca1b6b4a0e10cb45cf0f50076df1dca06166ea71c576d0ff42": "golden_dict_int_keys", + "dict_order.int_keys.row[{\"k\":\"r0\"}].data_hash": "arrow_v1:0000020485f86d650cc4b25a2066867c86cef2b910c26cf43f4a8dc31a7347bc50a774", + "dict_order.int_keys.row[{\"k\":\"r1\"}]._tag_record_id::a0b3c7a7c610bfca1b6b4a0e10cb45cf0f50076df1dca06166ea71c576d0ff42": "7e203bc987835298ac9b02baf41ca4cf", + "dict_order.int_keys.row[{\"k\":\"r1\"}]._tag_source_id::a0b3c7a7c610bfca1b6b4a0e10cb45cf0f50076df1dca06166ea71c576d0ff42": "golden_dict_int_keys", + "dict_order.int_keys.row[{\"k\":\"r1\"}].data_hash": "arrow_v1:0000020485f86d650cc4b25a2066867c86cef2b910c26cf43f4a8dc31a7347bc50a774", + "dict_order.int_keys.system_tag_columns": [ + "_tag_source_id::a0b3c7a7c610bfca1b6b4a0e10cb45cf0f50076df1dca06166ea71c576d0ff42", + "_tag_record_id::a0b3c7a7c610bfca1b6b4a0e10cb45cf0f50076df1dca06166ea71c576d0ff42" + ], + "dict_order.nested.content_hash": "semantic_v1:d5697d3ae891cc50f0c17ce018b6189187c86831ef7c5b63aa6e30312496dd38", + "dict_order.nested.pipeline_hash": "semantic_v1:62e571e4f5d10fc56473c422811afa759f1f14eff284c6991cfc9dc61f92c0cf", + "dict_order.nested.row[{\"k\":\"r0\"}]._tag_record_id::62e571e4f5d10fc56473c422811afa759f1f14eff284c6991cfc9dc61f92c0cf": "f2163b15031857e7b0edc5c8092fe897", + "dict_order.nested.row[{\"k\":\"r0\"}]._tag_source_id::62e571e4f5d10fc56473c422811afa759f1f14eff284c6991cfc9dc61f92c0cf": "golden_dict_nested", + "dict_order.nested.row[{\"k\":\"r0\"}].data_hash": "arrow_v1:0000023adae205c66a7e47e30426b0b33eb081c7f899b5f12c41bec79437372e0de449", + "dict_order.nested.row[{\"k\":\"r1\"}]._tag_record_id::62e571e4f5d10fc56473c422811afa759f1f14eff284c6991cfc9dc61f92c0cf": "f6ab4ac474e750cb89debd9cb2697ccb", + "dict_order.nested.row[{\"k\":\"r1\"}]._tag_source_id::62e571e4f5d10fc56473c422811afa759f1f14eff284c6991cfc9dc61f92c0cf": "golden_dict_nested", + "dict_order.nested.row[{\"k\":\"r1\"}].data_hash": "arrow_v1:0000023adae205c66a7e47e30426b0b33eb081c7f899b5f12c41bec79437372e0de449", + "dict_order.nested.system_tag_columns": [ + "_tag_source_id::62e571e4f5d10fc56473c422811afa759f1f14eff284c6991cfc9dc61f92c0cf", + "_tag_record_id::62e571e4f5d10fc56473c422811afa759f1f14eff284c6991cfc9dc61f92c0cf" + ], + "dict_order.str_keys.content_hash": "semantic_v1:c5cd4d056d1fe1a459f367e802e2d71f9f6b2993cce05f3c861c4d636df0c084", + "dict_order.str_keys.pipeline_hash": "semantic_v1:41ec07f117de7f219486b87a87f98d4b9f15a6363958fb1d0d1827d134fab3ef", + "dict_order.str_keys.row[{\"k\":\"r0\"}]._tag_record_id::41ec07f117de7f219486b87a87f98d4b9f15a6363958fb1d0d1827d134fab3ef": "bf6c81a8f6a75d7a9615d4f2034b772f", + "dict_order.str_keys.row[{\"k\":\"r0\"}]._tag_source_id::41ec07f117de7f219486b87a87f98d4b9f15a6363958fb1d0d1827d134fab3ef": "golden_dict_str_keys", + "dict_order.str_keys.row[{\"k\":\"r0\"}].data_hash": "arrow_v1:0000028fbf610bc86bde65d7d10b0c2e017772a9fca384042dd68cbc5ff14413238ad6", + "dict_order.str_keys.row[{\"k\":\"r1\"}]._tag_record_id::41ec07f117de7f219486b87a87f98d4b9f15a6363958fb1d0d1827d134fab3ef": "2cec7386d5645ec48bf8dfa5d661196a", + "dict_order.str_keys.row[{\"k\":\"r1\"}]._tag_source_id::41ec07f117de7f219486b87a87f98d4b9f15a6363958fb1d0d1827d134fab3ef": "golden_dict_str_keys", + "dict_order.str_keys.row[{\"k\":\"r1\"}].data_hash": "arrow_v1:0000028fbf610bc86bde65d7d10b0c2e017772a9fca384042dd68cbc5ff14413238ad6", + "dict_order.str_keys.system_tag_columns": [ + "_tag_source_id::41ec07f117de7f219486b87a87f98d4b9f15a6363958fb1d0d1827d134fab3ef", + "_tag_record_id::41ec07f117de7f219486b87a87f98d4b9f15a6363958fb1d0d1827d134fab3ef" + ], "list_path.content_hash": "semantic_v1:5713235dbd02b9580552fbde91eeb4ac38371f74366a000eac2fc495aa6b42ad", "list_path.pipeline_hash": "semantic_v1:50abad03423f579ea55cada3560e5009a9d9cfb5110be8b95336aaa41d310397", "list_path.row[{\"k\":\"r1\"}]._tag_record_id::50abad03423f579ea55cada3560e5009a9d9cfb5110be8b95336aaa41d310397": "059c77251e845bafb5a3839bf547a2f5", diff --git a/tests/test_hashing/golden/data/pipeline.json b/tests/test_hashing/golden/data/pipeline.json index 84d3297bc..b08876711 100644 --- a/tests/test_hashing/golden/data/pipeline.json +++ b/tests/test_hashing/golden/data/pipeline.json @@ -236,10 +236,10 @@ "index_pick.index.pipeline_hash": "semantic_v1:e70ee3e32ddb810c39dae31d3b178554a10df387b728ef319d08fb547a41f8d0", "index_pick.index.row[{\"subject\":\"s1\"}]._tag_record_id::e668b355366a549312dd441a27ad7e566ddaee4f7e4ab754335a277b51ce5175": "dd65b531e3d957a0ae3f2fba11e021d5", "index_pick.index.row[{\"subject\":\"s1\"}]._tag_source_id::e668b355366a549312dd441a27ad7e566ddaee4f7e4ab754335a277b51ce5175": "lists", - "index_pick.index.row[{\"subject\":\"s1\"}].data_hash": "arrow_v1:0000025f5739a1fc427e583ba103ab5a7c1a5b868aadb554a722a1c0bd1d2081e4a34f", + "index_pick.index.row[{\"subject\":\"s1\"}].data_hash": "arrow_v1:000002948e686a8a041a7323c3474c6d7861e4536e57ddb8096704f395daecca80e742", "index_pick.index.row[{\"subject\":\"s2\"}]._tag_record_id::e668b355366a549312dd441a27ad7e566ddaee4f7e4ab754335a277b51ce5175": "00c12388e8c4522893b029b39a8ec7aa", "index_pick.index.row[{\"subject\":\"s2\"}]._tag_source_id::e668b355366a549312dd441a27ad7e566ddaee4f7e4ab754335a277b51ce5175": "lists", - "index_pick.index.row[{\"subject\":\"s2\"}].data_hash": "arrow_v1:000002d9f79b11b0d6928476289f10dc1ffce27ef6af09affa82b079560b96435f62eb", + "index_pick.index.row[{\"subject\":\"s2\"}].data_hash": "arrow_v1:000002ecef3b9df41f6928426d3a063d9eb64485f7e78fc72ef50dad5b3c06bd8aee3a", "index_pick.index.system_tag_columns": [ "_tag_source_id::e668b355366a549312dd441a27ad7e566ddaee4f7e4ab754335a277b51ce5175", "_tag_record_id::e668b355366a549312dd441a27ad7e566ddaee4f7e4ab754335a277b51ce5175" diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index fea3b4e79..74a69e523 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -230,6 +230,25 @@ def test_dict_inside_dataclass_field_reordered(self): second = Holder("h", {"b": 2, "a": 1}) assert _row_hash(first, Holder) == _row_hash(second, Holder) + def test_dataclass_dict_storage_is_canonical_but_fresh_object_is_unchanged(self): + """User objects keep their dict order; stored and read-back values are canonical.""" + original = Holder("h", {"b": 2, "a": 1}) + # Registered dataclasses work at runtime but are absent from DataValue's union. + fresh_values: dict[str, Any] = {"m": original} + ordered_values: dict[str, Any] = {"m": Holder("h", {"a": 1, "b": 2})} + fresh = Datagram(fresh_values, python_schema={"m": Holder}) + ordered = Datagram(ordered_values, python_schema={"m": Holder}) + assert fresh.as_dict()["m"] is original + assert list(original.counts) == ["b", "a"] + assert fresh.content_hash() == ordered.content_hash() + assert fresh.as_table().equals(ordered.as_table()) + + restored = Datagram(fresh.as_table()).as_dict()["m"] + assert isinstance(restored, Holder) + assert list(restored.counts) == ["a", "b"] + assert restored == original + assert list(original.counts) == ["b", "a"] + # --------------------------------------------------------------------------- # Arrow tables that already hold key/value lists @@ -347,6 +366,28 @@ async def test_async_output_matches_sync_and_is_order_independent(self): class TestDatagramColumnUpdates: """``update`` and ``with_columns`` hold dict values in canonical key order.""" + @pytest.mark.parametrize("operation", ["update", "with_columns"]) + def test_arrow_backed_dict_write_keeps_schema_and_canonical_order( + self, operation: str + ): + """Dict writes through a known Arrow field retain its schema and canonical keys.""" + original = Datagram({"m": {"a": 1}}, python_schema={"m": dict[str, int]}) + field = original.as_table().schema.field("m") + value = {"b": 2, "a": 1} + if operation == "update": + result = original.update(m=value) + name = "m" + else: + result = original.with_columns(column_types={"added": field}, added=value) + name = "added" + assert ( + result.as_table() + .schema.field(name) + .equals(field.with_name(name), check_metadata=True) + ) + assert _keys(result.as_dict()[name]) == ["a", "b"] + assert list(value) == ["b", "a"] + def test_update_holds_dict_in_key_order(self): """A replaced dict value iterates in key order, like a constructed one.""" datagram = Datagram({"m": {"a": 1}}, python_schema={"m": dict[str, int]}) @@ -457,11 +498,16 @@ class TestV020Store: """A Delta result store written by the released v0.2.0 code. ``data/v0_2_0_dict_store.tar.gz`` was produced by v0.2.0 running ``total`` over two - inputs: ``k=unsorted`` holding ``{"b": 2, "a": 1}`` (stored in that order, so its - v0.2.0 input hash is order dependent) and ``k=sorted`` holding ``{"a": 1, "c": 3}`` - (already in key order, so its hash does not move); and ``tally`` over ``n=5``, - whose output ``{"z": 5, "a": 5}`` is stored in insertion order. ``data/generate_v0_2_0_dict_store.py`` - regenerates it from a checkout of the ``v0.2.0`` tag. + inputs: ``k=unsorted`` holding ``{"b": 2, "a": 1}`` (stored in that order) and + ``k=sorted`` holding ``{"a": 1, "c": 3}``; and ``tally`` over ``n=5``, whose output + ``{"z": 5, "a": 5}`` is stored in insertion order. + ``data/generate_v0_2_0_dict_store.py`` regenerates it from a checkout of the + ``v0.2.0`` tag. + + v0.3 reads no table v0.2 wrote (new table layouts and paths, ITL-944/ITL-945), so + no row stored in v0.2's insertion order is ever served: every input is computed + again, whether its dict was in key order or not, and the v0.2 tables stay as they + were. """ @pytest.fixture @@ -485,12 +531,26 @@ def _inputs(self) -> DictSource: source_id="itl905_v020", ) - def test_unsorted_row_recomputes_once_and_old_rows_stay(self, store): - """An out-of-order v0.2.0 input recomputes once; its old row stays.""" + @staticmethod + def _row_counts( + store: DeltaTableDatabase, paths: list[tuple[str, ...]] + ) -> dict[tuple[str, ...], int]: + """Return the number of rows in each of ``paths``.""" + counts = {} + for path in paths: + records = store.get_all_records(path) + counts[path] = 0 if records is None else records.num_rows + return counts + + def test_v020_rows_are_not_read_and_stay(self, store): + """Every input is computed once; the v0.2.0 tables are left as they were.""" + old_paths = store.list_table_paths(("total",)) + old_counts = self._row_counts(store, old_paths) + assert sorted(old_counts.values()) == [2] + pod = CachedFunctionPod(_pod(total, "total"), result_database=store) - before = store.get_all_records(pod.record_path) - assert before is not None - assert before.num_rows == 2 + assert pod.record_path not in old_paths + assert store.get_all_records(pod.record_path) is None results = { tag.as_dict()["k"]: out @@ -498,12 +558,14 @@ def test_unsorted_row_recomputes_once_and_old_rows_stay(self, store): } store.flush() - assert results["sorted"].get_meta_value(pod.RESULT_COMPUTED_FLAG) is False + # Neither the key-ordered nor the out-of-order input is served from v0.2.0. + assert results["sorted"].get_meta_value(pod.RESULT_COMPUTED_FLAG) is True assert results["unsorted"].get_meta_value(pod.RESULT_COMPUTED_FLAG) is True assert results["unsorted"].as_dict()["total"] == 3 - after = store.get_all_records(pod.record_path) - assert after is not None - assert after.num_rows == 3 # both old rows kept, one new row + assert self._row_counts(store, old_paths) == old_counts + new_records = store.get_all_records(pod.record_path) + assert new_records is not None + assert new_records.num_rows == 2 again = { tag.as_dict()["k"]: out @@ -514,27 +576,29 @@ def test_unsorted_row_recomputes_once_and_old_rows_stay(self, store): for out in again.values() ) - def test_cached_unsorted_dict_output_is_served_as_stored(self, store): - """A cached v0.2.0 dict output is served in its stored order. - - An Arrow table is never re-sorted, so a cache hit returns the old insertion order - and hash until the entry is recomputed. - """ + def test_v020_dict_output_in_insertion_order_is_never_served(self, store): + """A dict output v0.2.0 stored out of key order is recomputed, not served.""" inputs = DictSource( data=[{"k": "r", "n": 5}], tag_columns=["k"], data_schema={"k": str, "n": int}, source_id="itl905_v020_int", ) + old_paths = store.list_table_paths(("tally",)) + [old_path] = old_paths + old_row = store.get_all_records(old_path) + assert old_row is not None + # The released store holds the output in insertion order. + assert [e["key"] for e in old_row.column("counts")[0].as_py()] == ["z", "a"] + pod = CachedFunctionPod(_pod(tally, "counts"), result_database=store) + [(_, computed)] = list(pod.process(inputs).iter_data()) + store.flush() [(_, cached)] = list(pod.process(inputs).iter_data()) - fresh_pod = CachedFunctionPod( - _pod(tally, "counts"), result_database=InMemoryArrowDatabase() - ) - [(_, fresh)] = list(fresh_pod.process(inputs).iter_data()) + assert computed.get_meta_value(pod.RESULT_COMPUTED_FLAG) is True assert cached.get_meta_value(pod.RESULT_COMPUTED_FLAG) is False - assert _keys(cached.as_dict()["counts"]) == ["z", "a"] - assert _keys(fresh.as_dict()["counts"]) == ["a", "z"] - assert cached.as_dict() == fresh.as_dict() - assert cached.content_hash() != fresh.content_hash() + assert _keys(computed.as_dict()["counts"]) == ["a", "z"] + assert _keys(cached.as_dict()["counts"]) == ["a", "z"] + assert cached.content_hash() == computed.content_hash() + assert self._row_counts(store, old_paths) == {old_path: 1} diff --git a/tests/test_hashing/test_set_order_hash.py b/tests/test_hashing/test_set_order_hash.py index b6590f841..ffab2dd8d 100644 --- a/tests/test_hashing/test_set_order_hash.py +++ b/tests/test_hashing/test_set_order_hash.py @@ -14,8 +14,10 @@ from typing import Any import numpy as np +import pyarrow as pa import pytest +from orcapod.core.datagrams import Datagram from orcapod.sources import DictSource _ROW_HASH_SCRIPT = textwrap.dedent( @@ -91,3 +93,44 @@ def test_numpy_nan_element_is_rejected(self): tag_columns=["k"], data_schema={"k": str, "s": set[float]}, ).as_table() + + @pytest.mark.parametrize("operation", ["update", "with_columns"]) + def test_arrow_backed_set_write_preserves_order_and_schema(self, operation: str): + """Set writes through an Arrow list schema sort without changing its fields.""" + field = pa.field( + "s", + pa.large_list(pa.field("item", pa.int16(), nullable=False)), + nullable=False, + metadata={b"purpose": b"set-order"}, + ) + original = Datagram( + pa.Table.from_arrays( + [pa.array([[1]], type=field.type)], schema=pa.schema([field]) + ) + ) + value = {300, -1, 7} + if operation == "update": + result = original.update(s=value) + result_field = field + else: + result = original.with_columns(column_types={"added": field}, added=value) + result_field = field.with_name("added") + table = result.as_table() + assert table.schema.field(result_field.name).equals( + result_field, check_metadata=True + ) + assert table.column(result_field.name).to_pylist() == [[-1, 7, 300]] + assert original.as_table().column("s").to_pylist() == [[1]] + + def test_materialized_set_update_keeps_fresh_hash(self): + """Updating a materialized set preserves the hash of an equal fresh value.""" + value = {300, -1, 7} + fresh = Datagram({"s": value}, python_schema={"s": set[int]}) + before = fresh.content_hash() + assert fresh.update(s=value).content_hash() == before + + def test_arrow_backed_list_update_keeps_list_order(self): + """An ordered list remains ordered as supplied at the same boundary.""" + original = Datagram({"s": [1]}, python_schema={"s": list[int]}) + original.as_table() + assert original.update(s=[300, -1, 7]).as_dict()["s"] == [300, -1, 7] From 37c89138a42de133c77a6f069eeebf0495e69f30 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 06:47:41 -0800 Subject: [PATCH 21/22] docs(hashing): record refresh decisions and validation Co-Authored-By: GPT-6 --- docs/reference/hashing.md | 4 ++-- ...2026-11-12-itl-905-refresh-against-main.md | 24 +++++++++++++++++++ 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index cc4a3d4c1..d734e1508 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -558,8 +558,8 @@ which is also the order it is read back in. The semantic hasher (section 1a) sep orders the *hash input* it builds for a Python `dict` or `set` object, by each item's canonical JSON text (ITL-942); that order is never stored and never seen. The two rules are different on purpose (JSON text puts `10` before `2`; storage order is numeric) and neither -is defined in terms of the other: changing the hasher's order would move every semantic -hash, and changing the storage order would move row hashes and what a pod reads back. +is defined in terms of the other: changing the hasher's order would move semantic hashes +of affected inputs, and changing storage order would move row hashes and what a pod reads back. `set[T]` of a logical type (`set[Path]`, `set[UUID]`) is ordered by `ListLogicalType`, not by this rule. diff --git a/superpowers/plans/2026-11-12-itl-905-refresh-against-main.md b/superpowers/plans/2026-11-12-itl-905-refresh-against-main.md index 0f4806101..a3addb58b 100644 --- a/superpowers/plans/2026-11-12-itl-905-refresh-against-main.md +++ b/superpowers/plans/2026-11-12-itl-905-refresh-against-main.md @@ -61,3 +61,27 @@ November 12, 2026 and stay as implemented. untracked deleted `migrations/` directory retained `__pycache__` and imported as a namespace package. Preserved that directory under `/tmp/itl905-stale-migrations-20261112`; the failing old-layout test now passes. Full suite will rerun after the set-update fix. + +### Validation checkpoint + +- Code/docs committed as `a32387ce`; no push yet. +- Three public set-write regressions failed before the boundary fix and now pass. +- Focused dict/set/golden suite: 1145 passed; no additional golden changes. +- Full Python 3.12.15 suite: 9445 passed, 101 skipped, 19 xfailed, exit 0. +- Ruff check/format, changelog/design checks pass; basedpyright exits 0 (0 errors). +- Logical-set measurement: ten processes (seeds 0,1,2,42,random, twice each) agree. + UUID bytes and Path strings use natural sorting and match `sort_set_elements`; + frozen dataclass str/int fields use repr-sorted dict storage, rejected by the helper. +- Python 3.11 first run: 9440 passed with five packaging fixture errors from sandbox DNS; + full rerun uses ordinary `uv run --python 3.11` with network permission. +- Scratch reports/logs are in `.untracked/`; the preserved stale migrations bytecode directory + was moved there too. Developer withdrew the temporary custom-cache/environment instruction; + subsequent runs use ordinary uv and the worktree's `.venv`. +- Coordinator approved replies: qualify nested-canonical unchanged-hash wording; document and + defer user-class reconstruction; fix Arrow-backed set updates. Twelve older threads already + have signed answers; two latest threads and two review-body set-update findings await replies. +- Full ordinary-uv Python 3.11.18 rerun: 9445 passed, 101 skipped, 19 xfailed, + exit 0 (270.52 seconds); packaging passed. Python 3.13 full suite now running. + +- Full ordinary-uv Python 3.13.18 suite: 9445 passed, 101 skipped, 19 xfailed, + exit 0 (282.25 seconds). All three interpreter suites are green. From f4897ad11e536e36b969351762d73d846ca226ab Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 07:02:49 -0800 Subject: [PATCH 22/22] docs(hashing): align changelog and plan with final behavior Co-Authored-By: GPT-6 --- changelog.d/ITL-905.md | 7 ++++--- superpowers/plans/2026-11-11-itl-905-dict-order-hash.md | 4 ++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/changelog.d/ITL-905.md b/changelog.d/ITL-905.md index efe9f58c3..1546fb79a 100644 --- a/changelog.d/ITL-905.md +++ b/changelog.d/ITL-905.md @@ -11,9 +11,10 @@ value becomes Arrow; see the hashing reference, section 1e, for the order. A NaN element, values that cannot be compared, and two keys that convert to the same stored value raise `CanonicalOrderError` (a `ValueError`) naming the column. A datagram built from Python values (and values set with `update` or `with_columns`) holds its dicts in the same -order, so a newly computed pod output and one served from the cache iterate alike. An Arrow -table is still hashed exactly as stored: a key/value list that arrives in Arrow form is never -re-sorted. +order, so a newly computed pod output and one served from the cache iterate alike. Dicts +nested in dataclass or pydantic instances are the exception: a fresh instance keeps insertion +order, while a read-back instance uses key order. An Arrow table is still hashed exactly as +stored: a key/value list that arrives in Arrow form is never re-sorted. Set values passed to `update` after Arrow materialization, or to `with_columns` with an explicit Arrow list field, are also stored in canonical order. The existing Arrow field diff --git a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md index 653b8b9d8..0b35c30fe 100644 --- a/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md +++ b/superpowers/plans/2026-11-11-itl-905-dict-order-hash.md @@ -139,7 +139,7 @@ Golden suite (`tests/test_hashing/golden`): * `DictSource` with `data_schema={"m": dict[str, int]}` → equal. * function pod `-> dict[str, int]` returning the two orders → equal output hashes; sync and async. * nested `dict[str, dict[str, int]]`, `list[dict[str, int]]`, `dict[str, list[dict[str,int]]]` reordered at inner and outer level → equal. - * `ArrowTableSource` with `large_list>` in both orders → equal; and equal to the `DictSource` hash. + * `ArrowTableSource` with `large_list>` in both orders → different, because Arrow lists hash as stored. Only the key-ordered list matches the `DictSource` hash. * key types: int (`{10:..., 9:...}`), bool, bytes, float incl. `-0.0`, date, aware datetime, tuple keys, Optional key with `None` → reorder gives equal hash. * NaN key → `ValueError` naming the column; Arrow-native duplicate key → `ValueError` naming the column. * `list[tuple[str,int]]` (ordered pairs) in two orders → hashes **differ** (non-dict lists are never sorted). @@ -220,7 +220,7 @@ Commits on `eywalker/itl-905-a-dict-values-insertion-order-changes-its`: (`arrow:dict_order`, `datagram:dict_order`). 6. plain `set[T]` sorted at conversion (+ module/error rename). 7. revert hash-time sort; Arrow-native unsorted vs sorted pinned as DIFFERENT. -8. v0.2.0 fixture extended with a dict-returning pod: cache hit serves the stored order/hash. +8. v0.2.0 fixture extended with a dict-returning pod: v0.3 does not serve v0.2.0 rows. Every input is recomputed, and the v0.2.0 tables remain unchanged. 9. docs (hashing reference 1c), CHANGELOG `[0.3.0]`, DESIGN_ISSUES DO1. Correction to section 5: the golden suite DID hold one out-of-order dict