diff --git a/.zed/rules b/.zed/rules index 0c226b8ac..6fbbc92d4 100644 --- a/.zed/rules +++ b/.zed/rules @@ -384,7 +384,7 @@ Prefixes are computed from SystemConstant in system_constants.py. directly to derive record identity. They fold to a deterministic digest via arrow_utils.fold_system_tag_values and their column name gains ::{pipeline_hash}. - The fold is SHA-based (uuid5 for record_id, combine_hashes for source_id) and so is stable + The fold is SHA-based (uuid5 for record_id, SHA-256 of the canonical JSON member list for source_id) and so is stable across processes. Never use hash() or a set-based construction there: the digest becomes a cache key, so a per-process value would look correct in a single-process test and miss the cache on every new driver run. diff --git a/CHANGELOG.md b/CHANGELOG.md index dfdc26d3b..4964e6126 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,11 @@ #### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883) **The array format and every array hash changed in v0.3.** A store written by v0.2 that holds -`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays -computes again into new tables, and its v0.2 results are no longer returned. Nothing written by -v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for -v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889). +`numpy.ndarray` columns is **not migrated**. Every node that handles arrays gets new tables, +so its v0.2 results are no longer found; and since every Arrow-derived hash changed too +(ITL-910, below), no v0.2 result is reused at all: v0.3 ignores the rows a v0.2 store holds and +computes every input again. Nothing written by v0.2 is changed or deleted, and those tables stay +readable with v0.2. A migration is planned after v0.3 (ITL-889). - **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata `{"version":1}`) over `struct, @@ -29,8 +30,10 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0. change too, and so do the table paths of those nodes. Pipeline hashes are chained, so the pipeline and operator tables of every node downstream of an array column move as well, even when it has no array column itself. A pod whose inputs and outputs hold no array - keeps its function result table, so it reuses its v0.2 results without computing again. - Nodes with no array column upstream do not move. + keeps its function result table, and nodes with no array column upstream keep their tables; + but an array-free pod does **not** reuse its v0.2 results: every Arrow-derived hash changed + as well (ITL-910, below), so the v0.2 rows in such tables are ignored, the inputs are computed + again, and the new rows are stored next to the old ones. - **Refused at write time** with `orcapod.errors.UnsupportedValueError` (a `ValueError`), naming the column and dtype: masked arrays (the mask used to be dropped silently), object dtypes, NumPy 2 `StringDType` (used to fail with a misleading pickle error), `longdouble` and @@ -47,6 +50,111 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0. the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its subclasses, whatever the registration order. +#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910) + +**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1` +to `arrow_v0.2`.** starfix 0.4 (hash format 1) lost information for some inputs, so different +data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and +`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean +columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash +format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept. + +- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default + `source_id`s and the `record_id`s and source content hashes built from them, function + input-data hashes, pipeline base and entry ids, operator and cached-source record hashes, + invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The + hash label in front of each (`arrow_v0.2:000002...`) shows the new format. Pipeline hashes, + schema hashes, system-tag column names and table paths under the default + `table_scope="pipeline_hash"` do not move. Paths that embed a content hash + (`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or + `Batch` orders members by `record_id`, the order of members can change too. +- **Results stored by 0.2.0 are ignored and computed again; they are not reused, changed or + deleted.** Rows a v0.2 store holds are keyed under `arrow_v0.1` and can never match v0.3 + inputs. v0.3 does not refuse such a store. A value-only change does not move a table, so a + v0.3 job writes its `arrow_v0.2` rows into the same tables, next to the `arrow_v0.1` rows, + and uses rows of its own label only: + - The first v0.3 run on a v0.2 store **computes every input again**. Nothing stored by v0.2 + counts as a cached result, also for functions and data that did not change. + - Reading a node (`as_table()`, `iter_data()`, `get_all_records()`, `as_source()`, also on a + job loaded from a file, including a file saved by v0.2), `get_all_cached_outputs()` of a + cached function, a stored (`LOG` / `REPLAY`) operator node and a `CachedSource` return + rows hashed under `arrow_v0.2` only. Before the first v0.3 run, such a read on a v0.2 + store returns no rows; after it, one row per input, never the v0.2 rows as well. + - The v0.2 rows stay where they are, byte for byte, and stay readable with v0.2. Each + stored key names its label (`arrow_v0.1:000001...` or `arrow_v0.2:000002...`), and + `orcapod.hashing.stored_hash_labels.stored_hash_labels(store, record_path)` counts the + rows of a table by label, for example `{"arrow_v0.1": 3, "arrow_v0.2": 3}`. + - When a read leaves out rows of another label, Orcapod logs one warning per table and + reader with the total, on the logger `orcapod.hashing.stored_hash_labels`: + `Skipped 3 rows of stored table 'my_job/.../pdb_v1': written under another hash label + than the current one. The rows are left in place unchanged and can be migrated later.` + - **Side-effect nodes are the exception: they fail instead of delivering again.** A + side-effect pod finds a delivery it already made by an Arrow-derived record id, so + deliveries logged by v0.2 cannot be found, and running the node would deliver every side + effect a second time. A side effect is an external action, not a value that can be + computed again, so a side-effect node whose invocation log holds deliveries logged under + another label raises `orcapod.errors.SideEffectLogHashLabelError` (a `StoreError`, so it + fails the run whatever the `error_policy`) before it delivers or writes anything. The + error names the log, the label found and the current one. Function, operator and source + nodes of the same job are not refused. Use a fresh store for the side-effect node, or + keep using v0.2 for it, until the log can be migrated. + + **Migration:** none is needed to run the function, operator and source nodes of a v0.2 + store under v0.3, at the price of computing everything again and of tables that hold both + generations; side-effect nodes need a fresh store. A tool that converts + `arrow_v0.1` rows into `arrow_v0.2` rows is planned (ITL-913, ITL-914); until then the old + rows are kept for it. Operator-node and cached-source tables store their record hashes + without a label; there the rows of v0.2 are told apart by the starfix hash format at the + start of each record hash (`000001` for v0.2, `000002` for v0.3). That is exact for these + two versions, but a later Arrow hash label that keeps starfix hash format 2 could not be + told apart in those two kinds of table. A table that `table_scope="content_hash"` or a + `CachedSource` places under a content hash moves to a new path instead, so its v0.2 rows + are in another table. +- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator + carrying one works, and its type is kept. A `fixed_size_list[n]` hashes differently from a + `large_list` with the same values. pyarrow's canonical extension types (`arrow.uuid`, + `arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid` + column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and + `StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846). +- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed + silently: a non-nullable field holding a null; a zero-row table with a `map` column; two + schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a + column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`, + interval, `string_view` and `binary_view` columns, union and run-end-encoded types are + still refused, now with `ValueError` instead of `NotImplementedError`. +- Column order and struct field order are still not part of an Arrow hash. + +#### Wrong hashes fixed: values that collided or depended on data, process or spelling (ITL-942) + +A hash-site audit (ITL-938) found hashes that were **wrong**: two different things shared a +hash, or one thing hashed differently depending on its data, the process or how it was +written. A wrong hash is a bug that overrides hash stability, so these values move. Each fix +changes only the values that were wrong. + +| Fix | Before | Now | What moves | +|---|---|---|---| +| Batch / GroupBy source-id fold | members concatenated with no separator: `["ab", "c"]` folded like `["a", "bc"]`, and a `None` member like `""` | SHA-256 of the canonical JSON list of members, in order | the `_tag_source_id` value of every row after a `Batch` or `GroupBy`, and the entry ids downstream of it. Column names and paths do not move. Source-info columns are unchanged | +| Arrow hashing of nested types | the hashed type was rebuilt from the first non-null value, so nested nullability counted for an empty or all-null column and not for a column with data | the hashed type is the declared type: struct and list fields keep their nullability, at every depth, with or without data | row hashes of struct or list columns with non-nullable nested fields, and of all-null or zero-row `Path`, `File`, `Directory` and `UUID` columns. Key values only | +| Set order in structure hashes | items sorted by `str()`, so `{1, "1"}` hashed differently per process (`PYTHONHASHSEED`) | items sorted by their canonical JSON text | structure hashes of sets whose `str()` order differs from their JSON order (mixed types, sets of tuples) | +| Dict keys in structure hashes | keys turned into `str()`: `{1: "x"}` equalled `{"1": "x"}`, and `{1: "a", "1": "b"}` lost a key | a dict with only `str` keys is unchanged; any other dict hashes as a sorted list of `[key, value]` pairs | structure hashes of dicts with non-`str` keys | +| Dicts that spell out an encoding | `{"__type__": "tuple", "items": [1, 2]}` hashed like `(1, 2)` (also set, namedtuple, generic alias, union) | only the hasher and type handlers write `"__type__"` objects; a dict with a `"__type__"` key hashes as a pair list | structure hashes of dicts with a `"__type__"` key. Handlers that returned a plain `{"__type__": ...}` dict must return `orcapod.hashing.semantic_hashing.Encoded(tag, fields)` | +| Namedtuple field order | fields sorted by name: `Point(x=1, y=2)` hashed like a `Point` declared `(y, x)` holding `y=2, x=1`, although the tuples differ | fields hashed in declared order | structure hashes of namedtuples | +| Column selection operators | `SelectTagColumns`, `SelectDataColumns`, `DropTagColumns`, `DropDataColumns` hashed `["x"]`, `("x",)`, `{"x"}`, `["x", "x"]` differently, and `["y", "x"]` unlike `["x", "y"]`, although the output is the same | identity is the set of names, hashed as a sorted list | pipelines that spelled the columns as a tuple, a set, or an unsorted or repeating list: their hashes **and table paths** move. Sorted lists (the usual spelling) keep their hashes | +| Record-id system tag type | `fixed_size_binary[16]` out of a source, `large_binary` after a `Tag`, so one row had two operator-cache keys (`cache_mode` LOG/REPLAY, sync run versus async orchestrator) | `large_binary` everywhere | operator-cache and cached-source row keys, and the content hash of a raw stream holding source system tags. On Delta Lake the stored column type does not change: Delta has a single binary type | +| Default function version | `v0.0` through `@function_pod`, `v1.0` through `FunctionPod.from_fn` and `side_effect_function_pod` | `v0.0` for every entry point | **table paths** of pods built with `FunctionPod.from_fn` or `side_effect_function_pod` without an explicit version: from `.../v1/...` to `.../v0/...` | + +**Migration.** A pod built with `FunctionPod.from_fn` or `side_effect_function_pod` that relied +on the old default keeps its tables by passing `version="v1.0"`. A pipeline that spelled +selected columns as a tuple or a set gets new tables for that operator and everything +downstream; no stored row is changed or deleted. Every other fix changes values only, never a +path or a stored schema. + +Reviewed and kept by decision: `MergeJoin(m1, m1)` (a stream joined with itself) has a +different pipeline hash from `MergeJoin(m1, m2)` with the same schema, because a pipeline hash +covers the pipeline's structure; two inputs with the same source identity count as one. +Function default values in the function-value hash are tracked in ITL-953. See +`docs/reference/hashing.md`. + #### A changed column type is refused when appending to a Delta table (ITL-884) Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows' @@ -80,6 +188,13 @@ stay queued for the next flush. A flush that fails only for other reasons still the backend's exception as its cause. (ITL-884 #290) - **`orcapod.errors.StoreError`** — the base of `StoreReadError` and `StoreWriteError`; `except StoreError` handles both. (ITL-884 #290) +- **`orcapod.errors.SideEffectLogHashLabelError`** — a `StoreError` raised by a side-effect + node whose invocation log holds deliveries logged under another Arrow hash label (a log + written by v0.2), before anything is delivered. It has `record_path`, `found_label`, + `current_label` and `node_label`. (ITL-910) +- **`orcapod.hashing.stored_hash_labels.stored_hash_labels(database, record_path)`** — counts + the rows of a stored table by the Arrow hash label they were hashed under, to tell rows + written by v0.2 (`arrow_v0.1`) from rows written by v0.3 (`arrow_v0.2`). (ITL-910) ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 1908ec144..267a014ea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -417,8 +417,8 @@ Prefixes are computed from `SystemConstant` in `system_constants.py`. The `const to a deterministic digest via `arrow_utils.fold_system_tag_values` and their column name gains `::{pipeline_hash}` — a blend of rules 2 and 3. - The fold is SHA-based (`uuid5` for `record_id`, `combine_hashes` for `source_id`) and so is - stable across processes. Never use `hash()` or a set-based construction there: the digest + The fold is SHA-based (`uuid5` for `record_id`, SHA-256 of the canonical JSON member list + for `source_id`) and so is stable across processes. Never use `hash()` or a set-based construction there: the digest becomes a cache key, so a per-process value would look correct in a single-process test and miss the cache on every new driver run. diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 695708a20..2fa9b88c9 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2442,3 +2442,106 @@ results. Known compromise: the struct's children are nullable at the Arrow level, because Polars drops nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null instead. Tighten under a new storage version once nested nullability survives Polars. + +--- + +## `src/orcapod/hashing/arrow_hashers.py` — Arrow hash format (starfix) + +### HS2 — starfix hash format 1 collided on nested lists, null structs and Booleans +**Status:** resolved +**Severity:** high +**Issue:** ITL-910 (fixes the starfix side of ITL-848; ITL-927) + +`StarfixArrowHasher` used starfix 0.4 (hash format 1, label `arrow_v0.1`), which lost +information for some inputs, so different data reached the same row hash: nested-list lengths +(`[[[1]], [[2], [3]]]` and `[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are +all null, and non-nullable Boolean columns (`[True]` and `[True, False, False]`); a Boolean +column's hash also depended on chunking. starfix 0.4 also could not hash `fixed_size_list` +data, so a source with such a column failed at construction (ITL-927). + +**Fix:** starfix 0.5 (hash format 2, prefix `00 00 02`) under the new label `arrow_v0.2` +(developer's label rule: the label moves whenever any table hash can change). Every +Arrow-derived hash moves; the golden files were regenerated once, explicitly. Order options +stay off. pyarrow's canonical extension types and `hash_schema` on extension columns hash with +their extension identity instead of as their storage type. + +Stored rows: under the default `table_scope`, table paths do not move (developer, November +11, 2026: only a change of a table's schema goes into its path; a value-only change stays in +the same table), so a v0.3 node writes into the same tables as a v0.2 store. The first decision +that day was to refuse such a store (`StoreHashLabelError`); the developer reversed it in the +ITL-938 design session: v0.3 does not refuse a v0.2 store. It ignores rows hashed under +`arrow_v0.1`, computes every input again under `arrow_v0.2`, and leaves the old rows in place, +unchanged and distinguishable, for a later migration (ITL-913, ITL-914). + +Key lookups cannot match a row of another label. Every read of a whole table drops the rows +of another label through one function, `hashing/stored_hash_labels.keep_rows_hashed_under` +(without it: 4 rows for 2 inputs): `FunctionJobNode._fetch_joined_records` (phase 1, +`get_all_records`, `as_table`, `iter_data`, `as_source`, loaded jobs) on +`__pipeline_base_entry_id`; `ResultCache.get_all_records` on `__input_data_hash`; +`OperatorJobNode._read_stored_records` and `CachedSource._read_cached_records` on the starfix +format at the start of the `_record_hash` record id. Each reader logs one warning with the +total number of rows it left out (`report_skipped_rows`; developer: total counts are enough, +eventually old entries should be detected and a migration suggested). +`stored_hash_labels(database, record_path)` counts a table's rows by label. + +Side-effect invocation logs are the one exception (developer, same day): they are read by key +only, so deliveries logged under `arrow_v0.1` would not be found and every side effect would +be delivered again. A side-effect node whose log holds such rows raises +`SideEffectLogHashLabelError` before delivering anything. The developer wants a separate, +later discussion of what side-effect logs hold. + +Known limits, follow-ups: +- The warning is logged once per reader object (node, result cache, cached source) and + table, not once per process: stored paths are relative to the store, so a process-wide + record of reported tables would silence a second store with the same layout. +- A global hash-version identifier stored with every row is being designed (not decided); it + would also give operator-node and cached-source rows a label. +- Operator-node and cached-source record hashes carry no label, only the starfix format at + the start of the hex digest. That separates `arrow_v0.1` (format 1) from `arrow_v0.2` + (format 2) exactly, but a future label that keeps the starfix format could not be told apart + there. Storing the label with those rows (a schema change) would close that gap. +- The node content hash does not separate the generations: a source with an explicit + `source_id` gives the same node content hash under both labels. +- A `DeltaTableDatabase` object keeps the table version it read first, so a job object that + read a table before a run does not see the rows the run added until the store is reopened + (not specific to hash labels; seen when re-reading a node of the same job after an + incremental run). +- Tables placed under a content hash (`table_scope="content_hash"`, `CachedSource` without an + explicit `source_id`) move to new paths when the content hash moves, so their v0.2 rows are + in another table. + +--- + +## Hash-site audit — wrong hashes fixed before v0.3 (ITL-942) + +### WH1 — Hashes that collided or depended on data, process or spelling + +**Status:** resolved (ITL-942) + +The ITL-938 hash-site audit (Appendix A.9 of its design) found values that were wrong: two +different things shared a hash, or one thing hashed differently depending on its data, the +process or its spelling. Fixed, one commit each, every moved golden value listed: + +- **H** Batch/GroupBy source-id fold concatenated members with no separator + (`["ab","c"]` == `["a","bc"]`, `None` == `""`). **Fix:** SHA-256 of the canonical JSON + member list (`arrow_utils.fold_system_tag_values`); record-id fold encodes `None` as `null`. +- **I** Arrow hashing rebuilt nested types from the first non-null value, dropping nested + nullability for columns with data only. **Fix:** `SemanticHashingVisitor.hashed_type` + derives the hashed type from the declared type alone. +- **L1** Set items sorted by `str()`, so `{1, "1"}` hashed per process. **Fix:** sorted by + canonical JSON. (L2, function defaults entering as `repr()`, moved to ITL-953.) +- **Dict keys** went through `str()`. **Fix:** non-`str` keys, and any `"__type__"` key, give + the pair-list form `{"__type__":"dict","items":[...]}`; type handlers return `Encoded` for + tagged objects, so data cannot forge a tagged value. +- **F** Column selection operators hashed the container type and order of `columns`. + **Fix:** identity is the sorted list of unique names. +- **G** `_tag_record_id` was `fixed_size_binary[16]` from a source and `large_binary` after a + `Tag`. **Fix:** `large_binary` everywhere. +- **N** Default function version `v0.0` (`@function_pod`) versus `v1.0` (`from_fn`, + `side_effect_function_pod`). **Fix:** `v0.0` everywhere. +- **E** (reviewed, kept by decision): `MergeJoin(m1, m1)` differs from `MergeJoin(m1, m2)`; + inputs with one source identity count as one. Pinned by tests. +- **R** (subclass hashing as its registered base, with fields dropped) is fixed by ITL-881. + +- **T** a namedtuple's fields were sorted by name, so `Point(x=1, y=2)` and a namedtuple + declared `(y, x)` holding `y=2, x=1` shared a hash. **Fix:** fields in declared order. diff --git a/docs/concepts/arrays.md b/docs/concepts/arrays.md index 987d03582..fceaf7f53 100644 --- a/docs/concepts/arrays.md +++ b/docs/concepts/arrays.md @@ -28,11 +28,11 @@ for tag, data in scale.pod(source).iter_data(): !!! warning "v0.3 changed the array format" Orcapod v0.2 stored arrays as `.npy` files under the extension name `numpy.ndarray` and hashed the file. v0.3 replaces both, as described below, and does not migrate older - stores. Under v0.3, every node that handles arrays computes again into new tables, and its - v0.2 results are no longer returned. They stay on disk, readable with v0.2. Nodes - downstream of an array column get new pipeline and operator tables too; a function pod - among them that takes and returns no array reuses its stored results. Start v0.3 - pipelines with a fresh store, or keep using v0.2 for an existing one. A migration is + stores. Every node that handles arrays, and every node downstream of an array column, gets + new tables in v0.3. Because v0.3 also changed every hash of Arrow data, v0.3 reuses no + result of a store written by v0.2: it ignores the v0.2 rows and computes every input + again (see [A store written by an older version](function-pods.md#a-store-written-by-an-older-version)). + The v0.2 tables and rows stay on disk unchanged, readable with v0.2. A migration is planned after v0.3 (ITL-889). ## What counts as the same array diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 4a6a5aa77..a14e5582e 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -307,6 +307,31 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro This is the contract for now. An opt-in mode that tolerates store failures may be added later. +### A store written by an older version + +Stored rows are keyed by hashes of their Arrow data, and each hash carries the label of the +scheme that produced it: `arrow_v0.1` up to Orcapod v0.2, `arrow_v0.2` since v0.3 (see +[Arrow Table Hash](../reference/hashing.md#9-arrow-table-hash-arrow_v02)). Rows keyed under +another label can never match current inputs, so Orcapod **ignores** them: it does not refuse +the store, and it does not reuse those rows. + +- The first run of v0.3 on a store written by v0.2 computes every input again and stores the + new rows in the same tables, next to the old ones. +- Reading a node (`as_table()`, `iter_data()`, `get_all_records()`) returns the rows of the + current label only: nothing before that first run, one row per input after it. +- The old rows are never changed or deleted. They stay readable with the version that wrote + them and are kept for a later migration. To see which labels a table holds, use + `stored_hash_labels` (see + [Stores written under another label](../reference/hashing.md#stores-written-under-another-label)). +- When a read leaves such rows out, Orcapod logs one warning with their number: + `Skipped 3 rows of stored table '...': written under another hash label than the current + one. The rows are left in place unchanged and can be migrated later.` +- Side-effect pods are the exception. A side effect is an external action, and deliveries + logged by v0.2 cannot be matched with current inputs, so running the node would deliver + everything a second time. Instead the node raises `SideEffectLogHashLabelError` (from + `orcapod.errors`, a `StoreError`) before it delivers anything. Use a fresh store for that + node, or keep using the version that wrote the log. + ## Multiple input streams If you pass multiple streams to a function pod, they are automatically joined (using diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 13ddcf73a..e304ad8b1 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -20,7 +20,7 @@ appended, because it is a variant of the same system tag mechanism.) | 4 | Default `source_id` | `StarfixArrowHasher.hash_table(table).to_hex(n)` | Truncated hex `str` | Unique per raw table content; used as source identifier when none is provided | | 5 | Per-row `record_id` (system tag value) | `uuid.uuid5(NAMESPACE, f"{source_id}::{provenance_token}")` | `bytes` (16, UUID v5) | Deterministic per `(source_id, row_identity)`; stable across re-runs of the same source | | 6 | Join system tag suffix | `stream.pipeline_hash().to_hex(n)` + `:{idx}` appended to column name | Column name suffix | Unique per `(input topology, canonical join position)`; encodes full join lineage in column name | -| 6b | Aggregating operator system tag fold | `uuid.uuid5(NAMESPACE, joined_member_hex)` for `record_id`; `combine_hashes(*member_values)` otherwise | `bytes` (16) / hex `str`, plus a column name suffix | Deterministic per ordered member set; keeps system tags scalar through a many-to-one reduction | +| 6b | Aggregating operator system tag fold | `uuid.uuid5(NAMESPACE, joined_member_hex)` for `record_id`; SHA-256 of the canonical JSON list of member values otherwise | `bytes` (16) / hex `str`, plus a column name suffix | Deterministic per ordered member set; keeps system tags scalar through a many-to-one reduction | | 7 | `compute_base_entry_id()` | `StarfixArrowHasher.hash_table(system_tags + INPUT_DATA_HASH_COL)` | `bytes` (`b"method:digest"`) | Unique per `(node, tag lineage, input_data content)` across all recomputation attempts | | 8 | `compute_pipeline_entry_id()` | `StarfixArrowHasher.hash_table(system_tags + INPUT_DATA_HASH_COL + recomputation_index)` | `bytes` (`b"method:digest"`) | Unique per `(node, tag lineage, input_data content, recomputation attempt)` | | 9 | Side-effect `record_id` | `StarfixArrowHasher.hash_table(system_tags + INPUT_DATA_HASH_COL + recomputation_index=0)` | `bytes` | Unique per `(tag lineage, input_data content)`; pod-version scoped via table path (`uri` + `pipeline_hash()`) | @@ -132,24 +132,24 @@ each suffix is the stream's `pipeline_hash()` (sites 1b and 6). | Site | Row | Value (hex) | |------|-----|-------------| -| 7 | `alice` `base_entry_id` | `6172726f775f76302e313a00000113f967f1187ac2982e65b6afbf324d264d50a6efe3357a35f09b8a2b32454d61` | -| 7 | `bob` `base_entry_id` | `6172726f775f76302e313a000001b6162a6faa76b5e065e89849acc09f41edfb5bb1a7c7ea626fd2920de6046586` | -| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 7 | `alice` `base_entry_id` | `6172726f775f76302e323a000002d147e6dfdc31690391034ed592bb75d4c4407034504165d168717c0779b64d3f` | +| 7 | `bob` `base_entry_id` | `6172726f775f76302e323a000002250edea78c481fa87d262c5938c9288a5538a17a10636b54a6ea83a18c77eb82` | +| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | -> The hex prefix `6172726f775f76302e31` decodes to the ASCII string `arrow_v0.1` — the +> The hex prefix `6172726f775f76302e32` decodes to the ASCII string `arrow_v0.2` — the > `method` portion of the `b"{method}:{digest}"` format described in §3. #### §4 — Side-Effect Record ID & Invocation Hash | Site | Row | Value | |------|-----|-------| -| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | > Both rows share the same `semantic_v0.1:1f71386f...` component — this is the > `SideEffectPodStream.pipeline_hash()`, which depends only on the pod and its upstream -> topology, not on row content. The `arrow_v0.1:...` component is the per-row `record_id` +> topology, not on row content. The `arrow_v0.2:...` component is the per-row `record_id` > from site 9, which differs per row. #### §5 — Data Function URI Hash @@ -187,6 +187,29 @@ SHA-256 digest. 2. Recursively expand the structure: `ContentIdentifiable` objects are replaced by their hash via a **resolver callback** (see below); containers (list, dict, tuple, set) are serialised to nested JSON. 3. JSON-serialise the expanded structure and compute `hashlib.sha256(json_bytes).digest()` → `ContentHash`. +How containers expand (canonical JSON: sorted object keys, no whitespace, UTF-8): + +| Container | Expanded form | What counts | +|---|---|---| +| `list` | JSON array | order | +| `tuple` | `{"__type__": "tuple", "items": [...]}` | order | +| `set`, `frozenset` | `{"__type__": "set", "items": [...]}`, items sorted by their canonical JSON text | membership only; never iteration order or `PYTHONHASHSEED` (ITL-942: before v0.3, items were sorted by `str()`, so `{1, "1"}` hashed differently per process) | +| `dict` with only `str` keys, none of them `"__type__"` | JSON object | keys and values, not insertion order | +| any other `dict` | `{"__type__": "dict", "items": [[key, value], ...]}`, pairs sorted by canonical JSON | key types: `{1: "x"}` and `{"1": "x"}` differ (ITL-942: before v0.3, keys went through `str()` and could collapse) | +| namedtuple | `{"__type__": "namedtuple", "name": ..., "fields": [[name, value], ...]}` | class name, field names and values, in declared order (ITL-942: before v0.3 fields were sorted by name, so `Point(x=1, y=2)` and a `Point` declared `(y, x)` holding `y=2, x=1` shared a hash) | +| a type handler's `Encoded(tag, fields)` | `{"__type__": tag, **fields}` | the handler's tag and fields | + +**Only the hasher writes `"__type__"`.** A JSON object with a `"__type__"` key comes either +from the hasher's own encodings (tags `tuple`, `set`, `dict`, `namedtuple`, `hash`) or from a +type handler that returned an `Encoded` value (`generic_alias`, `union`). A mapping that has a +`"__type__"` key is hashed in the pair-list form, so a user value such as +`{"__type__": "tuple", "items": [1, 2]}` never shares the hash of `(1, 2)` (ITL-942). For +authors of a `PythonTypeHandlerProtocol`: return +`orcapod.hashing.semantic_hashing.Encoded(tag, fields)` to produce a tagged object; a plain +dict with a `"__type__"` key is hashed as an ordinary mapping. `Encoded` rejects the hasher's +own tags and a `"__type__"` field, and is trusted only as a handler's return value: an +`Encoded` inside a user value has no handler and is refused by the strict hasher. + **Resolver pattern:** The resolver callback is threaded through the entire recursive hash computation to ensure one consistent context per call: - **`content_resolver`** (used by `content_hash()`): routes every `ContentIdentifiable` leaf through `leaf.content_hash(hasher)`. @@ -213,6 +236,15 @@ The table below shows what each class returns from `identity_structure()`. | `DerivedSource` | Origin node's `content_hash` | Inherits its generating node's identity | | Operators (unary) | `(operator_class_name, upstream_stream)` | Stream reference resolved via `content_resolver` | | Operators (binary/N-ary) | `(operator_class_name, argument_symmetry(streams))` | `frozenset` for commutative (Join, MergeJoin); `tuple` for ordered (SemiJoin) | +| `SelectTagColumns`, `SelectDataColumns`, `DropTagColumns`, `DropDataColumns` | `(operator_class_name, sorted unique column names as a list, strict)` | The operators act on a set of names: a list, a tuple, a set, a single string, a repeated or a reordered name list give one hash (ITL-942) | + +**Inputs of a commutative operator are a set (kept by decision, ITL-942).** A pipeline hash +covers the structure of the pipeline, not only its schemas. `MergeJoin()(m1, m1)`, a stream +joined with itself, is a different pipeline from `MergeJoin()(m1, m2)` and has a different +pipeline hash even when both inputs share one schema. Inputs are an unordered set compared by +identity, so input order never matters, and two inputs with the same source identity (same +class, schemas and `source_id`) count as one input: `MergeJoin()(m1, m1b)` hashes like +`MergeJoin()(m1, m1)` when `m1b` has the `source_id` of `m1`. **Known exclusions from `identity_structure()`:** @@ -334,7 +366,7 @@ topology suffix, encoding the full join lineage into each column name. | Field | Value | |---|---| | **Inputs** | Full raw Arrow table (all columns, before any system tag injection) | -| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — versioned SHA-256 via the `starfix` Rust crate | +| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — starfix hash format 2: the 3-byte format number `00 00 02` followed by a SHA-256 digest (`starfix` 0.5, byte-identical to the Rust crate) | | **Output format** | Truncated hex `str` (length = `path_n_char`) | | **Uniqueness guarantee** | Unique per raw table content; changes if any cell value changes | | **Known exclusions** | Used **only** as a fallback when no explicit `source_id` is provided. A user-supplied `source_id` bypasses this computation entirely. | @@ -347,7 +379,7 @@ topology suffix, encoding the full join lineage into each column name. |---|---| | **Inputs** | `source_id` string + provenance token: `"{col}={value}"` when `record_id_column` is specified, otherwise `"row_{index}"` | | **Algorithm** | `uuid.uuid5(_SOURCE_RECORD_ID_NAMESPACE, f"{source_id}::{provenance_token}")` where `_SOURCE_RECORD_ID_NAMESPACE = uuid.uuid5(NAMESPACE_URL, "orcapod::record_id")` is a fixed constant | -| **Output format** | `bytes` (16 bytes, UUID v5 bit pattern), stored in a `pa.binary(16)` Arrow column | +| **Output format** | `bytes` (16 bytes, UUID v5 bit pattern), stored in a `large_binary` Arrow column; the column keeps that type through tags, operators and stored tables (ITL-942: before v0.3 a source gave it `fixed_size_binary[16]`, so the operator cache key of one row differed between a direct run and the async orchestrator) | | **Uniqueness guarantee** | Deterministic per `(source_id, row_identity)`; stable across identical re-runs of the same source | | **Known exclusions** | When `record_id_column` is not specified, the row index is used as the provenance token; `record_id` changes if rows are **reordered** within the source table | @@ -370,7 +402,7 @@ topology suffix, encoding the full join lineage into each column name. | Field | Value | |---|---| | **Inputs** | The ordered list of a group's member values for one system tag column, plus that column's name (which selects the fold) | -| **Algorithm** | `arrow_utils.fold_system_tag_values()`. For a `record_id` column: `uuid.uuid5(_AGGREGATED_RECORD_ID_NAMESPACE, BLOCK_SEPARATOR.join(v.hex() for v in values))`, where `_AGGREGATED_RECORD_ID_NAMESPACE = uuid.uuid5(NAMESPACE_URL, "https://orcapod.org/namespaces/aggregated-record-id")` is a fixed constant. For every other system tag column: `combine_hashes(*[str(v) for v in values], order=False)` — SHA-256 over the members concatenated in order. `None` members contribute the empty string. The folded column name then gains `{BLOCK_SEPARATOR}{stream.pipeline_hash().to_hex(system_tag_n_char)}` via `arrow_utils.append_to_system_tags()`. | +| **Algorithm** | `arrow_utils.fold_system_tag_values()`. For a `record_id` column: `uuid.uuid5(_AGGREGATED_RECORD_ID_NAMESPACE, BLOCK_SEPARATOR.join(v.hex() for v in values))`, where `_AGGREGATED_RECORD_ID_NAMESPACE = uuid.uuid5(NAMESPACE_URL, "https://orcapod.org/namespaces/aggregated-record-id")` is a fixed constant. A `None` member contributes `null`. For every other system tag column: SHA-256 of the canonical JSON array of the members in order (`json.dumps([None if v is None else str(v) for v in values], separators=(",", ":"), ensure_ascii=False)`, UTF-8), as 64 hex characters. The array marks where each member ends, so `["ab", "c"]` and `["a", "bc"]` fold differently, and a `None` member (`null`) differs from `""` (ITL-942; before v0.3 the members were concatenated with no separator). The folded column name then gains `{BLOCK_SEPARATOR}{stream.pipeline_hash().to_hex(system_tag_n_char)}` via `arrow_utils.append_to_system_tags()`. | | **Output format** | `bytes` (16, UUID v5 bit pattern) for a `record_id` column; 64-character hex `str` otherwise — in both cases **scalar**, matching the input column's type | | **Uniqueness guarantee** | Deterministic per ordered member set, and stable across processes (both primitives are SHA-based). `Batch` and `GroupBy` both order their members deterministically before folding, so an unchanged member set always folds to the same digest | | **Known exclusions** | Member order is significant; a permuted member set folds differently. This is why `GroupBy` sorts members by their non-key tag values with `record_id` as a tiebreaker rather than relying on upstream emission order | @@ -487,7 +519,10 @@ uri = (canonical_function_name, output_schema_hash, major_version, data_function ``` The `output_schema_hash` component (site 11) is the only hash in the URI; the other -components are plain strings. +components are plain strings. `major_version` comes from the `version` argument, which +defaults to `"v0.0"` (major `v0`) for every entry point: `@function_pod`, +`FunctionPod.from_fn`, `side_effect_function_pod` and `PythonDataFunction` (ITL-942: before +v0.3, `from_fn` and `side_effect_function_pod` defaulted to `"v1.0"`). --- @@ -618,3 +653,119 @@ print(hasher.hash_object(np.arange(6, dtype=np.float32).reshape(2, 3)).to_string print(hasher.hash_object(np.array(2.5)).to_string()) # ndarray_v1:47f182d862f5dd597d88836deddaed609ed94490827302e75e8364e877633cae ``` + +## 9. Arrow Table Hash (`arrow_v0.2`) + +Every Arrow-derived hash in the sites above (row data hashes, `record_id`s built from a +default `source_id`, base and pipeline entry ids, operator and cached-source record hashes, +`pa.Table` and pandas values) comes from `StarfixArrowHasher.hash_table`, which returns +`ContentHash(method="arrow_v0.2", digest=D)`. `D` is 35 bytes: the starfix hash format +number `00 00 02` followed by a SHA-256 digest, computed by `starfix` 0.5 (hash format 2) and +byte-identical to the Rust `starfix` crate. The byte layout is specified in starfix's +`docs/byte-layout-spec.md`. + +**Label rule.** The label `arrow_vX.Y` moves whenever the hash of any table can change, +whatever the cause. Its number is not kept equal to the starfix format number; the data +context's changelog records which starfix format each label uses. + +| Label | starfix | Format prefix | Orcapod | +|---|---|---|---| +| `arrow_v0.1` | 0.4 | `00 00 01` | up to v0.2 | +| `arrow_v0.2` | 0.5 | `00 00 02` | v0.3 | + +**What is part of the hash.** Column names, Arrow types (with large and non-large layouts of +strings, binaries and lists hashing alike), nullability at every depth (top-level columns, +struct fields and list items, whether or not the column holds data), every value, row order, list element +order, list lengths at every depth, null structs (distinct from structs whose fields are all +null) and the row count. Column order and struct field order are not part of the hash. How a +table is split into chunks or slices is not part of it either. A `fixed_size_list[n]` +column hashes differently from a `large_list` column with the same values. Extension +columns are hashed as their storage type plus their extension name and metadata, including +pyarrow's canonical extension types (`arrow.uuid`, `arrow.fixed_shape_tensor`); only +`ARROW:extension:*` metadata is hashed, all other metadata is ignored. List item names +(`item`, `element`) are not part of the hash. + +**The hashed type is the declared type.** Before starfix sees a table, Orcapod replaces the +values of logical types that have a value hasher (`Path`, `File`, `Directory`, `UUID`, ...) +by their content-hash tokens, typed `large_binary`. The type each column is hashed under is +derived from its declared type alone (`SemanticHashingVisitor.hashed_type`): every struct and +list field keeps its name, nullability and metadata. A column therefore hashes under one type +whether it has data, only nulls or no rows (ITL-942: before v0.3 the type was rebuilt from the +first non-null value, which dropped nested nullability for a column with data). + +**What cannot be hashed.** starfix raises `ValueError`, naming the column path and type, for: +data of `null`, `map`, interval, `string_view` and `binary_view` columns (whatever the row +count; their schemas hash); union and run-end-encoded types; two schema nodes sharing a path +(duplicate column names, or a name containing `/` that aliases a struct child, such as a +column `"a/b"` next to a struct column `a` with a child `b`); and a non-nullable field holding +a null. A source whose table has such a column fails when it is constructed. + +#### Stores written under another label + +A change of the Arrow hash label changes the stored keys but not the tables they are stored +in, so one table can hold rows of several labels. Orcapod writes and uses rows of its +current label only. Key lookups cannot match a row of another label, and every read of a +whole table (a node's `as_table()`, `iter_data()`, `get_all_records()`, `as_source()`, a +cached function's `get_all_cached_outputs()`, a stored operator node, a `CachedSource`) leaves +the rows of another label out. Those rows are never changed or deleted: a store written by +Orcapod v0.2 keeps its `arrow_v0.1` rows, v0.3 computes every input again and adds +`arrow_v0.2` rows next to them, and a later migration can convert the old ones. + +How a stored row names its label: + +| Table | Key | Stored form | +|---|---|---| +| Function-node pipeline table | `__pipeline_base_entry_id` | `b"