diff --git a/CHANGELOG.md b/CHANGELOG.md index dfdc26d3b..308894c4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,11 @@ #### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883) **The array format and every array hash changed in v0.3.** A store written by v0.2 that holds -`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays -computes again into new tables, and its v0.2 results are no longer returned. Nothing written by -v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for -v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889). +`numpy.ndarray` columns is **not migrated**. Every node that handles arrays gets new tables, +so its v0.2 results are no longer found; and since every Arrow-derived hash changed too +(ITL-910, below), no v0.2 result is reused at all: v0.3 ignores the rows a v0.2 store holds and +computes every input again. Nothing written by v0.2 is changed or deleted, and those tables stay +readable with v0.2. A migration is planned after v0.3 (ITL-889). - **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata `{"version":1}`) over `struct, @@ -29,8 +30,10 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0. change too, and so do the table paths of those nodes. Pipeline hashes are chained, so the pipeline and operator tables of every node downstream of an array column move as well, even when it has no array column itself. A pod whose inputs and outputs hold no array - keeps its function result table, so it reuses its v0.2 results without computing again. - Nodes with no array column upstream do not move. + keeps its function result table, and nodes with no array column upstream keep their tables; + but an array-free pod does **not** reuse its v0.2 results: every Arrow-derived hash changed + as well (ITL-910, below), so the v0.2 rows in such tables are ignored, the inputs are computed + again, and the new rows are stored next to the old ones. - **Refused at write time** with `orcapod.errors.UnsupportedValueError` (a `ValueError`), naming the column and dtype: masked arrays (the mask used to be dropped silently), object dtypes, NumPy 2 `StringDType` (used to fail with a misleading pickle error), `longdouble` and @@ -47,6 +50,80 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0. the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its subclasses, whatever the registration order. +#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910) + +**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1` +to `arrow_v0.2`.** starfix 0.4 (hash format 1) lost information for some inputs, so different +data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and +`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean +columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash +format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept. + +- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default + `source_id`s and the `record_id`s and source content hashes built from them, function + input-data hashes, pipeline base and entry ids, operator and cached-source record hashes, + invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The + hash label in front of each (`arrow_v0.2:000002...`) shows the new format. Pipeline hashes, + schema hashes, system-tag column names and table paths under the default + `table_scope="pipeline_hash"` do not move. Paths that embed a content hash + (`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or + `Batch` orders members by `record_id`, the order of members can change too. +- **Results stored by 0.2.0 are ignored and computed again; they are not reused, changed or + deleted.** Rows a v0.2 store holds are keyed under `arrow_v0.1` and can never match v0.3 + inputs. v0.3 does not refuse such a store. A value-only change does not move a table, so a + v0.3 job writes its `arrow_v0.2` rows into the same tables, next to the `arrow_v0.1` rows, + and uses rows of its own label only: + - The first v0.3 run on a v0.2 store **computes every input again**. Nothing stored by v0.2 + counts as a cached result, also for functions and data that did not change. + - Reading a node (`as_table()`, `iter_data()`, `get_all_records()`, `as_source()`, also on a + job loaded from a file, including a file saved by v0.2), `get_all_cached_outputs()` of a + cached function, a stored (`LOG` / `REPLAY`) operator node and a `CachedSource` return + rows hashed under `arrow_v0.2` only. Before the first v0.3 run, such a read on a v0.2 + store returns no rows; after it, one row per input, never the v0.2 rows as well. + - The v0.2 rows stay where they are, byte for byte, and stay readable with v0.2. Each + stored key names its label (`arrow_v0.1:000001...` or `arrow_v0.2:000002...`), and + `orcapod.hashing.stored_hash_labels.stored_hash_labels(store, record_path)` counts the + rows of a table by label, for example `{"arrow_v0.1": 3, "arrow_v0.2": 3}`. + - When a read leaves out rows of another label, Orcapod logs one warning per table and + reader with the total, on the logger `orcapod.hashing.stored_hash_labels`: + `Skipped 3 rows of stored table 'my_job/.../pdb_v1': written under another hash label + than the current one. The rows are left in place unchanged and can be migrated later.` + - **Side-effect nodes are the exception: they fail instead of delivering again.** A + side-effect pod finds a delivery it already made by an Arrow-derived record id, so + deliveries logged by v0.2 cannot be found, and running the node would deliver every side + effect a second time. A side effect is an external action, not a value that can be + computed again, so a side-effect node whose invocation log holds deliveries logged under + another label raises `orcapod.errors.SideEffectLogHashLabelError` (a `StoreError`, so it + fails the run whatever the `error_policy`) before it delivers or writes anything. The + error names the log, the label found and the current one. Function, operator and source + nodes of the same job are not refused. Use a fresh store for the side-effect node, or + keep using v0.2 for it, until the log can be migrated. + + **Migration:** none is needed to run the function, operator and source nodes of a v0.2 + store under v0.3, at the price of computing everything again and of tables that hold both + generations; side-effect nodes need a fresh store. A tool that converts + `arrow_v0.1` rows into `arrow_v0.2` rows is planned (ITL-913, ITL-914); until then the old + rows are kept for it. Operator-node and cached-source tables store their record hashes + without a label; there the rows of v0.2 are told apart by the starfix hash format at the + start of each record hash (`000001` for v0.2, `000002` for v0.3). That is exact for these + two versions, but a later Arrow hash label that keeps starfix hash format 2 could not be + told apart in those two kinds of table. A table that `table_scope="content_hash"` or a + `CachedSource` places under a content hash moves to a new path instead, so its v0.2 rows + are in another table. +- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator + carrying one works, and its type is kept. A `fixed_size_list[n]` hashes differently from a + `large_list` with the same values. pyarrow's canonical extension types (`arrow.uuid`, + `arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid` + column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and + `StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846). +- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed + silently: a non-nullable field holding a null; a zero-row table with a `map` column; two + schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a + column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`, + interval, `string_view` and `binary_view` columns, union and run-end-encoded types are + still refused, now with `ValueError` instead of `NotImplementedError`. +- Column order and struct field order are still not part of an Arrow hash. + #### A changed column type is refused when appending to a Delta table (ITL-884) Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows' @@ -80,6 +157,13 @@ stay queued for the next flush. A flush that fails only for other reasons still the backend's exception as its cause. (ITL-884 #290) - **`orcapod.errors.StoreError`** — the base of `StoreReadError` and `StoreWriteError`; `except StoreError` handles both. (ITL-884 #290) +- **`orcapod.errors.SideEffectLogHashLabelError`** — a `StoreError` raised by a side-effect + node whose invocation log holds deliveries logged under another Arrow hash label (a log + written by v0.2), before anything is delivered. It has `record_path`, `found_label`, + `current_label` and `node_label`. (ITL-910) +- **`orcapod.hashing.stored_hash_labels.stored_hash_labels(database, record_path)`** — counts + the rows of a stored table by the Arrow hash label they were hashed under, to tell rows + written by v0.2 (`arrow_v0.1`) from rows written by v0.3 (`arrow_v0.2`). (ITL-910) ### Fixed diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 695708a20..d84a41c79 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2442,3 +2442,70 @@ results. Known compromise: the struct's children are nullable at the Arrow level, because Polars drops nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null instead. Tighten under a new storage version once nested nullability survives Polars. + +--- + +## `src/orcapod/hashing/arrow_hashers.py` — Arrow hash format (starfix) + +### HS2 — starfix hash format 1 collided on nested lists, null structs and Booleans +**Status:** resolved +**Severity:** high +**Issue:** ITL-910 (fixes the starfix side of ITL-848; ITL-927) + +`StarfixArrowHasher` used starfix 0.4 (hash format 1, label `arrow_v0.1`), which lost +information for some inputs, so different data reached the same row hash: nested-list lengths +(`[[[1]], [[2], [3]]]` and `[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are +all null, and non-nullable Boolean columns (`[True]` and `[True, False, False]`); a Boolean +column's hash also depended on chunking. starfix 0.4 also could not hash `fixed_size_list` +data, so a source with such a column failed at construction (ITL-927). + +**Fix:** starfix 0.5 (hash format 2, prefix `00 00 02`) under the new label `arrow_v0.2` +(developer's label rule: the label moves whenever any table hash can change). Every +Arrow-derived hash moves; the golden files were regenerated once, explicitly. Order options +stay off. pyarrow's canonical extension types and `hash_schema` on extension columns hash with +their extension identity instead of as their storage type. + +Stored rows: under the default `table_scope`, table paths do not move (developer, November +11, 2026: only a change of a table's schema goes into its path; a value-only change stays in +the same table), so a v0.3 node writes into the same tables as a v0.2 store. The first decision +that day was to refuse such a store (`StoreHashLabelError`); the developer reversed it in the +ITL-938 design session: v0.3 does not refuse a v0.2 store. It ignores rows hashed under +`arrow_v0.1`, computes every input again under `arrow_v0.2`, and leaves the old rows in place, +unchanged and distinguishable, for a later migration (ITL-913, ITL-914). + +Key lookups cannot match a row of another label. Every read of a whole table drops the rows +of another label through one function, `hashing/stored_hash_labels.keep_rows_hashed_under` +(without it: 4 rows for 2 inputs): `FunctionJobNode._fetch_joined_records` (phase 1, +`get_all_records`, `as_table`, `iter_data`, `as_source`, loaded jobs) on +`__pipeline_base_entry_id`; `ResultCache.get_all_records` on `__input_data_hash`; +`OperatorJobNode._read_stored_records` and `CachedSource._read_cached_records` on the starfix +format at the start of the `_record_hash` record id. Each reader logs one warning with the +total number of rows it left out (`report_skipped_rows`; developer: total counts are enough, +eventually old entries should be detected and a migration suggested). +`stored_hash_labels(database, record_path)` counts a table's rows by label. + +Side-effect invocation logs are the one exception (developer, same day): they are read by key +only, so deliveries logged under `arrow_v0.1` would not be found and every side effect would +be delivered again. A side-effect node whose log holds such rows raises +`SideEffectLogHashLabelError` before delivering anything. The developer wants a separate, +later discussion of what side-effect logs hold. + +Known limits, follow-ups: +- The warning is logged once per reader object (node, result cache, cached source) and + table, not once per process: stored paths are relative to the store, so a process-wide + record of reported tables would silence a second store with the same layout. +- A global hash-version identifier stored with every row is being designed (not decided); it + would also give operator-node and cached-source rows a label. +- Operator-node and cached-source record hashes carry no label, only the starfix format at + the start of the hex digest. That separates `arrow_v0.1` (format 1) from `arrow_v0.2` + (format 2) exactly, but a future label that keeps the starfix format could not be told apart + there. Storing the label with those rows (a schema change) would close that gap. +- The node content hash does not separate the generations: a source with an explicit + `source_id` gives the same node content hash under both labels. +- A `DeltaTableDatabase` object keeps the table version it read first, so a job object that + read a table before a run does not see the rows the run added until the store is reopened + (not specific to hash labels; seen when re-reading a node of the same job after an + incremental run). +- Tables placed under a content hash (`table_scope="content_hash"`, `CachedSource` without an + explicit `source_id`) move to new paths when the content hash moves, so their v0.2 rows are + in another table. diff --git a/docs/concepts/arrays.md b/docs/concepts/arrays.md index 987d03582..fceaf7f53 100644 --- a/docs/concepts/arrays.md +++ b/docs/concepts/arrays.md @@ -28,11 +28,11 @@ for tag, data in scale.pod(source).iter_data(): !!! warning "v0.3 changed the array format" Orcapod v0.2 stored arrays as `.npy` files under the extension name `numpy.ndarray` and hashed the file. v0.3 replaces both, as described below, and does not migrate older - stores. Under v0.3, every node that handles arrays computes again into new tables, and its - v0.2 results are no longer returned. They stay on disk, readable with v0.2. Nodes - downstream of an array column get new pipeline and operator tables too; a function pod - among them that takes and returns no array reuses its stored results. Start v0.3 - pipelines with a fresh store, or keep using v0.2 for an existing one. A migration is + stores. Every node that handles arrays, and every node downstream of an array column, gets + new tables in v0.3. Because v0.3 also changed every hash of Arrow data, v0.3 reuses no + result of a store written by v0.2: it ignores the v0.2 rows and computes every input + again (see [A store written by an older version](function-pods.md#a-store-written-by-an-older-version)). + The v0.2 tables and rows stay on disk unchanged, readable with v0.2. A migration is planned after v0.3 (ITL-889). ## What counts as the same array diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 4a6a5aa77..a14e5582e 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -307,6 +307,31 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro This is the contract for now. An opt-in mode that tolerates store failures may be added later. +### A store written by an older version + +Stored rows are keyed by hashes of their Arrow data, and each hash carries the label of the +scheme that produced it: `arrow_v0.1` up to Orcapod v0.2, `arrow_v0.2` since v0.3 (see +[Arrow Table Hash](../reference/hashing.md#9-arrow-table-hash-arrow_v02)). Rows keyed under +another label can never match current inputs, so Orcapod **ignores** them: it does not refuse +the store, and it does not reuse those rows. + +- The first run of v0.3 on a store written by v0.2 computes every input again and stores the + new rows in the same tables, next to the old ones. +- Reading a node (`as_table()`, `iter_data()`, `get_all_records()`) returns the rows of the + current label only: nothing before that first run, one row per input after it. +- The old rows are never changed or deleted. They stay readable with the version that wrote + them and are kept for a later migration. To see which labels a table holds, use + `stored_hash_labels` (see + [Stores written under another label](../reference/hashing.md#stores-written-under-another-label)). +- When a read leaves such rows out, Orcapod logs one warning with their number: + `Skipped 3 rows of stored table '...': written under another hash label than the current + one. The rows are left in place unchanged and can be migrated later.` +- Side-effect pods are the exception. A side effect is an external action, and deliveries + logged by v0.2 cannot be matched with current inputs, so running the node would deliver + everything a second time. Instead the node raises `SideEffectLogHashLabelError` (from + `orcapod.errors`, a `StoreError`) before it delivers anything. Use a fresh store for that + node, or keep using the version that wrote the log. + ## Multiple input streams If you pass multiple streams to a function pod, they are automatically joined (using diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 13ddcf73a..42b4b175a 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -132,24 +132,24 @@ each suffix is the stream's `pipeline_hash()` (sites 1b and 6). | Site | Row | Value (hex) | |------|-----|-------------| -| 7 | `alice` `base_entry_id` | `6172726f775f76302e313a00000113f967f1187ac2982e65b6afbf324d264d50a6efe3357a35f09b8a2b32454d61` | -| 7 | `bob` `base_entry_id` | `6172726f775f76302e313a000001b6162a6faa76b5e065e89849acc09f41edfb5bb1a7c7ea626fd2920de6046586` | -| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 7 | `alice` `base_entry_id` | `6172726f775f76302e323a000002d147e6dfdc31690391034ed592bb75d4c4407034504165d168717c0779b64d3f` | +| 7 | `bob` `base_entry_id` | `6172726f775f76302e323a000002250edea78c481fa87d262c5938c9288a5538a17a10636b54a6ea83a18c77eb82` | +| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | -> The hex prefix `6172726f775f76302e31` decodes to the ASCII string `arrow_v0.1` — the +> The hex prefix `6172726f775f76302e32` decodes to the ASCII string `arrow_v0.2` — the > `method` portion of the `b"{method}:{digest}"` format described in §3. #### §4 — Side-Effect Record ID & Invocation Hash | Site | Row | Value | |------|-----|-------| -| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | > Both rows share the same `semantic_v0.1:1f71386f...` component — this is the > `SideEffectPodStream.pipeline_hash()`, which depends only on the pod and its upstream -> topology, not on row content. The `arrow_v0.1:...` component is the per-row `record_id` +> topology, not on row content. The `arrow_v0.2:...` component is the per-row `record_id` > from site 9, which differs per row. #### §5 — Data Function URI Hash @@ -334,7 +334,7 @@ topology suffix, encoding the full join lineage into each column name. | Field | Value | |---|---| | **Inputs** | Full raw Arrow table (all columns, before any system tag injection) | -| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — versioned SHA-256 via the `starfix` Rust crate | +| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — starfix hash format 2: the 3-byte format number `00 00 02` followed by a SHA-256 digest (`starfix` 0.5, byte-identical to the Rust crate) | | **Output format** | Truncated hex `str` (length = `path_n_char`) | | **Uniqueness guarantee** | Unique per raw table content; changes if any cell value changes | | **Known exclusions** | Used **only** as a fallback when no explicit `source_id` is provided. A user-supplied `source_id` bypasses this computation entirely. | @@ -618,3 +618,109 @@ print(hasher.hash_object(np.arange(6, dtype=np.float32).reshape(2, 3)).to_string print(hasher.hash_object(np.array(2.5)).to_string()) # ndarray_v1:47f182d862f5dd597d88836deddaed609ed94490827302e75e8364e877633cae ``` + +## 9. Arrow Table Hash (`arrow_v0.2`) + +Every Arrow-derived hash in the sites above (row data hashes, `record_id`s built from a +default `source_id`, base and pipeline entry ids, operator and cached-source record hashes, +`pa.Table` and pandas values) comes from `StarfixArrowHasher.hash_table`, which returns +`ContentHash(method="arrow_v0.2", digest=D)`. `D` is 35 bytes: the starfix hash format +number `00 00 02` followed by a SHA-256 digest, computed by `starfix` 0.5 (hash format 2) and +byte-identical to the Rust `starfix` crate. The byte layout is specified in starfix's +`docs/byte-layout-spec.md`. + +**Label rule.** The label `arrow_vX.Y` moves whenever the hash of any table can change, +whatever the cause. Its number is not kept equal to the starfix format number; the data +context's changelog records which starfix format each label uses. + +| Label | starfix | Format prefix | Orcapod | +|---|---|---|---| +| `arrow_v0.1` | 0.4 | `00 00 01` | up to v0.2 | +| `arrow_v0.2` | 0.5 | `00 00 02` | v0.3 | + +**What is part of the hash.** Column names, Arrow types (with large and non-large layouts of +strings, binaries and lists hashing alike), nullability, every value, row order, list element +order, list lengths at every depth, null structs (distinct from structs whose fields are all +null) and the row count. Column order and struct field order are not part of the hash. How a +table is split into chunks or slices is not part of it either. A `fixed_size_list[n]` +column hashes differently from a `large_list` column with the same values. Extension +columns are hashed as their storage type plus their extension name and metadata, including +pyarrow's canonical extension types (`arrow.uuid`, `arrow.fixed_shape_tensor`); only +`ARROW:extension:*` metadata is hashed, all other metadata is ignored. + +**What cannot be hashed.** starfix raises `ValueError`, naming the column path and type, for: +data of `null`, `map`, interval, `string_view` and `binary_view` columns (whatever the row +count; their schemas hash); union and run-end-encoded types; two schema nodes sharing a path +(duplicate column names, or a name containing `/` that aliases a struct child, such as a +column `"a/b"` next to a struct column `a` with a child `b`); and a non-nullable field holding +a null. A source whose table has such a column fails when it is constructed. + +#### Stores written under another label + +A change of the Arrow hash label changes the stored keys but not the tables they are stored +in, so one table can hold rows of several labels. Orcapod writes and uses rows of its +current label only. Key lookups cannot match a row of another label, and every read of a +whole table (a node's `as_table()`, `iter_data()`, `get_all_records()`, `as_source()`, a +cached function's `get_all_cached_outputs()`, a stored operator node, a `CachedSource`) leaves +the rows of another label out. Those rows are never changed or deleted: a store written by +Orcapod v0.2 keeps its `arrow_v0.1` rows, v0.3 computes every input again and adds +`arrow_v0.2` rows next to them, and a later migration can convert the old ones. + +How a stored row names its label: + +| Table | Key | Stored form | +|---|---|---| +| Function-node pipeline table | `__pipeline_base_entry_id` | `b"