diff --git a/.zed/rules b/.zed/rules index 0c226b8ac..b14340af1 100644 --- a/.zed/rules +++ b/.zed/rules @@ -372,6 +372,46 @@ hashes, forming a Merkle chain. Prefixes are computed from SystemConstant in system_constants.py. +Stored tables: data context and layout versions (developer, November 11, 2026, ITL-944) + +Every row Orcapod stores carries __data_context (constants.DATA_CONTEXT_COL, large_string): +the key of the data context that wrote it (std:v1:default since v0.3). The key names one +generation of hashing; rows of the current data context are the current entries. + + Function pipeline — pdb_v2 (PIPELINE_DB_SCHEMA_VERSION) + Function result — rdb_v2 (RESULT_DB_SCHEMA_VERSION) + Side-effect log — tdb_v2 (TRACKING_DB_SCHEMA_VERSION) + Operator cache — odb_v1 (OPERATOR_DB_SCHEMA_VERSION) + Cached source — sdb_v1 (SOURCE_DB_SCHEMA_VERSION) + +- The layout version is the last component of a table path. A change to the bookkeeping + columns of a table kind moves its layout version, so the table gets a new path. Never add, + rename or retype a stored column without it. +- Every new stored table kind writes __data_context (contexts/stored.with_data_context), and + every read or lookup keeps rows of the current data context only + (keep_rows_of_data_context, or a lookup constraint on the column). +- A table at an older layout is never read as current and never modified. What a node does + about such tables, and about rows of another data context, is decided in one place, + contexts/stored.account_for_stale_tables: count and report, or, for a node that delivers + side effects (a SideEffectPod node, or a function node whose pod takes the invocation + context, ctx_arg), raise SideEffectLogDataContextError before any delivery. A new kind of + node goes through that function; never decide it locally. +- A released data context manifest (contexts/data/.json) is frozen: its sha256 is + pinned in contexts/released_manifests.json. A change that moves an existing golden hash + value for a reason other than a logical type's own change needs a new data context version + with its own manifest file, never an edit of a released one. +- In memory the data context column is still _context_key (ColumnConfig context). +- A writer refuses an input of another data context than its own + (contexts/stored.require_same_data_context, InputValidationError naming both keys) before + anything is written; a whole-input condition is checked outside the row error policy. The + eventual behaviour is ITL-951. +- std:v0.1:default is not shipped; saved files that name it load through + contexts.resolve_saved_context. There is no SchemaVersionError, ignore_schema or + orcapod.migrations: do not reintroduce a per-layout migration path without a decision. +- Logical types and the value handlers that hash their Python values are registered in + contexts/logical_types.json, never in a data context manifest: a logical type is + versioned by its own identifier, and that file is not frozen. + ### System tag evolution rules 1. Name-preserving — single-stream ops. Column name/value pass through unchanged. diff --git a/CHANGELOG.md b/CHANGELOG.md index dfdc26d3b..559aac9d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,11 @@ #### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883) **The array format and every array hash changed in v0.3.** A store written by v0.2 that holds -`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays -computes again into new tables, and its v0.2 results are no longer returned. Nothing written by -v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for -v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889). +`numpy.ndarray` columns is **not migrated**. Every node that handles arrays gets new tables, +so its v0.2 results are no longer found; and since every Arrow-derived hash changed too +(ITL-910, below), no v0.2 result is reused at all: v0.3 ignores the rows a v0.2 store holds and +computes every input again. Nothing written by v0.2 is changed or deleted, and those tables stay +readable with v0.2. A migration is planned after v0.3 (ITL-889). - **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata `{"version":1}`) over `struct, @@ -29,8 +30,10 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0. change too, and so do the table paths of those nodes. Pipeline hashes are chained, so the pipeline and operator tables of every node downstream of an array column move as well, even when it has no array column itself. A pod whose inputs and outputs hold no array - keeps its function result table, so it reuses its v0.2 results without computing again. - Nodes with no array column upstream do not move. + keeps its function result table, and nodes with no array column upstream keep their tables; + but an array-free pod does **not** reuse its v0.2 results: every Arrow-derived hash changed + as well (ITL-910, below) and every table moved to a new layout (ITL-944, below), so the + v0.2 tables are not read and the inputs are computed again. - **Refused at write time** with `orcapod.errors.UnsupportedValueError` (a `ValueError`), naming the column and dtype: masked arrays (the mask used to be dropped silently), object dtypes, NumPy 2 `StringDType` (used to fail with a misleading pickle error), `longdouble` and @@ -47,6 +50,164 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0. the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its subclasses, whatever the registration order. +#### Data context `std:v1:default`: a `__data_context` column on every table, and every table at a new path (ITL-944) + +**v0.3 writes every stored table at a new path, and records on every row which generation of +hashing wrote it.** A store written by v0.2 is not migrated and not modified: v0.3 does not +read its tables as results, and computes every input again into its own tables. + +- **One data context key for the whole of Orcapod's hashing.** The default data context is + now `std:v1:default` (manifest `orcapod/contexts/data/v1.json`). The version increments + when, and only when, an existing hash value changes for a reason other than a logical + type's own change. Everything stored before v0.3 (rows that carry `std:v0.1:default`, and + tables without the new column) is the earlier generation. A manifest is frozen at the + release of the library version that contains it, and may be edited in place only while + that release is prepared: `orcapod/contexts/released_manifests.json` pins the sha256 of + each released one, and a test fails when a pinned file is edited. `v1.json` is pinned + when v0.3.0 is tagged. +- **Manifest contents.** Besides the wiring of the hashers, a manifest now has + `released_in` (the library version that froze it; `null` while open), `previous` (the data + context it follows), `definitions` (a name for each definition that lives in code), + `libraries` (the libraries the core hashes depend on, with versions), `golden` (a digest + of the golden hash files, set when the manifest is frozen) and `changes` (what differs + from `previous`). +- **Logical types moved out of the manifest.** The registered logical types and the value + handlers that hash their Python values are now in + `orcapod/contexts/logical_types.json`, which is not frozen: a logical type is versioned by + its own identifier, not by the data context. What the default context registers is + unchanged. `JSONDataContextRegistry` and `contexts.create_registry` take + `logical_types_file` to use another registration file; a manifest that lists its logical + types inline is still built as written. +- **New column `__data_context`** (`large_string`) on every row of the function pipeline + table, the function result table, the operator-cache table, the cached-source table and + the side-effect invocation log. It **replaces** `__input_data_context_key` in the pipeline + table and `_context_key` in the result table. In memory nothing is renamed: + `ColumnConfig(context=True)` still shows `_context_key`. +- **New column `__pf_var_function_version`** in the function result table: the full version + string the function was declared with (`"v3.7"`). Only the major part is in the table + path, and until now no stored column held the version. +- **New table layouts, so new paths.** The last component of every table path moved: + + | Table | v0.2 | v0.3 | + |---|---|---| + | Function pipeline table | `…/schema:/pdb_v1` | `…/schema:/pdb_v2` | + | Function result table | `…/_result//rdb_v1` | `…/_result//rdb_v2` | + | Side-effect invocation log | `…/schema:/tdb_v1` | `…/schema:/tdb_v2` | + | Operator-cache table | `/schema:` | `/schema:/odb_v1` | + | Cached-source table | `source/node:` | `source/node:/sdb_v1` | + + The layout component is always the last one: `CachedSource(..., cache_path=("my", "cache"))` + now stores at `("my", "cache", "sdb_v1")`, and `CachedSource.cache_path` returns that full + path. Pass the path without the layout component as `cache_path`. +- **Reads and lookups use rows of the current data context only.** A result lookup matches + on the input hash and the data context; every read of a whole table (a node's `as_table()`, + `iter_data()`, `get_all_records()`, `as_source()`, `get_all_cached_outputs()` of a cached + function, a stored operator node, a `CachedSource`) leaves rows of another data context + out, leaves them unchanged, and logs one warning per table with their number (logger + `orcapod.contexts.stored`). +- **What a v0.2 store looks like to v0.3.** + - Its tables are at the old layout. v0.3 never reads them as results and never writes to + them; they stay byte for byte as v0.2 left them and stay readable with v0.2. + - The first v0.3 run **computes every input again**, also for functions and data that did + not change, and stores the rows in the new tables. Before that run, reading a node + returns no rows. + - When v0.3 finds a table at an old layout, it logs one warning with the number of rows: + `Found 3 rows in stored table 'my_job/.../pdb_v1' written by an earlier version of + Orcapod (older table layout, data context std:v0.1:default). They are not used and are + left unchanged; they can be migrated later.` + - **Side-effecting nodes fail instead of delivering again.** A side effect is an external + action, and a completion stored by v0.2 cannot be matched with current inputs. Both + kinds of side-effecting node are covered: a side-effect pod (`@side_effect_pod`, + `@sink_pod`, `@tap_pod`), whose completions are in its invocation log, and a function + pod that takes the invocation context (`@function_pod(ctx_arg=...)`, + `@side_effect_function_pod`), whose completions are in its function tables. Such a + node that finds its tables at an old layout (or completions of another data context in + its current tables) raises `orcapod.errors.SideEffectLogDataContextError` (a + `StoreError`, so it fails the run whatever the `error_policy`) before it delivers or + writes anything. The error names the tables. Other function, operator and source nodes + of the same job are not refused. Use a fresh store for the node, or keep using v0.2 + for it. + - `orcapod.contexts.stored_data_contexts(store, record_path)` counts the rows of a table + by data context, for example `{"std:v1:default": 3}`; a table without the column reports + its rows under `std:v0.1:default`. +- **`std:v0.1:default` is no longer shipped.** `orcapod/contexts/data/v0.1.json` is removed + (it will be kept in `orcapod-sanctuary`), so `resolve_context("std:v0.1:default")` and + `resolve_context("v0.1")` raise `ContextResolutionError` naming it as the data context of + Orcapod before v0.3. A pipeline or job file saved by v0.2 still loads: each node that + names `std:v0.1:default` is loaded under the current data context with a warning, and + shows no rows until v0.3 has run (its v0.2 tables are at the old layout). +- **Removed: the v0 to v1 migration machinery.** `orcapod.errors.SchemaVersionError`, + `NodeConfig(ignore_schema=...)` (and `NodeConfig.ignore_schema`), the package + `orcapod.migrations` (`migrate_pipeline_db`, `migrate_result_db`, `MigrationResult`), the + command `orcapod migrate` (`pipeline-db`, `result-db`), `ResultCache.base_record_path`, + `ResultCache.set_ignore_schema` and `CachedFunctionPod.set_ignore_schema`. They converted + tables without a layout component to `pdb_v1` / `rdb_v1`, which v0.3 does not read. A + table without a layout component is now counted and reported like any other old layout, + where it used to raise `SchemaVersionError`. Remove `ignore_schema` from your code; the + migration to the v0.3 layouts is a later tool (ITL-913, ITL-914). +- **An input of another data context is refused.** A node, cache or cached source that + would store a row for an input whose data context differs from its own raises + `InputValidationError` naming both keys, before anything is computed, delivered or + written. For a function node this is checked for the whole input, so it fails the run + whatever the `error_policy`. Only one data context is shipped, so this needs a custom + context registry to happen. It applies to nodes, caches and cached sources used + directly. Inside a `PipelineJob` it is not reached: a job wraps every input in a source + node of the default data context, so an input of another data context is not seen as + such there (a known gap, ITL-951). `ResultCache.store` refuses an output of another + data context in the same way. +- **A full data context key must match its manifest.** `resolve_context` selects a manifest + by the version part of a key, and used to return it whatever the rest of the key said + (`"custom:v1:production"` gave `std:v1:default`). A full key that is not the manifest's + key now raises `ContextResolutionError`. +- **Hash values.** The content hash of a stream that has no producer (a bare + `ArrowTableStream`) covers its `_context_key` column, so it changes with the key. No other + hash value changes because of this entry. +- **Removed (introduced earlier in this unreleased cycle, never released):** + `orcapod.hashing.stored_hash_labels` and `orcapod.errors.SideEffectLogHashLabelError`. + Rows are told apart by `__data_context`, not by the label inside a hash. +- **Internal API.** `ResultCache.lookup`, `ResultCache.store` and + `ResultCache.get_all_records` take the caller's `data_context_key`. + + **Migration:** none is needed to run the function, operator and source nodes of a v0.2 + store under v0.3, at the price of computing everything again; side-effect nodes need a + fresh store. A tool that converts old-layout tables is planned (ITL-913, ITL-914); until + then they are kept, untouched, for it. + +#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910) + +**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1` +to `arrow_v0.2`.** starfix 0.4 (hash format 1) lost information for some inputs, so different +data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and +`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean +columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash +format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept. + +- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default + `source_id`s and the `record_id`s and source content hashes built from them, function + input-data hashes, pipeline base and entry ids, operator and cached-source record hashes, + invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The + hash label in front of each (`arrow_v0.2:000002...`) shows the new format. Pipeline hashes, + schema hashes, system-tag column names and table paths under the default + `table_scope="pipeline_hash"` do not move. Paths that embed a content hash + (`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or + `Batch` orders members by `record_id`, the order of members can change too. +- **Results stored by 0.2.0 are not reused.** Every table v0.3 writes is at a new path and + every row names its data context; see "Data context `std:v1:default`" below (ITL-944) for + what a v0.2 store looks like to v0.3. +- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator + carrying one works, and its type is kept. A `fixed_size_list[n]` hashes differently from a + `large_list` with the same values. pyarrow's canonical extension types (`arrow.uuid`, + `arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid` + column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and + `StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846). +- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed + silently: a non-nullable field holding a null; a zero-row table with a `map` column; two + schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a + column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`, + interval, `string_view` and `binary_view` columns, union and run-end-encoded types are + still refused, now with `ValueError` instead of `NotImplementedError`. +- Column order and struct field order are still not part of an Arrow hash. + #### A changed column type is refused when appending to a Delta table (ITL-884) Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows' @@ -80,6 +241,13 @@ stay queued for the next flush. A flush that fails only for other reasons still the backend's exception as its cause. (ITL-884 #290) - **`orcapod.errors.StoreError`** — the base of `StoreReadError` and `StoreWriteError`; `except StoreError` handles both. (ITL-884 #290) +- **`orcapod.errors.SideEffectLogDataContextError`** — a `StoreError` raised by a + side-effecting node (a side-effect pod, or a function pod that takes the invocation + context) whose stored completions are at an older table layout or belong to another data + context, before anything is delivered. It has `record_paths` (every table found), + `record_path` (the first), `found_context`, `current_context` and `node_label`. (ITL-944) +- **`orcapod.contexts.stored_data_contexts(database, record_path)`** — counts the rows of a + stored table by the data context that wrote them. (ITL-944) ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 1908ec144..6d00adcb1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -403,6 +403,48 @@ The pipeline hash uses a **resolver pattern** — `PipelineElementProtocol` obje Prefixes are computed from `SystemConstant` in `system_constants.py`. The `constants` singleton (with no global prefix) is used throughout. +### Stored tables: data context and layout versions + +Every row Orcapod stores carries `__data_context` (`constants.DATA_CONTEXT_COL`, +`large_string`): the key of the data context that wrote it (`std:v1:default` since v0.3; +developer, November 11, 2026, ITL-944). The key names one generation of hashing; rows of the +current data context are the current entries. + +| Table | Layout (last path component) | Constant | +|---|---|---| +| Function pipeline | `pdb_v2` | `PIPELINE_DB_SCHEMA_VERSION` | +| Function result | `rdb_v2` | `RESULT_DB_SCHEMA_VERSION` | +| Side-effect log | `tdb_v2` | `TRACKING_DB_SCHEMA_VERSION` | +| Operator cache | `odb_v1` | `OPERATOR_DB_SCHEMA_VERSION` | +| Cached source | `sdb_v1` | `SOURCE_DB_SCHEMA_VERSION` | + +- A change to the bookkeeping columns of a table kind moves its layout version, so the table + gets a new path. Never add, rename or retype a stored column without it. +- Every new stored table kind writes `__data_context` (`contexts/stored.with_data_context`), + and every read or lookup keeps rows of the current data context only + (`keep_rows_of_data_context`, or a lookup constraint on the column). +- A table at an older layout is never read as current and never modified. What a node does + about such tables, and about rows of another data context, is decided in one place, + `contexts/stored.account_for_stale_tables`: count and report, or, for a node that delivers + side effects (a `SideEffectPod` node, or a function node whose pod takes the invocation + context, `ctx_arg`), raise `SideEffectLogDataContextError` before any delivery. A new kind + of node goes through that function; never decide it locally. +- A released data context manifest (`contexts/data/.json`) is frozen: its sha256 is + pinned in `contexts/released_manifests.json`. A change that moves an existing golden hash + value for a reason other than a logical type's own change needs a new data context version + with its own manifest file, never an edit of a released one. +- In memory the data context column is still `_context_key` (`ColumnConfig(context=True)`). +- A writer refuses an input of another data context than its own + (`contexts/stored.require_same_data_context`, `InputValidationError` naming both keys) + before anything is written; a whole-input condition is checked outside the row error + policy. The eventual behaviour is ITL-951. +- `std:v0.1:default` is not shipped; saved files that name it load through + `contexts.resolve_saved_context`. There is no `SchemaVersionError`, `ignore_schema` or + `orcapod.migrations`: do not reintroduce a per-layout migration path without a decision. +- Logical types and the value handlers that hash their Python values are registered in + `contexts/logical_types.json`, never in a data context manifest: a logical type is + versioned by its own identifier, and that file is not frozen. + ### System tag evolution rules 1. **Name-preserving** — single-stream ops (filter, select, map). Column name and value pass diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 695708a20..4afdec29b 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2442,3 +2442,149 @@ results. Known compromise: the struct's children are nullable at the Arrow level, because Polars drops nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null instead. Tighten under a new storage version once nested nullability survives Polars. + +--- + +## `src/orcapod/hashing/arrow_hashers.py` — Arrow hash format (starfix) + +### HS2 — starfix hash format 1 collided on nested lists, null structs and Booleans +**Status:** resolved +**Severity:** high +**Issue:** ITL-910 (fixes the starfix side of ITL-848; ITL-927) + +`StarfixArrowHasher` used starfix 0.4 (hash format 1, label `arrow_v0.1`), which lost +information for some inputs, so different data reached the same row hash: nested-list lengths +(`[[[1]], [[2], [3]]]` and `[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are +all null, and non-nullable Boolean columns (`[True]` and `[True, False, False]`); a Boolean +column's hash also depended on chunking. starfix 0.4 also could not hash `fixed_size_list` +data, so a source with such a column failed at construction (ITL-927). + +**Fix:** starfix 0.5 (hash format 2, prefix `00 00 02`) under the new label `arrow_v0.2` +(developer's label rule: the label moves whenever any table hash can change). Every +Arrow-derived hash moves; the golden files were regenerated once, explicitly. Order options +stay off. pyarrow's canonical extension types and `hash_schema` on extension columns hash with +their extension identity instead of as their storage type. + +Stored rows: under the default `table_scope`, table paths do not move (developer, November +11, 2026: only a change of a table's schema goes into its path; a value-only change stays in +the same table), so a v0.3 node writes into the same tables as a v0.2 store. The first decision +that day was to refuse such a store (`StoreHashLabelError`); the developer reversed it in the +ITL-938 design session: v0.3 does not refuse a v0.2 store. It ignores rows hashed under +`arrow_v0.1`, computes every input again under `arrow_v0.2`, and leaves the old rows in place, +unchanged and distinguishable, for a later migration (ITL-913, ITL-914). + +Key lookups cannot match a row of another label. Every read of a whole table drops the rows +of another label through one function, `hashing/stored_hash_labels.keep_rows_hashed_under` +(without it: 4 rows for 2 inputs): `FunctionJobNode._fetch_joined_records` (phase 1, +`get_all_records`, `as_table`, `iter_data`, `as_source`, loaded jobs) on +`__pipeline_base_entry_id`; `ResultCache.get_all_records` on `__input_data_hash`; +`OperatorJobNode._read_stored_records` and `CachedSource._read_cached_records` on the starfix +format at the start of the `_record_hash` record id. Each reader logs one warning with the +total number of rows it left out (`report_skipped_rows`; developer: total counts are enough, +eventually old entries should be detected and a migration suggested). +`stored_hash_labels(database, record_path)` counts a table's rows by label. + +Side-effect invocation logs are the one exception (developer, same day): they are read by key +only, so deliveries logged under `arrow_v0.1` would not be found and every side effect would +be delivered again. A side-effect node whose log holds such rows raises +`SideEffectLogHashLabelError` before delivering anything. The developer wants a separate, +later discussion of what side-effect logs hold. + +Known limits, follow-ups: +- The warning is logged once per reader object (node, result cache, cached source) and + table, not once per process: stored paths are relative to the store, so a process-wide + record of reported tables would silence a second store with the same layout. +- A global hash-version identifier stored with every row is being designed (not decided); it + would also give operator-node and cached-source rows a label. +- Operator-node and cached-source record hashes carry no label, only the starfix format at + the start of the hex digest. That separates `arrow_v0.1` (format 1) from `arrow_v0.2` + (format 2) exactly, but a future label that keeps the starfix format could not be told apart + there. Storing the label with those rows (a schema change) would close that gap. +- The node content hash does not separate the generations: a source with an explicit + `source_id` gives the same node content hash under both labels. +- A `DeltaTableDatabase` object keeps the table version it read first, so a job object that + read a table before a run does not see the rows the run added until the store is reopened + (not specific to hash labels; seen when re-reading a node of the same job after an + incremental run). +- Tables placed under a content hash (`table_scope="content_hash"`, `CachedSource` without an + explicit `source_id`) move to new paths when the content hash moves, so their v0.2 rows are + in another table. + +--- + +## `src/orcapod/contexts/` — data context of stored rows + +### DC1 — a stored row did not say which generation of hashing wrote it +**Status:** resolved, with open follow-ups listed below +**Severity:** high +**Issue:** ITL-944 + +The data context key (`std:v0.1:default`) never changed although its manifest +`contexts/data/v0.1.json` was edited in place 44 times, so the key stored in result rows +(`_context_key`) and pipeline rows (`__input_data_context_key`) told a reader nothing. +Operator-cache, cached-source and side-effect tables stored no key at all. PR #309 told the +rows of v0.2 from the rows of v0.3 by the label inside each hash, which names an algorithm, +not a generation, and is missing from operator-cache and cached-source record ids. + +**Fix:** (developer's decisions of November 11, 2026, ITL-938 design document Parts 5.5 to 8) +- The default data context is `std:v1:default`, manifest `contexts/data/v1.json`. + `contexts/released_manifests.json` pins the sha256 of every released manifest (`null` while + a manifest is open); `tests/test_contexts/test_data_context_v1.py` fails when a pinned file + changes, when more than the default manifest is open, or when a final release ships an open + manifest. `v1.json` is pinned once, in the v0.3.0 release-preparation commit. +- Every stored row carries `__data_context` (`large_string`): function pipeline and result + tables, operator-cache tables, cached-source tables, side-effect invocation logs. It + replaces `__input_data_context_key` and the result table's `_context_key`. The value is the + key of the node, cache or source that writes the row. +- Layouts moved: `pdb_v2`, `rdb_v2`, `tdb_v2`, and new components `odb_v1` (operator cache) + and `sdb_v1` (cached source, also after an explicit `cache_path`). +- Reads and lookups keep rows of the current data context only, through + `contexts/stored.keep_rows_of_data_context` or a lookup constraint on the column. A table + at an old layout is counted and reported once per reader object, never read as current, + never written. +- What a node does about stored rows of other generations is decided in one place, + `contexts/stored.account_for_stale_tables`: report and skip, or, for a node that delivers + side effects, raise `SideEffectLogDataContextError` before anything is delivered. Both + kinds of side-effecting node are covered (developer, November 11, 2026): `SideEffectPod` + nodes (invocation log) and function nodes whose pod takes the invocation context + (`ctx_arg`; pipeline and result tables). +- `orcapod.contexts.stored_data_contexts(database, record_path)` lists the data contexts a + table holds. The label filtering of PR #309 (`hashing/stored_hash_labels.py`) is removed. +- The result table also stores `__pf_var_function_version`, the full function version. + +Known limits, follow-ups: +- Row keys do not include the data context. A later data context under which a row's key is + unchanged collides with the old row on write (`skip_duplicates`), and the new row is not + stored. The design's answer is that migration relabels unaffected rows; until a migration + exists this must be revisited before `std:v2:default` is introduced. +- The content hash of a stream without a producer covers its `_context_key` column, so it + moves with the data context key. +- Old-layout tables are reported once per reader object, not once per process or job, and + counting their rows reads the whole table (the store protocol has no row count). +- The manifest has the fields of the ITL-938 design document, section 7.4 (`released_in`, + `previous`, `definitions`, `libraries`, `golden`, `changes`), and the logical types and + their value handlers moved to the unfrozen `contexts/logical_types.json`. The names under + `definitions` are labels only: nothing in the code reads them or checks that a definition + changed, and `changes` lists what is true on this branch (starfix format 2); the entries + for ITL-935 and ITL-942 are added when those land. `golden.digest` stays `null` until the + manifest is frozen, because the golden files still move. +- `v0.1.json` is removed (to live in `orcapod-sanctuary`); `std:v0.1:default` raises + `ContextResolutionError`, and a node saved under it loads under the current data context + with a warning (`contexts.resolve_saved_context`). +- The v0 to v1 machinery is removed: `SchemaVersionError`, `NodeConfig(ignore_schema=)`, + `orcapod.migrations`, `orcapod migrate`. A table without a layout component is an old + layout like `pdb_v1`. +- An input of another data context raises `InputValidationError` + (`contexts/stored.require_same_data_context`) before anything is written. The developer + wants the node's data context to win eventually: ITL-951. +- Found while testing that rule, not fixed here: a data context is not carried reliably in + memory. `ArrowTableStream` gives every row the stream object's data context and ignores + the `_context_key` column of its table (the two can disagree); a source's `data_context` + is not given to the rows of its stream; and `PipelineJob` wraps each input in a source + node of the default data context. So inside a job an input of another data context is + not seen as such, and the check only fires for nodes and caches driven directly. +- For the same reason there is no per-row data context check on the cache-hit path of a + function node (CodeRabbit, PR #313): every row of a stream carries the stream object's + data context, so the whole-input check covers hits and misses. To revisit with ITL-951. +- Open: whether the in-memory column `_context_key` (`ColumnConfig(context=True)`) is + renamed to `__data_context` as well. It is unchanged here. diff --git a/docs/concepts/arrays.md b/docs/concepts/arrays.md index 987d03582..fceaf7f53 100644 --- a/docs/concepts/arrays.md +++ b/docs/concepts/arrays.md @@ -28,11 +28,11 @@ for tag, data in scale.pod(source).iter_data(): !!! warning "v0.3 changed the array format" Orcapod v0.2 stored arrays as `.npy` files under the extension name `numpy.ndarray` and hashed the file. v0.3 replaces both, as described below, and does not migrate older - stores. Under v0.3, every node that handles arrays computes again into new tables, and its - v0.2 results are no longer returned. They stay on disk, readable with v0.2. Nodes - downstream of an array column get new pipeline and operator tables too; a function pod - among them that takes and returns no array reuses its stored results. Start v0.3 - pipelines with a fresh store, or keep using v0.2 for an existing one. A migration is + stores. Every node that handles arrays, and every node downstream of an array column, gets + new tables in v0.3. Because v0.3 also changed every hash of Arrow data, v0.3 reuses no + result of a store written by v0.2: it ignores the v0.2 rows and computes every input + again (see [A store written by an older version](function-pods.md#a-store-written-by-an-older-version)). + The v0.2 tables and rows stay on disk unchanged, readable with v0.2. A migration is planned after v0.3 (ITL-889). ## What counts as the same array diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 4a6a5aa77..eb2916e78 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -307,6 +307,39 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro This is the contract for now. An opt-in mode that tolerates store failures may be added later. +### A store written by an older version + +Every row Orcapod stores names the **data context** that wrote it, in the column +`__data_context`: `std:v1:default` since v0.3. The data context is one key for the whole of +Orcapod's hashing, so rows that share it were keyed in exactly the same way (see +[The data context of a stored row](../reference/hashing.md#the-data-context-of-a-stored-row)). +Tables written before v0.3 have no such column and are stored at an older table layout +(`pdb_v1`, `rdb_v1`). Orcapod **does not read them as results**: it does not refuse the +store, and it does not reuse those rows. + +- The first run of v0.3 on a store written by v0.2 computes every input again and stores the + rows in its own tables (`pdb_v2`, `rdb_v2`). +- Reading a node (`as_table()`, `iter_data()`, `get_all_records()`) returns the rows of the + current data context only: nothing before that first run, one row per input after it. +- The old tables are never changed or deleted. They stay readable with the version that + wrote them and are kept for a later migration. To see which data contexts a table holds, + use `orcapod.contexts.stored_data_contexts`. +- When Orcapod finds a table at an older layout, it logs one warning with the number of + rows: `Found 3 rows in stored table '...' written by an earlier version of Orcapod (older + table layout, data context std:v0.1:default). They are not used and are left unchanged; + they can be migrated later.` +- Nodes that perform side effects are the exception: side-effect pods, and function pods + that take the invocation context (`ctx_arg`). A side effect is an external action, and + completions stored by v0.2 cannot be matched with current inputs, so running the node + would do everything a second time. Instead the node raises + `SideEffectLogDataContextError` (from `orcapod.errors`, a `StoreError`), naming the + tables, before it does anything. Use a fresh store for that node, or keep using the + version that wrote them. + +Each result row also records the full version string of the function that produced it, in +the column `__pf_var_function_version` (for example `v3.7`); only the major part is in the +table path. + ## Multiple input streams If you pass multiple streams to a function pod, they are automatically joined (using diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 13ddcf73a..b343abb0d 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -132,24 +132,24 @@ each suffix is the stream's `pipeline_hash()` (sites 1b and 6). | Site | Row | Value (hex) | |------|-----|-------------| -| 7 | `alice` `base_entry_id` | `6172726f775f76302e313a00000113f967f1187ac2982e65b6afbf324d264d50a6efe3357a35f09b8a2b32454d61` | -| 7 | `bob` `base_entry_id` | `6172726f775f76302e313a000001b6162a6faa76b5e065e89849acc09f41edfb5bb1a7c7ea626fd2920de6046586` | -| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 7 | `alice` `base_entry_id` | `6172726f775f76302e323a000002d147e6dfdc31690391034ed592bb75d4c4407034504165d168717c0779b64d3f` | +| 7 | `bob` `base_entry_id` | `6172726f775f76302e323a000002250edea78c481fa87d262c5938c9288a5538a17a10636b54a6ea83a18c77eb82` | +| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | -> The hex prefix `6172726f775f76302e31` decodes to the ASCII string `arrow_v0.1` — the +> The hex prefix `6172726f775f76302e32` decodes to the ASCII string `arrow_v0.2` — the > `method` portion of the `b"{method}:{digest}"` format described in §3. #### §4 — Side-Effect Record ID & Invocation Hash | Site | Row | Value | |------|-----|-------| -| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | > Both rows share the same `semantic_v0.1:1f71386f...` component — this is the > `SideEffectPodStream.pipeline_hash()`, which depends only on the pod and its upstream -> topology, not on row content. The `arrow_v0.1:...` component is the per-row `record_id` +> topology, not on row content. The `arrow_v0.2:...` component is the per-row `record_id` > from site 9, which differs per row. #### §5 — Data Function URI Hash @@ -334,7 +334,7 @@ topology suffix, encoding the full join lineage into each column name. | Field | Value | |---|---| | **Inputs** | Full raw Arrow table (all columns, before any system tag injection) | -| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — versioned SHA-256 via the `starfix` Rust crate | +| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — starfix hash format 2: the 3-byte format number `00 00 02` followed by a SHA-256 digest (`starfix` 0.5, byte-identical to the Rust crate) | | **Output format** | Truncated hex `str` (length = `path_n_char`) | | **Uniqueness guarantee** | Unique per raw table content; changes if any cell value changes | | **Known exclusions** | Used **only** as a fallback when no explicit `source_id` is provided. A user-supplied `source_id` bypasses this computation entirely. | @@ -618,3 +618,138 @@ print(hasher.hash_object(np.arange(6, dtype=np.float32).reshape(2, 3)).to_string print(hasher.hash_object(np.array(2.5)).to_string()) # ndarray_v1:47f182d862f5dd597d88836deddaed609ed94490827302e75e8364e877633cae ``` + +## 9. Arrow Table Hash (`arrow_v0.2`) + +Every Arrow-derived hash in the sites above (row data hashes, `record_id`s built from a +default `source_id`, base and pipeline entry ids, operator and cached-source record hashes, +`pa.Table` and pandas values) comes from `StarfixArrowHasher.hash_table`, which returns +`ContentHash(method="arrow_v0.2", digest=D)`. `D` is 35 bytes: the starfix hash format +number `00 00 02` followed by a SHA-256 digest, computed by `starfix` 0.5 (hash format 2) and +byte-identical to the Rust `starfix` crate. The byte layout is specified in starfix's +`docs/byte-layout-spec.md`. + +**Label rule.** The label `arrow_vX.Y` moves whenever the hash of any table can change, +whatever the cause. Its number is not kept equal to the starfix format number; the data +context's changelog records which starfix format each label uses. + +| Label | starfix | Format prefix | Orcapod | +|---|---|---|---| +| `arrow_v0.1` | 0.4 | `00 00 01` | up to v0.2 | +| `arrow_v0.2` | 0.5 | `00 00 02` | v0.3 | + +**What is part of the hash.** Column names, Arrow types (with large and non-large layouts of +strings, binaries and lists hashing alike), nullability, every value, row order, list element +order, list lengths at every depth, null structs (distinct from structs whose fields are all +null) and the row count. Column order and struct field order are not part of the hash. How a +table is split into chunks or slices is not part of it either. A `fixed_size_list[n]` +column hashes differently from a `large_list` column with the same values. Extension +columns are hashed as their storage type plus their extension name and metadata, including +pyarrow's canonical extension types (`arrow.uuid`, `arrow.fixed_shape_tensor`); only +`ARROW:extension:*` metadata is hashed, all other metadata is ignored. + +**What cannot be hashed.** starfix raises `ValueError`, naming the column path and type, for: +data of `null`, `map`, interval, `string_view` and `binary_view` columns (whatever the row +count; their schemas hash); union and run-end-encoded types; two schema nodes sharing a path +(duplicate column names, or a name containing `/` that aliases a struct child, such as a +column `"a/b"` next to a struct column `a` with a child `b`); and a non-nullable field holding +a null. A source whose table has such a column fails when it is constructed. + +#### The data context of a stored row + +A label names the algorithm that produced one hash. It does not say which generation of +Orcapod's hashing a stored row belongs to. That is the **data context**: one key that changes +whenever any existing hash value can change (the hashers, the identity definitions, the +record-id recipes, the libraries behind them). Orcapod v0.3 writes `std:v1:default`; +everything stored before v0.3 is the earlier generation, `std:v0.1:default`. The manifest of +the current data context is `orcapod/contexts/data/v1.json`. It is frozen at the release of +the library version that contains it and never edited afterwards. It names the hashers, the +definitions that live in code, the libraries the hashes depend on and what changed from the +previous data context. Logical types are not part of it: they are registered in +`orcapod/contexts/logical_types.json` and versioned by their own identifier. + +Every row Orcapod stores names its data context in the column `__data_context`: + +| Table | Layout (last path component) | +|---|---| +| Function-node pipeline table | `pdb_v2` | +| Function result table | `rdb_v2` | +| Operator-node cache table | `odb_v1` | +| Cached-source table | `sdb_v1` | +| Side-effect invocation log | `tdb_v2` | + +The layout component is always the last component of a table's path, also when a +`CachedSource` is given an explicit `cache_path`. + +**Rows of the current data context are the current entries.** Key lookups match rows of the +current data context only, and every read of a whole table (a node's `as_table()`, +`iter_data()`, `get_all_records()`, `as_source()`, a cached function's +`get_all_cached_outputs()`, a stored operator node, a `CachedSource`) leaves rows of another +data context out. Those rows are never changed or deleted, and a read that leaves rows out +logs one warning per table with their number (logger `orcapod.contexts.stored`). + +**Tables written before v0.3** have no `__data_context` column and sit at an older layout +(`pdb_v1`, `rdb_v1`, `tdb_v1`; operator-node and cached-source tables, and the function +tables of Orcapod v0.1, had no layout component). Orcapod does not read them as results and never modifies them. When it finds +one it logs one warning with the number of rows: + +```text +Found 3 rows in stored table '...' written by an earlier version of Orcapod (older table +layout, data context std:v0.1:default). They are not used and are left unchanged; they can +be migrated later. +``` + +Nodes that perform side effects are the exception. A completion is looked up by its key, so +completions stored under another data context would not be found and every side effect would +be performed again. There are two kinds: a side-effect pod (`@side_effect_pod`, `@sink_pod`, +`@tap_pod`), whose completions are in its invocation log, and a function pod that takes the +invocation context (`@function_pod(ctx_arg=...)`), whose completions are in its pipeline and +result tables. Either raises `SideEffectLogDataContextError`, naming the tables, before +doing anything when it finds its tables at an older layout or completions of another data +context in its current tables. + +A node stores rows for inputs of its own data context only: an input of another data context +raises `InputValidationError` naming both keys before anything is written. This holds for +nodes, caches and cached sources used directly; inside a `PipelineJob` every input is wrapped +in a source node of the default data context, so the refusal is not reached there. (What +should eventually happen instead is tracked in ITL-951.) `std:v0.1:default` itself is not shipped +with v0.3 and does not resolve; a pipeline or job file saved by v0.2 loads under the current +data context with a warning. + +`stored_data_contexts(database, record_path)` reads a stored table and counts its rows by +data context. A table without the column reports all its rows under `std:v0.1:default`. + +```python +from orcapod import PipelineJob, function_pod +from orcapod.contexts import stored_data_contexts +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + + +@function_pod(output_keys="doubled") +def double_for_contexts(x: int) -> int: + return 2 * x + + +context_store = InMemoryArrowDatabase() +context_job = PipelineJob(name="contexts_demo", store=context_store) +with context_job: + rows = DictSource(data=[{"k": "a", "x": 1}, {"k": "b", "x": 2}], tag_columns=["k"]) + _ = double_for_contexts.pod(rows, label="dbl") +context_job.run() + +for path in sorted(context_store.list_sources()): + print(path[-1], stored_data_contexts(context_store, path)) +# rdb_v2 {'std:v1:default': 2} +# pdb_v2 {'std:v1:default': 2} +``` + +```python +import pyarrow as pa +from orcapod.contexts import get_default_context + +hasher = get_default_context().arrow_hasher +digest = hasher.hash_table(pa.table({"x": pa.array([1, 2, 3], pa.int64())})) +print(digest.method, digest.digest[:3].hex()) +# arrow_v0.2 000002 +``` diff --git a/pyproject.toml b/pyproject.toml index efd70d255..8f0e26bbf 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,7 +19,7 @@ dependencies = [ "graphviz>=0.21", "gitpython>=3.1.45", "universal-pathlib>=0.3.8", - "starfix~=0.4.0", + "starfix>=0.5.0,<0.6", "tzdata>=2024.1", "uuid-utils>=0.11.1", "s3fs>=2025.12.0", diff --git a/src/orcapod/cli/__init__.py b/src/orcapod/cli/__init__.py index dbaba6004..53acef21d 100644 --- a/src/orcapod/cli/__init__.py +++ b/src/orcapod/cli/__init__.py @@ -9,7 +9,6 @@ import typer from orcapod.cli.warm_cache import warm_cache -from orcapod.cli.migrate import migrate_app app = typer.Typer( name="orcapod", @@ -24,4 +23,3 @@ def _main() -> None: app.command("warm-cache")(warm_cache) -app.add_typer(migrate_app, name="migrate") diff --git a/src/orcapod/cli/migrate.py b/src/orcapod/cli/migrate.py deleted file mode 100644 index 69181e823..000000000 --- a/src/orcapod/cli/migrate.py +++ /dev/null @@ -1,172 +0,0 @@ -"""``orcapod migrate`` sub-commands. - -Provides ``orcapod migrate pipeline-db`` and ``orcapod migrate result-db`` -for upgrading v0 pipeline/result DB tables to the v1 schema. -""" - -from __future__ import annotations - -import json - -import typer - -migrate_app = typer.Typer( - name="migrate", - help="Migrate Orcapod pipeline and result DB tables to the current schema version.", - no_args_is_help=True, -) - - -def _result_path_from_node_path(node_path_str: str) -> tuple[str, ...]: - """Derive the rdb record path (pod URI) from a node identity path string. - - A node identity path has the form ``pod/uri/schema:`` (and optionally - ``instance:`` for fine-grained paths). The result DB is scoped to - just the pod URI portion — i.e. all components before any ``schema:`` or - ``instance:`` segment. - - Args: - node_path_str: Slash-separated node identity path as passed on the CLI. - - Returns: - Tuple of path components with ``schema:`` and ``instance:`` suffix - components stripped. - """ - return tuple( - p - for p in node_path_str.split("/") - if not p.startswith("schema:") and not p.startswith("instance:") - ) - - -@migrate_app.command("pipeline-db") -def migrate_pipeline_db( - pipeline_db_path: str = typer.Argument( - ..., help="Path to the pipeline DB (Delta Lake root)." - ), - result_db_path: str = typer.Argument( - ..., help="Path to the result DB (Delta Lake root)." - ), - node_paths: list[str] = typer.Argument( - ..., - help="One or more bare v0 node paths (slash-separated, e.g. 'my_node/schema:abc123').", - ), - dry_run: bool = typer.Option( - False, "--dry-run", help="Count rows to migrate without writing." - ), - batch_size: int = typer.Option( - 500, "--batch-size", help="Rows processed per batch." - ), - progress: bool = typer.Option( - True, "--progress/--no-progress", help="Log progress messages." - ), - json_summary: bool = typer.Option( - False, "--json-summary", help="Print JSON summary to stdout on completion." - ), -) -> None: - """Migrate one or more pipeline DB node paths from v0 to v1 schema.""" - from orcapod.databases.delta_lake_databases import DeltaTableDatabase - from orcapod.migrations.pipeline_db import migrate_pipeline_v0_to_v1 - - pipeline_db = DeltaTableDatabase(base_path=pipeline_db_path) - result_db = DeltaTableDatabase(base_path=result_db_path) - - for node_path_str in node_paths: - pipeline_path = tuple(node_path_str.split("/")) - # The rdb is scoped to the pod URI (function_pod.uri), which is the - # node identity path with schema:/instance: suffix components removed. - result_path = _result_path_from_node_path(node_path_str) - - if progress: - typer.echo(f"Migrating pipeline DB: {pipeline_db_path}") - typer.echo(f" node path: {node_path_str}") - - result = migrate_pipeline_v0_to_v1( - pipeline_db=pipeline_db, - pipeline_path=pipeline_path, - result_db=result_db, - result_path=result_path, - dry_run=dry_run, - batch_size=batch_size, - progress=progress, - ) - - if progress: - typer.echo( - f" migrated: {result.rows_migrated} " - f"skipped (already v1): {result.rows_skipped} " - f"unresolvable: {result.rows_unresolvable}" - ) - typer.echo(f" elapsed: {result.elapsed_s:.1f}s") - - if json_summary: - summary = { - "rows_total": result.rows_total, - "rows_migrated": result.rows_migrated, - "rows_skipped": result.rows_skipped, - "rows_unresolvable": result.rows_unresolvable, - "elapsed_s": result.elapsed_s, - "dry_run": result.dry_run, - } - typer.echo(json.dumps(summary)) - - -@migrate_app.command("result-db") -def migrate_result_db( - result_db_path: str = typer.Argument( - ..., help="Path to the result DB (Delta Lake root)." - ), - record_paths: list[str] = typer.Argument( - ..., help="One or more bare v0 record paths (slash-separated)." - ), - dry_run: bool = typer.Option( - False, "--dry-run", help="Count rows to migrate without writing." - ), - batch_size: int = typer.Option( - 500, "--batch-size", help="Rows processed per batch." - ), - progress: bool = typer.Option( - True, "--progress/--no-progress", help="Log progress messages." - ), - json_summary: bool = typer.Option( - False, "--json-summary", help="Print JSON summary to stdout on completion." - ), -) -> None: - """Migrate one or more result DB record paths from v0 to v1 schema.""" - from orcapod.databases.delta_lake_databases import DeltaTableDatabase - from orcapod.migrations.result_db import migrate_result_v0_to_v1 - - result_db = DeltaTableDatabase(base_path=result_db_path) - - for record_path_str in record_paths: - result_path = tuple(record_path_str.split("/")) - - if progress: - typer.echo(f"Migrating result DB: {result_db_path}") - typer.echo(f" record path: {record_path_str}") - - result = migrate_result_v0_to_v1( - result_db=result_db, - result_path=result_path, - dry_run=dry_run, - batch_size=batch_size, - progress=progress, - ) - - if progress: - typer.echo( - f" migrated: {result.rows_migrated} " - f"skipped (already v1): {result.rows_skipped}" - ) - typer.echo(f" elapsed: {result.elapsed_s:.1f}s") - - if json_summary: - summary = { - "rows_total": result.rows_total, - "rows_migrated": result.rows_migrated, - "rows_skipped": result.rows_skipped, - "rows_unresolvable": result.rows_unresolvable, - "elapsed_s": result.elapsed_s, - "dry_run": result.dry_run, - } - typer.echo(json.dumps(summary)) diff --git a/src/orcapod/contexts/__init__.py b/src/orcapod/contexts/__init__.py index ea4b4ab20..0105572fa 100644 --- a/src/orcapod/contexts/__init__.py +++ b/src/orcapod/contexts/__init__.py @@ -15,7 +15,7 @@ context = resolve_context() # Get specific version - context = resolve_context("v0.1") + context = resolve_context("v1") # Use context components registry = context.semantic_type_registry @@ -32,7 +32,10 @@ from orcapod.protocols import semantic_types_protocols as sp from .core import ContextResolutionError, ContextValidationError, DataContext +from orcapod.system_constants import PRE_V1_DATA_CONTEXT_KEY + from .registry import JSONDataContextRegistry +from .stored import stored_data_contexts if TYPE_CHECKING: from pathlib import Path @@ -62,7 +65,7 @@ def resolve_context(context_info: str | DataContext | None = None) -> DataContex Args: context_info: One of: - None: Use default context - - str: Version string ("v0.1") or full key ("std:v0.1:default") + - str: Version string ("v1") or full key ("std:v1:default") - DataContext: Return as-is Returns: @@ -73,8 +76,8 @@ def resolve_context(context_info: str | DataContext | None = None) -> DataContex Examples: >>> context = resolve_context() # Default - >>> context = resolve_context("v0.1") # Specific version - >>> context = resolve_context("std:v0.1:default") # Full key + >>> context = resolve_context("v1") # Specific version + >>> context = resolve_context("std:v1:default") # Full key >>> context = resolve_context("latest") # Latest version """ # If already a DataContext, return as-is @@ -86,6 +89,37 @@ def resolve_context(context_info: str | DataContext | None = None) -> DataContex return registry.get_context(context_info) +def resolve_saved_context(context_key: str | None) -> DataContext: + """Resolve the data context named by a saved pipeline or job file. + + A file saved by Orcapod before v0.3 names ``std:v0.1:default``, which this + version cannot hash under. Such a node is loaded under the current default data + context, with a warning: results the earlier version stored for it are at an + older table layout and are not read. + + Args: + context_key: The ``data_context_key`` of a saved node, or ``None``. + + Returns: + The data context to load the node under. + + Raises: + ContextResolutionError: If the key names any other unknown data context. + """ + if context_key == PRE_V1_DATA_CONTEXT_KEY: + current = get_default_context() + logger.warning( + "A node was saved under the data context %r (Orcapod before v0.3). It is " + "loaded under the current data context %r; results stored under %r are " + "not read.", + context_key, + current.context_key, + context_key, + ) + return current + return resolve_context(context_key) + + def get_available_contexts() -> list[str]: """ Get list of all available context versions. @@ -112,7 +146,7 @@ def get_context_info(version: str) -> dict[str, Any]: Dictionary with context metadata Example: - >>> info = get_context_info("v0.1") + >>> info = get_context_info("v1") >>> print(info['description']) 'Initial stable release with basic Path semantic type support' """ @@ -208,7 +242,8 @@ def get_default_type_converter() -> "sp.TypeConverterProtocol": def create_registry( contexts_dir: str | None = None, schema_file: str | None = None, - default_version: str = "v0.1", + default_version: str = "v1", + logical_types_file: str | None = None, ) -> JSONDataContextRegistry: """ Create a custom context registry. @@ -219,6 +254,8 @@ def create_registry( contexts_dir: Directory containing context JSON files schema_file: JSON schema file for validation default_version: Default version to use + logical_types_file: The logical type registration file; defaults to the + one shipped in ``orcapod.contexts`` Returns: JSONDataContextRegistry instance @@ -228,7 +265,9 @@ def create_registry( >>> test_registry = create_registry("/path/to/test/contexts") >>> test_context = test_registry.get_context("test") """ - return JSONDataContextRegistry(contexts_dir, schema_file, default_version) + return JSONDataContextRegistry( + contexts_dir, schema_file, default_version, logical_types_file + ) def enable_file_hash_caching( @@ -311,7 +350,7 @@ def enable_file_hash_caching( raise RuntimeError( "enable_file_hash_caching(): no FileHandler registered for " "orcapod.File in the default context. This should not happen " - "with the standard v0.1 context." + "with the standard v1 context." ) base_hasher = existing_handler.file_hasher @@ -364,7 +403,7 @@ def enable_file_hash_caching( raise RuntimeError( "enable_file_hash_caching(): no DirectoryHandler registered for " "orcapod.Directory in the default context. This should not happen " - "with the standard v0.1 context." + "with the standard v1 context." ) existing_dir_hasher = existing_dir_handler.directory_hasher registry.register( @@ -387,6 +426,7 @@ def enable_file_hash_caching( "ContextResolutionError", # Main functions "resolve_context", + "resolve_saved_context", "get_available_contexts", "get_context_info", "get_default_context", @@ -399,4 +439,6 @@ def enable_file_hash_caching( # Advanced usage "create_registry", "JSONDataContextRegistry", + # Stored rows + "stored_data_contexts", ] diff --git a/src/orcapod/contexts/core.py b/src/orcapod/contexts/core.py index 65a30421b..3d162a74c 100644 --- a/src/orcapod/contexts/core.py +++ b/src/orcapod/contexts/core.py @@ -14,8 +14,8 @@ class DataContext: """Data context containing all versioned components needed for data interpretation. Attributes: - context_key: Unique identifier (e.g., "std:v0.1:default") - version: Version string (e.g., "v0.1") + context_key: Unique identifier (e.g., "std:v1:default") + version: Version string (e.g., "v1") description: Human-readable description type_converter: Type converter for Python ↔ Arrow conversion and registration. This is the single public API for all type operations. diff --git a/src/orcapod/contexts/data/schemas/context_schema.json b/src/orcapod/contexts/data/schemas/context_schema.json index 38c377a8f..972cd473a 100644 --- a/src/orcapod/contexts/data/schemas/context_schema.json +++ b/src/orcapod/contexts/data/schemas/context_schema.json @@ -15,9 +15,10 @@ "properties": { "context_key": { "type": "string", - "pattern": "^[a-zA-Z0-9_-]+:v[0-9]+\\.[0-9]+:[a-zA-Z0-9_-]+$", - "description": "Unique identifier for this context (e.g., 'std:v0.1:default')", + "pattern": "^[a-zA-Z0-9_-]+:v[0-9]+(\\.[0-9]+)?:[a-zA-Z0-9_-]+$", + "description": "Unique identifier for this context (e.g., 'std:v1:default')", "examples": [ + "std:v1:default", "std:v0.1:default", "std:v0.2:fast", "custom:v1.0:production" @@ -25,9 +26,10 @@ }, "version": { "type": "string", - "pattern": "^v[0-9]+\\.[0-9]+$", + "pattern": "^v[0-9]+(\\.[0-9]+)?$", "description": "Version identifier matching the filename", "examples": [ + "v1", "v0.1", "v1.0", "v2.3" @@ -41,6 +43,20 @@ "Enhanced version with timestamp support and improved hashing" ] }, + "released_in": { + "type": [ + "string", + "null" + ], + "description": "Library version whose release froze this file; null while that release is being prepared. A file whose released_in is set must not be edited" + }, + "previous": { + "type": [ + "string", + "null" + ], + "description": "Key of the data context this one follows (e.g. 'std:v0.1:default')" + }, "type_converter": { "$ref": "#/$defs/objectspec", "description": "ObjectSpec for the python-arrow type converter" @@ -69,6 +85,60 @@ "$ref": "#/$defs/objectspec", "description": "ObjectSpec for the function semantic hasher (used by FunctionHandler)" }, + "definitions": { + "type": "object", + "additionalProperties": { + "type": "string" + }, + "description": "A name for each definition that lives in code and not in a swappable component: identity compositions, record-id, entry-id and fold recipes, naming grammars, type conversion rules" + }, + "libraries": { + "type": "object", + "additionalProperties": { + "type": "string" + }, + "description": "Critical libraries the core hashes depend on, with their version" + }, + "golden": { + "type": "object", + "properties": { + "files": { + "type": "string" + }, + "algorithm": { + "type": "string" + }, + "digest": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "digest" + ], + "description": "Digest of the golden hash files this data context stands for; null while the manifest is open" + }, + "changes": { + "type": "array", + "items": { + "type": "object", + "properties": { + "definition": { + "type": "string" + }, + "change": { + "type": "string" + } + }, + "required": [ + "definition", + "change" + ] + }, + "description": "Which definitions differ from the previous data context" + }, "metadata": { "type": "object", "description": "Optional metadata about this context", @@ -118,7 +188,9 @@ "oneOf": [ { "type": "object", - "required": ["_class"], + "required": [ + "_class" + ], "properties": { "_class": { "type": "string", @@ -135,17 +207,27 @@ }, { "type": "object", - "required": ["_ref"], + "required": [ + "_ref" + ], "properties": { - "_ref": {"type": "string", "description": "Reference to a named component"} + "_ref": { + "type": "string", + "description": "Reference to a named component" + } }, "additionalProperties": false }, { "type": "object", - "required": ["_type"], + "required": [ + "_type" + ], "properties": { - "_type": {"type": "string", "description": "Dotted Python type string, e.g. 'pathlib.Path'"} + "_type": { + "type": "string", + "description": "Dotted Python type string, e.g. 'pathlib.Path'" + } }, "additionalProperties": false }, @@ -157,7 +239,12 @@ } }, { - "type": ["string", "number", "boolean", "null"], + "type": [ + "string", + "number", + "boolean", + "null" + ], "description": "Primitive values" } ] @@ -165,9 +252,9 @@ }, "examples": [ { - "context_key": "std:v0.1:default", - "version": "v0.1", - "description": "Initial stable release with extension type hashing support", + "context_key": "std:v1:default", + "version": "v1", + "description": "Data context of Orcapod v0.3", "type_converter": { "_class": "orcapod.semantic_types.universal_converter.UniversalTypeConverter", "_config": {} @@ -175,16 +262,22 @@ "arrow_hasher": { "_class": "orcapod.hashing.arrow_hashers.StarfixArrowHasher", "_config": { - "hasher_id": "arrow_v0.1", - "type_converter": {"_ref": "type_converter"}, - "semantic_hasher": {"_ref": "semantic_hasher"} + "hasher_id": "arrow_v0.2", + "type_converter": { + "_ref": "type_converter" + }, + "semantic_hasher": { + "_ref": "semantic_hasher" + } } }, "semantic_hasher": { "_class": "orcapod.hashing.semantic_hashing.semantic_hasher.SemanticAwarePythonHasher", "_config": { "hasher_id": "semantic_v0.1", - "type_handler_registry": {"_ref": "python_type_handler_registry"} + "type_handler_registry": { + "_ref": "python_type_handler_registry" + } } }, "metadata": { @@ -197,4 +290,4 @@ } } ] -} \ No newline at end of file +} diff --git a/src/orcapod/contexts/data/v0.1.json b/src/orcapod/contexts/data/v0.1.json deleted file mode 100644 index a830d71cd..000000000 --- a/src/orcapod/contexts/data/v0.1.json +++ /dev/null @@ -1,171 +0,0 @@ -{ - "context_key": "std:v0.1:default", - "version": "v0.1", - "description": "Initial stable release with extension type hashing support", - "file_hasher": { - "_class": "orcapod.hashing.file_hashers.FileHasher", - "_config": { - "algorithm": "sha256" - } - }, - "directory_hasher": { - "_class": "orcapod.hashing.directory_hashers.BasicDirectoryHasher", - "_config": { - "file_hasher": {"_ref": "file_hasher"}, - "algorithm": "sha256" - } - }, - "type_converter": { - "_class": "orcapod.semantic_types.universal_converter.UniversalTypeConverter", - "_config": { - "logical_type_registry": { - "_class": "orcapod.logical_types.registry.LogicalTypeRegistry", - "_config": { - "logical_types": [ - { - "_class": "orcapod.logical_types.builtin_logical_types.LogicalPath", - "_config": {} - }, - { - "_class": "orcapod.logical_types.builtin_logical_types.LogicalUPath", - "_config": {} - }, - { - "_class": "orcapod.logical_types.builtin_logical_types.LogicalUUID", - "_config": {} - }, - { - "_class": "orcapod.logical_types.file_type.LogicalFile", - "_config": {} - }, - { - "_class": "orcapod.logical_types.directory_type.LogicalDirectory", - "_config": {} - }, - { - "_class": "orcapod.logical_types.numpy_type.LogicalNumpyArray", - "_config": {} - }, - { - "_class": "orcapod.logical_types.pandas_type.LogicalPandasDataFrame", - "_config": {} - }, - { - "_class": "orcapod.logical_types.pandas_type.LogicalPandasSeries", - "_config": {} - } - ], - "factories": [ - { - "factory": { - "_class": "orcapod.logical_types.dataclass_logical_type_factory.DataclassLogicalTypeFactory", - "_config": {} - }, - "category": "orcapod.dataclass", - "python_bases": [{"_type": "builtins.object"}] - }, - { - "factory": { - "_class": "orcapod.logical_types.pydantic_logical_type_factory.PydanticLogicalTypeFactory", - "_config": {} - }, - "category": "orcapod.pydantic", - "python_bases": [{"_type": "pydantic.BaseModel"}] - }, - { - "factory": { - "_class": "orcapod.logical_types.list_logical_type_factory.ListLogicalTypeFactory", - "_config": {} - }, - "category": "list" - }, - { - "factory": { - "_class": "orcapod.logical_types.list_logical_type_factory.ListLogicalTypeFactory", - "_config": {} - }, - "category": "set" - } - ] - } - } - } - }, - "function_semantic_hasher": { - "_class": "orcapod.hashing.semantic_hashing.function_info_extractors.FunctionSignatureExtractor", - "_config": { - "include_module": true, - "include_defaults": true, - "type_converter": {"_ref": "type_converter"} - } - }, - "python_type_handler_registry": { - "_class": "orcapod.hashing.semantic_hashing.type_handler_registry.PythonTypeHandlerRegistry", - "_config": { - "handlers": [ - [{"_type": "builtins.bytes"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.BytesHandler", "_config": {}}], - [{"_type": "builtins.bytearray"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.BytesHandler", "_config": {}}], - [{"_type": "uuid.UUID"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.UUIDHandler", "_config": {}}], - [{"_type": "orcapod.logical_types.file_type.File"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FileHandler", "_config": {"file_hasher": {"_ref": "file_hasher"}}}], - [{"_type": "orcapod.logical_types.directory_type.Directory"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.DirectoryHandler", "_config": {"directory_hasher": {"_ref": "directory_hasher"}}}], - [{"_type": "types.FunctionType"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FunctionHandler", "_config": {"function_info_extractor": {"_ref": "function_semantic_hasher"}}}], - [{"_type": "types.BuiltinFunctionType"},{"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FunctionHandler", "_config": {"function_info_extractor": {"_ref": "function_semantic_hasher"}}}], - [{"_type": "types.MethodType"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FunctionHandler", "_config": {"function_info_extractor": {"_ref": "function_semantic_hasher"}}}], - [{"_type": "builtins.type"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.TypeObjectHandler", "_config": {"type_converter": {"_ref": "type_converter"}}}], - [{"_type": "types.GenericAlias"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.GenericAliasHandler", "_config": {}}], - [{"_type": "types.UnionType"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.UnionTypeHandler", "_config": {}}], - [{"_type": "typing._GenericAlias"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.GenericAliasHandler", "_config": {}}], - [{"_type": "typing._SpecialForm"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.SpecialFormHandler", "_config": {}}], - [{"_type": "pyarrow.Table"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.ArrowTableHandler", "_config": {}}], - [{"_type": "pyarrow.RecordBatch"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.ArrowTableHandler", "_config": {}}], - [{"_type": "numpy.ndarray"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.NumpyArrayHandler", "_config": {}}], - [{"_type": "pandas.DataFrame"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PandasDataFrameHandler", "_config": {}}], - [{"_type": "pandas.Series"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PandasSeriesHandler", "_config": {}}], - [{"_type": "polars.Expr"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PolarsExprHandler", "_config": {}}], - [{"_type": "orcapod.types.Schema"}, {"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.SchemaHandler", "_config": {"type_converter": {"_ref": "type_converter"}}}] - ] - } - }, - "semantic_hasher": { - "_class": "orcapod.hashing.semantic_hashing.semantic_hasher.SemanticAwarePythonHasher", - "_config": { - "hasher_id": "semantic_v0.1", - "type_handler_registry": { - "_ref": "python_type_handler_registry" - } - } - }, - "arrow_hasher": { - "_class": "orcapod.hashing.arrow_hashers.StarfixArrowHasher", - "_config": { - "hasher_id": "arrow_v0.1", - "type_converter": {"_ref": "type_converter"}, - "semantic_hasher": {"_ref": "semantic_hasher"} - } - }, - "metadata": { - "created_date": "2026-06-24", - "author": "OrcaPod Core Team", - "changelog": [ - "Initial release with Path semantic type support", - "Basic SHA-256 hashing for files and objects", - "Arrow logical serialization method", - "Introduced arrow_v0.1 StarfixArrowHasher using starfix ArrowDigester for cross-language-compatible Arrow hashing", - "Hard cut: replaced shape-based SemanticTypeRegistry with extension-type hashing; renamed all hashing classes to cleaner names", - "Removed PathHandler and UPathHandler; registered FileHandler for orcapod.File content hashing (ITL-450)", - "Migrated LogicalFile Arrow storage from plain path string to JSON {\"path\": \"...\"} for consistency with LogicalDirectory (ITL-451)", - "Added orcapod.Directory content-identified type with recursive Merkle tree hashing, LogicalDirectory Arrow extension, ignore filter support, and DirectoryHandler (ITL-451)", - "Added numpy.ndarray as a native value type via LogicalNumpyArray (large_binary/.npy) and NumpyArrayHandler; object-dtype arrays are rejected eagerly (ITL-460)", - "Added spikeinterface.BaseRecording as a native value type via LogicalSIRecording (large_string/JSON) and SIRecordingHandler (SHA-256 of JSON bytes); auto-registered when spikeinterface is installed via _optional entries in v0.1.json; added _optional flag support to parse_objectspec for optional-extras types (ITL-459)", - "Added pandas.DataFrame and pandas.Series as native value types via LogicalPandasDataFrame and LogicalPandasSeries (Arrow IPC / large_binary), with index preservation and PandasDataFrameHandler / PandasSeriesHandler for content hashing using StarfixArrowHasher (PLT-1869)", - "Added spikeinterface.BaseSorting as a native value type via LogicalSISorting (large_string/JSON) and SISortingHandler (SHA-256 of JSON bytes); auto-registered when spikeinterface is installed via _optional entries in v0.1.json (ITL-468)", - "Added spikeinterface.core.motion.Motion as a native value type via LogicalSIMotion (large_binary/.npz) and SIMotionHandler (SHA-256 of .npz bytes); auto-registered when spikeinterface is installed via _optional entries in v0.1.json (ITL-470)", - "Added spikeinterface.core.SortingAnalyzer as a native value type via LogicalSISortingAnalyzer (large_string/JSON path reference {folder, format}) and SISortingAnalyzerHandler (SHA-256 of folder path string, phase 1); auto-registered when spikeinterface is installed via _optional entries in v0.1.json (ITL-469)", - "Wired file_hasher reference into BasicDirectoryHasher so op.Directory hashing uses the same FileContentHasherProtocol as op.File hashing; required file_hasher constructor arg enforces explicit injection", - "Moved SpikeInterface types (LogicalSIRecording, LogicalSISorting, LogicalSIMotion, LogicalSISortingAnalyzer) and their handlers to the standalone orcapod-extension-spikeinterface package; removed all SI _optional entries from this spec and SI deps from pyproject.toml; introduced OrcapodExtension protocol and op.register_extension() for normalized extension registration (ITL-473)", - "Registered SchemaHandler for orcapod.types.Schema in the context registry; moved handler dispatch before _is_structure check in hash_object so SchemaHandler takes priority over _expand_mapping for Schema objects (ITL-639)", - "Registered PolarsExprHandler for polars.Expr: hashes an expression as ContentHash('polars_expr_v1', sha256 of the canonical JSON of an Orcapod-owned expression tree parsed strictly from Expr.meta.serialize(format='json')); expressions outside the supported subset (Python functions, multi-column selectors, aggregations, alias, when/then, non-finite float literals, pl.Series literals) raise ValueError naming the expression (ITL-828)", - "Hard cut (v0.3): numpy.ndarray moves to the value-based array format. LogicalNumpyArray stores extension orcapod.ndarray over struct (Orcapod dtype tag, shape, little-endian C-order packed bytes; extension metadata {\"version\":1}) and NumpyArrayHandler hashes by value with label ndarray_v1 (array_digest_v1 over the stored row). The old extension name numpy.ndarray (.npy bytes, label sha256) is no longer registered or written; v0.2 stores holding arrays are not migrated (ITL-888)" - ] - } -} diff --git a/src/orcapod/contexts/data/v1.json b/src/orcapod/contexts/data/v1.json new file mode 100644 index 000000000..1d4323668 --- /dev/null +++ b/src/orcapod/contexts/data/v1.json @@ -0,0 +1,242 @@ +{ + "context_key": "std:v1:default", + "version": "v1", + "description": "Data context of Orcapod v0.3: the hashing generation H1. Frozen when v0.3.0 is released; editable in place while that release is prepared.", + "released_in": null, + "previous": "std:v0.1:default", + "file_hasher": { + "_class": "orcapod.hashing.file_hashers.FileHasher", + "_config": { + "algorithm": "sha256" + } + }, + "directory_hasher": { + "_class": "orcapod.hashing.directory_hashers.BasicDirectoryHasher", + "_config": { + "file_hasher": { + "_ref": "file_hasher" + }, + "algorithm": "sha256" + } + }, + "type_converter": { + "_class": "orcapod.semantic_types.universal_converter.UniversalTypeConverter", + "_config": { + "logical_type_registry": { + "_ref": "logical_type_registry" + } + } + }, + "function_semantic_hasher": { + "_class": "orcapod.hashing.semantic_hashing.function_info_extractors.FunctionSignatureExtractor", + "_config": { + "include_module": true, + "include_defaults": true, + "type_converter": { + "_ref": "type_converter" + } + } + }, + "python_type_handler_registry": { + "_class": "orcapod.hashing.semantic_hashing.type_handler_registry.PythonTypeHandlerRegistry", + "_config": { + "handlers": [ + [ + { + "_type": "builtins.bytes" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.BytesHandler", + "_config": {} + } + ], + [ + { + "_type": "builtins.bytearray" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.BytesHandler", + "_config": {} + } + ], + [ + { + "_type": "types.FunctionType" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FunctionHandler", + "_config": { + "function_info_extractor": { + "_ref": "function_semantic_hasher" + } + } + } + ], + [ + { + "_type": "types.BuiltinFunctionType" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FunctionHandler", + "_config": { + "function_info_extractor": { + "_ref": "function_semantic_hasher" + } + } + } + ], + [ + { + "_type": "types.MethodType" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FunctionHandler", + "_config": { + "function_info_extractor": { + "_ref": "function_semantic_hasher" + } + } + } + ], + [ + { + "_type": "builtins.type" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.TypeObjectHandler", + "_config": { + "type_converter": { + "_ref": "type_converter" + } + } + } + ], + [ + { + "_type": "types.GenericAlias" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.GenericAliasHandler", + "_config": {} + } + ], + [ + { + "_type": "types.UnionType" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.UnionTypeHandler", + "_config": {} + } + ], + [ + { + "_type": "typing._GenericAlias" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.GenericAliasHandler", + "_config": {} + } + ], + [ + { + "_type": "typing._SpecialForm" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.SpecialFormHandler", + "_config": {} + } + ], + [ + { + "_type": "pyarrow.Table" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.ArrowTableHandler", + "_config": {} + } + ], + [ + { + "_type": "pyarrow.RecordBatch" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.ArrowTableHandler", + "_config": {} + } + ], + [ + { + "_type": "polars.Expr" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PolarsExprHandler", + "_config": {} + } + ], + [ + { + "_type": "orcapod.types.Schema" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.SchemaHandler", + "_config": { + "type_converter": { + "_ref": "type_converter" + } + } + } + ] + ] + } + }, + "semantic_hasher": { + "_class": "orcapod.hashing.semantic_hashing.semantic_hasher.SemanticAwarePythonHasher", + "_config": { + "hasher_id": "semantic_v0.1", + "type_handler_registry": { + "_ref": "python_type_handler_registry" + } + } + }, + "arrow_hasher": { + "_class": "orcapod.hashing.arrow_hashers.StarfixArrowHasher", + "_config": { + "hasher_id": "arrow_v0.2", + "type_converter": { + "_ref": "type_converter" + }, + "semantic_hasher": { + "_ref": "semantic_hasher" + } + } + }, + "definitions": { + "source_identity": "source_identity_v1", + "function_identity": "function_identity_v1", + "operator_identity": "operator_identity_v1", + "node_identity": "node_identity_v1", + "record_id": "record_id_v1", + "entry_id": "entry_id_v1", + "system_tag_fold": "system_tag_fold_v1", + "system_tag_naming": "system_tag_naming_v1", + "source_info_naming": "source_info_naming_v1", + "type_conversion": "type_conversion_v1" + }, + "libraries": { + "starfix": "0.5.0 (hash format 2, digest prefix 000002): Arrow table and schema hashes", + "pyarrow": "23.0.1: conversion of Python values to Arrow types, which are hashed", + "polars": "1.41.2: serialisation of filter expressions, which is hashed" + }, + "golden": { + "files": "tests/test_hashing/golden/data/*.json", + "algorithm": "sha256 over the files in name order; each contributes its name, a NUL byte, its bytes and a NUL byte", + "digest": null + }, + "changes": [ + { + "definition": "arrow_hasher", + "change": "starfix hash format 1 (starfix 0.4) to hash format 2 (starfix 0.5): every Arrow-derived hash value (ITL-910)" + } + ] +} diff --git a/src/orcapod/contexts/logical_types.json b/src/orcapod/contexts/logical_types.json new file mode 100644 index 000000000..271578e66 --- /dev/null +++ b/src/orcapod/contexts/logical_types.json @@ -0,0 +1,146 @@ +{ + "description": "Logical types registered in every data context, with the value handlers that hash their Python values. This file is not frozen: a logical type is versioned by its own identifier (Arrow extension name and metadata), not by the data context version, so adding one edits this file and no released data context manifest.", + "logical_type_registry": { + "_class": "orcapod.logical_types.registry.LogicalTypeRegistry", + "_config": { + "logical_types": [ + { + "_class": "orcapod.logical_types.builtin_logical_types.LogicalPath", + "_config": {} + }, + { + "_class": "orcapod.logical_types.builtin_logical_types.LogicalUPath", + "_config": {} + }, + { + "_class": "orcapod.logical_types.builtin_logical_types.LogicalUUID", + "_config": {} + }, + { + "_class": "orcapod.logical_types.file_type.LogicalFile", + "_config": {} + }, + { + "_class": "orcapod.logical_types.directory_type.LogicalDirectory", + "_config": {} + }, + { + "_class": "orcapod.logical_types.numpy_type.LogicalNumpyArray", + "_config": {} + }, + { + "_class": "orcapod.logical_types.pandas_type.LogicalPandasDataFrame", + "_config": {} + }, + { + "_class": "orcapod.logical_types.pandas_type.LogicalPandasSeries", + "_config": {} + } + ], + "factories": [ + { + "factory": { + "_class": "orcapod.logical_types.dataclass_logical_type_factory.DataclassLogicalTypeFactory", + "_config": {} + }, + "category": "orcapod.dataclass", + "python_bases": [ + { + "_type": "builtins.object" + } + ] + }, + { + "factory": { + "_class": "orcapod.logical_types.pydantic_logical_type_factory.PydanticLogicalTypeFactory", + "_config": {} + }, + "category": "orcapod.pydantic", + "python_bases": [ + { + "_type": "pydantic.BaseModel" + } + ] + }, + { + "factory": { + "_class": "orcapod.logical_types.list_logical_type_factory.ListLogicalTypeFactory", + "_config": {} + }, + "category": "list" + }, + { + "factory": { + "_class": "orcapod.logical_types.list_logical_type_factory.ListLogicalTypeFactory", + "_config": {} + }, + "category": "set" + } + ] + } + }, + "value_handlers": [ + [ + { + "_type": "uuid.UUID" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.UUIDHandler", + "_config": {} + } + ], + [ + { + "_type": "orcapod.logical_types.file_type.File" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.FileHandler", + "_config": { + "file_hasher": { + "_ref": "file_hasher" + } + } + } + ], + [ + { + "_type": "orcapod.logical_types.directory_type.Directory" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.DirectoryHandler", + "_config": { + "directory_hasher": { + "_ref": "directory_hasher" + } + } + } + ], + [ + { + "_type": "numpy.ndarray" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.NumpyArrayHandler", + "_config": {} + } + ], + [ + { + "_type": "pandas.DataFrame" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PandasDataFrameHandler", + "_config": {} + } + ], + [ + { + "_type": "pandas.Series" + }, + { + "_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PandasSeriesHandler", + "_config": {} + } + ] + ] +} diff --git a/src/orcapod/contexts/registry.py b/src/orcapod/contexts/registry.py index 6a0e9a76d..a3b498fb3 100644 --- a/src/orcapod/contexts/registry.py +++ b/src/orcapod/contexts/registry.py @@ -18,6 +18,10 @@ ContextValidationError, DataContext, ) +from orcapod.system_constants import ( + PRE_V1_DATA_CONTEXT_KEY, + PRE_V1_DATA_CONTEXT_VERSION, +) from orcapod.utils.object_spec import parse_objectspec logger = logging.getLogger(__name__) @@ -38,14 +42,15 @@ class JSONDataContextRegistry: - Validates JSON structure against schema - Lazy loading with caching - Robust error handling and logging - - Context string resolution (e.g., "v0.1", "std:v0.1:default") + - Context string resolution (e.g., "v1", "std:v1:default") """ def __init__( self, contexts_dir: Path | str | None = None, schema_file: Path | str | None = None, - default_version: str = "v0.1", + default_version: str = "v1", + logical_types_file: Path | str | None = None, ): """ Initialize the context registry. @@ -54,6 +59,11 @@ def __init__( contexts_dir: Directory containing JSON context files schema_file: JSON schema file for validation (optional) default_version: Default context version to use + logical_types_file: The logical type registration: the registered logical + types and the value handlers that hash their Python values. Defaults + to ``logical_types.json`` of the ``orcapod.contexts`` package. It is + kept outside the context files because a data context file is frozen + once released, while logical types are added independently. """ # Set up paths if contexts_dir is None: @@ -64,6 +74,10 @@ def __init__( schema_file = self.contexts_dir / "schemas" / "context_schema.json" self.schema_file = Path(schema_file) if schema_file else None + if logical_types_file is None: + logical_types_file = Path(__file__).parent / "logical_types.json" + self.logical_types_file = Path(logical_types_file) + # Internal state self._specs: dict[str, dict[str, Any]] = {} self._contexts: dict[str, DataContext] = {} @@ -123,7 +137,7 @@ def _load_all_specs(self) -> None: def _load_spec_file(self, json_file: Path) -> None: """Load and validate a single context specification file.""" - version = json_file.stem # e.g., "v0.1" from "v0.1.json" + version = json_file.stem # e.g., "v1" from "v1.json" # Load JSON with open(json_file, "r") as f: @@ -193,8 +207,8 @@ def resolve_context_string(self, context_string: str | None) -> str: Supports various formats: - None -> default version - - "v0.1" -> "v0.1" - - "std:v0.1:default" -> "v0.1" (extract version from full key) + - "v1" -> "v1" + - "std:v1:default" -> "v1" (extract version from full key) - "latest" -> highest version number """ if context_string is None: @@ -205,11 +219,11 @@ def resolve_context_string(self, context_string: str | None) -> str: versions = self.get_available_versions() return versions[-1] if versions else self._default_version - # If it looks like a simple version (v0.1), use directly + # If it looks like a simple version (v1), use directly if context_string.startswith("v") and ":" not in context_string: return context_string - # If it looks like a full context key (std:v0.1:default), extract version + # If it looks like a full context key (std:v1:default), extract version if ":" in context_string: parts = context_string.split(":") if len(parts) >= 2 and parts[1].startswith("v"): @@ -235,12 +249,34 @@ def get_context(self, context_string: str | None = None) -> DataContext: # Resolve to version version = self.resolve_context_string(context_string) + # A full key selects its manifest by the version part only, so the + # whole key must be the one that manifest defines. + if ( + context_string is not None + and ":" in context_string + and version in self._specs + and context_string != self._specs[version]["context_key"] + ): + raise ContextResolutionError( + f"Unknown data context {context_string!r}: version " + f"{version!r} is the data context " + f"{self._specs[version]['context_key']!r}." + ) + # Return cached context if available if version in self._contexts: logger.debug(f"Returning cached context for version {version}") return self._contexts[version] # Validate version exists + if version not in self._specs and version == PRE_V1_DATA_CONTEXT_VERSION: + raise ContextResolutionError( + f"The data context {PRE_V1_DATA_CONTEXT_KEY!r} is the one of " + "Orcapod before v0.3. This version of Orcapod does not ship its " + "manifest and cannot hash under it; the old manifest and its " + "code are to be kept in orcapod-sanctuary. Use an Orcapod " + "version before v0.3 for stores and jobs written under it." + ) if version not in self._specs: available = ", ".join(self.get_available_versions()) raise ContextResolutionError( @@ -266,7 +302,48 @@ def get_context(self, context_string: str | None = None) -> DataContext: ) # Top-level keys that are metadata, not instantiable components. - _METADATA_KEYS = frozenset({"context_key", "version", "description", "metadata"}) + _METADATA_KEYS = frozenset( + { + "context_key", + "version", + "description", + "metadata", + "released_in", + "previous", + "definitions", + "libraries", + "golden", + "changes", + } + ) + + # A context file that takes its logical types from the registration file refers + # to them by this name, as ``{"_ref": "logical_type_registry"}``. + _LOGICAL_TYPE_REGISTRY_REF = "logical_type_registry" + + def _uses_logical_type_registration(self, spec: dict[str, Any]) -> bool: + """Whether ``spec`` takes its logical types from the registration file. + + True when its type converter is configured with + ``{"_ref": "logical_type_registry"}`` and the spec defines no component of + that name itself. A spec that lists its logical types inline is built as is. + """ + if self._LOGICAL_TYPE_REGISTRY_REF in spec: + return False + converter_config = spec["type_converter"].get("_config", {}) + return converter_config.get("logical_type_registry") == { + "_ref": self._LOGICAL_TYPE_REGISTRY_REF + } + + def _load_logical_type_registration(self) -> dict[str, Any]: + """Read the logical type registration file. + + Returns: + Its ``logical_type_registry`` object spec and its ``value_handlers`` + list of ``[type, handler]`` object-spec pairs. + """ + with open(self.logical_types_file, "r") as f: + return json.load(f) def _create_context_from_spec(self, spec: dict[str, Any]) -> DataContext: """Create DataContext instance from validated specification. @@ -284,6 +361,14 @@ def _create_context_from_spec(self, spec: dict[str, Any]) -> DataContext: description = spec.get("description", "") ref_lut: dict[str, Any] = {} + # The registered logical types live outside the (frozen) context file. + registration: dict[str, Any] | None = None + if self._uses_logical_type_registration(spec): + registration = self._load_logical_type_registration() + ref_lut[self._LOGICAL_TYPE_REGISTRY_REF] = parse_objectspec( + registration["logical_type_registry"], ref_lut=ref_lut + ) + for key, value in spec.items(): if key in self._METADATA_KEYS: continue @@ -291,6 +376,17 @@ def _create_context_from_spec(self, spec: dict[str, Any]) -> DataContext: logger.debug(f"Creating {key} for context {version}") ref_lut[key] = parse_objectspec(value, ref_lut=ref_lut) + # The value handlers of the logical types join the core handlers of the + # context file; they may refer to its components (``file_hasher`` ...). + if registration is not None: + handler_registry = ref_lut["python_type_handler_registry"] + for pair_spec in registration["value_handlers"]: + # As for the handlers of the context file: a pair whose type or + # handler is optional and not installed is skipped. + pair = parse_objectspec(pair_spec, ref_lut=ref_lut) + if pair and len(pair) == 2 and all(x is not None for x in pair): + handler_registry.register(pair[0], pair[1]) + return DataContext( context_key=context_key, version=version, diff --git a/src/orcapod/contexts/released_manifests.json b/src/orcapod/contexts/released_manifests.json new file mode 100644 index 000000000..82c352d29 --- /dev/null +++ b/src/orcapod/contexts/released_manifests.json @@ -0,0 +1,3 @@ +{ + "v1": null +} diff --git a/src/orcapod/contexts/stored.py b/src/orcapod/contexts/stored.py new file mode 100644 index 000000000..474df5e18 --- /dev/null +++ b/src/orcapod/contexts/stored.py @@ -0,0 +1,284 @@ +"""The data context recorded on stored rows (ITL-944). + +Every row Orcapod stores carries the key of the data context that wrote it, in the +column ``__data_context`` (``constants.DATA_CONTEXT_COL``): function pipeline and +result tables, operator-cache tables, cached-source tables and side-effect invocation +logs. The key names one generation of hashing (``std:v1:default`` since Orcapod v0.3), +so two rows with the same key were hashed with exactly the same definitions. + +Rows written under the current data context are the current entries: every read and +lookup keeps those rows only. Rows of another data context stay in the table, +unchanged, until they are migrated. + +Tables written before v0.3 have no ``__data_context`` column and live at an older +table layout (another last path component). They are the generation +``std:v0.1:default``; Orcapod counts and reports them and does not read them as +results. + +To list the data contexts a stored table holds:: + + from orcapod.contexts import stored_data_contexts + + stored_data_contexts(store, record_path) # {"std:v1:default": 3} +""" + +from __future__ import annotations + +import logging +from typing import TYPE_CHECKING, NamedTuple, Protocol, cast + +from orcapod.errors import InputValidationError, SideEffectLogDataContextError +from orcapod.system_constants import PRE_V1_DATA_CONTEXT_KEY, constants +from orcapod.utils.lazy_module import LazyModule + +if TYPE_CHECKING: + import pyarrow as pa + import pyarrow.compute as pc +else: + pa = LazyModule("pyarrow") + pc = LazyModule("pyarrow.compute") + +logger = logging.getLogger(__name__) + + +def with_data_context(table: "pa.Table", context_key: str) -> "pa.Table": + """Return ``table`` with the ``__data_context`` column appended. + + Args: + table: Rows about to be stored. + context_key: Key of the data context that produced the rows. + + Returns: + A new table whose last column is ``__data_context``, a non-null + ``large_string`` holding ``context_key`` on every row. + """ + return table.append_column( + pa.field(constants.DATA_CONTEXT_COL, pa.large_string(), nullable=False), + pa.array([context_key] * table.num_rows, type=pa.large_string()), + ) + + +def require_same_data_context(input_context_key: str, own_context_key: str) -> None: + """Refuse an input that was hashed under another data context than the writer's. + + A stored row names one data context, the one whose hashers produced its keys. An + input of another data context would give a row whose keys mix two generations of + hashing, so the writer refuses it before anything is written. What should + eventually happen instead (the node's data context is meant to win) is tracked + in ITL-951. + + Args: + input_context_key: Data context key of the input (a row or a stream). + own_context_key: Data context key of the node, cache or source that writes. + + Raises: + InputValidationError: If the two keys differ. + """ + if input_context_key != own_context_key: + raise InputValidationError( + f"The input was produced under the data context {input_context_key!r}, " + f"but this node stores its rows under {own_context_key!r}. A stored row " + "carries one data context; convert the input to the node's data context " + "first, or run the node under the input's data context." + ) + + +def keep_rows_of_data_context(table: "pa.Table", context_key: str) -> "pa.Table": + """Drop the rows of ``table`` written under another data context. + + Every read of a whole stored table goes through this function, so rows of another + data context are never returned as results. + + Args: + table: Rows read from a stored table at the current layout. + context_key: Key of the current data context. + + Returns: + ``table`` itself when no row is dropped, otherwise ``table`` without the rows + whose ``__data_context`` is not ``context_key``. + + Raises: + KeyError: If ``table`` has no ``__data_context`` column. + """ + mine = cast( + "pa.BooleanArray", + pc.equal( + table.column(constants.DATA_CONTEXT_COL), + pa.scalar(context_key, pa.large_string()), + ).fill_null(False), + ) + if pc.all(mine).as_py() is not False: + return table + # The single place where stored rows of another data context are left out of a read. + return table.filter(mine) + + +def report_rows_of_other_data_contexts( + record_path: tuple[str, ...], skipped: int +) -> None: + """Warn that a read left out stored rows written under another data context. + + Args: + record_path: Path of the stored table from the root of the store. + skipped: Number of rows of the table left out of the read. + """ + logger.warning( + "Skipped %d %s of stored table %r: written under another data context than " + "the current one. They are not used and are left unchanged; they can be " + "migrated later.", + skipped, + "row" if skipped == 1 else "rows", + "/".join(record_path), + ) + + +def report_old_layout_rows(record_path: tuple[str, ...], rows: int) -> None: + """Warn that a table written at an older table layout was found and is not used. + + Args: + record_path: Path of the old-layout table from the root of the store. + rows: Number of rows the table holds. + """ + logger.warning( + "Found %d %s in stored table %r written by an earlier version of Orcapod " + "(older table layout, data context %s). They are not used and are left " + "unchanged; they can be migrated later.", + rows, + "row" if rows == 1 else "rows", + "/".join(record_path), + PRE_V1_DATA_CONTEXT_KEY, + ) + + +class _RecordStore(Protocol): + """The part of ``ArrowDatabaseProtocol`` this module needs (no import cycle).""" + + @property + def base_path(self) -> tuple[str, ...]: ... + + def table_exists(self, record_path: tuple[str, ...]) -> bool: ... + + def get_all_records( + self, record_path: tuple[str, ...], record_id_column: str | None = None + ) -> "pa.Table | None": ... + + +class StaleTable(NamedTuple): + """Stored rows a node will not use: an old-layout table, or rows of another data context. + + Attributes: + record_path: Path of the table from the root of the store. + data_context: Key of the data context that wrote the rows. + rows: Number of such rows in the table. + old_layout: True for a table at an older layout, False for rows of another + data context in a table at the current layout. + """ + + record_path: tuple[str, ...] + data_context: str + rows: int + old_layout: bool + + +def find_stale_tables( + database: _RecordStore, + old_paths: tuple[tuple[str, ...], ...], + current: tuple[tuple[str, ...], str] | None = None, +) -> list[StaleTable]: + """Find the stored rows of other generations that belong to one table of a node. + + The tables are only read: nothing is written to them. + + Args: + database: The store the paths belong to. + old_paths: Paths at which an earlier version of Orcapod stored this table. + current: The table's current path and the current data context key, to also + look for rows of another data context in the current table (this reads + the whole table). ``None`` looks at the old layouts only. + + Returns: + One entry per old-layout table that exists, then one per other data context + found in the current table (in key order). Empty when there is none. + """ + found: list[StaleTable] = [] + for path in old_paths: + if not database.table_exists(path): + continue + records = database.get_all_records(path) + rows = 0 if records is None else records.num_rows + found.append( + StaleTable(database.base_path + path, PRE_V1_DATA_CONTEXT_KEY, rows, True) + ) + if current is not None: + current_path, current_context = current + for key, rows in sorted(stored_data_contexts(database, current_path).items()): + if key != current_context: + found.append( + StaleTable(database.base_path + current_path, key, rows, False) + ) + return found + + +def account_for_stale_tables( + found: list[StaleTable], *, delivers_side_effects: bool, current_context: str +) -> None: + """Decide what a node does about stored rows of other generations it will not use. + + This is the one place where that is decided, for every kind of node: + + - A node whose results can be computed again (function, operator and source + nodes) reports each table with its number of rows and moves on. + - A node whose run is an external action (a side-effect pod, or a function pod + that takes the invocation context) must not act again for inputs it already + handled, and those completions cannot be matched under the current data + context. It fails before doing anything. + + Args: + found: What ``find_stale_tables`` found for the node's tables. + delivers_side_effects: Whether running the node is an external action. + current_context: Key of the node's data context. + + Raises: + SideEffectLogDataContextError: If ``delivers_side_effects`` and ``found`` is + not empty. The caller adds the node label. + """ + if not found: + return + if delivers_side_effects: + raise SideEffectLogDataContextError( + tuple(table.record_path for table in found), + found[0].data_context, + current_context, + ) + for table in found: + if table.old_layout: + report_old_layout_rows(table.record_path, table.rows) + else: + report_rows_of_other_data_contexts(table.record_path, table.rows) + + +def stored_data_contexts( + database: _RecordStore, record_path: tuple[str, ...] +) -> dict[str, int]: + """Count the rows of a stored table by the data context that wrote them. + + Reads the whole table. A table without a ``__data_context`` column was written + before Orcapod v0.3; all its rows are counted under ``std:v0.1:default``. + + Args: + database: The store, positioned so that ``record_path`` names the table. + record_path: Path of the stored table, including its layout component. + + Returns: + A mapping from data context key to number of rows, for example + ``{"std:v1:default": 3}``. A missing or empty table gives ``{}``. + """ + table = database.get_all_records(record_path) + if table is None or table.num_rows == 0: + return {} + if constants.DATA_CONTEXT_COL not in table.column_names: + return {PRE_V1_DATA_CONTEXT_KEY: table.num_rows} + per_value = pc.value_counts(table.column(constants.DATA_CONTEXT_COL)) + keys = cast("list[str]", per_value.field(0).to_pylist()) + counts = cast("list[int]", per_value.field(1).to_pylist()) + return dict(zip(keys, counts)) diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index 757b43692..6837b5011 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -8,6 +8,7 @@ from orcapod.core.datagrams import Datagram from orcapod.core.function_pod import WrappedFunctionPod +from orcapod.contexts.stored import StaleTable from orcapod.core.result_cache import ResultCache from orcapod.hooks import InvocationStatus from orcapod.protocols.core_protocols import ( @@ -54,6 +55,8 @@ def __init__( result_database=result_database, record_path=self.uri, # no prefix; db is pre-scoped auto_flush=auto_flush, + # A pod that takes the invocation context performs an external action. + delivers_side_effects=function_pod.ctx_arg_name is not None, ) @property @@ -66,15 +69,15 @@ def record_path(self) -> tuple[str, ...]: """Return the path to the cached records in the result store (versioned).""" return self._cache.record_path - def set_ignore_schema(self, ignore_schema: tuple[str, ...] | None) -> None: - """Propagate ``ignore_schema`` setting to the underlying ``ResultCache``. + def stale_result_tables(self) -> list[StaleTable]: + """Find stored results of other generations that this pod will not use. - Args: - ignore_schema: Tuple of schema version strings to tolerate (e.g. - ``("v0",)``), or ``None`` to use the default (raise on any - old schema). + Returns: + Result tables at an older layout, and, for a pod that takes the + invocation context, rows of another data context in the current + result table. Empty when there is nothing. """ - self._cache.set_ignore_schema(ignore_schema) + return self._cache.stale_tables(self.data_context_key) def lookup_cached_data(self, data: DataProtocol) -> DataProtocol | None: """Look up a cached result for ``data`` without triggering computation. @@ -96,7 +99,7 @@ def lookup_cached_data(self, data: DataProtocol) -> DataProtocol | None: The cached output ``Data`` (with ``RESULT_COMPUTED_FLAG=False`` appended), or ``None`` if no cached result exists for this input. """ - cached = self._cache.lookup(data) + cached = self._cache.lookup(data, data_context_key=self.data_context_key) if cached is None: return None return cached.with_meta_columns(**{self.RESULT_COMPUTED_FLAG: False}) @@ -128,7 +131,7 @@ def process_data( A ``(tag, output_data)`` tuple; output_data is ``None`` if the inner function filters the data out. """ - cached = self._cache.lookup(data) + cached = self._cache.lookup(data, data_context_key=self.data_context_key) if cached is not None: module_logger.info("Pod-level cache hit") cached = cached.with_meta_columns(**{self.RESULT_COMPUTED_FLAG: False}) @@ -149,7 +152,9 @@ def process_data( python_schema=pf.get_execution_data_schema(), data_context=pf.data_context, ) - self._cache.store(data, output, var_dg, exec_dg) + self._cache.store( + data, output, var_dg, exec_dg, data_context_key=self.data_context_key + ) output = output.with_meta_columns(**{self.RESULT_COMPUTED_FLAG: True}) return tag, output @@ -179,7 +184,7 @@ async def async_process_data( A ``(tag, output_data)`` tuple; output_data is ``None`` if the inner function filters the data out. """ - cached = self._cache.lookup(data) + cached = self._cache.lookup(data, data_context_key=self.data_context_key) if cached is not None: module_logger.info("Pod-level cache hit") cached = cached.with_meta_columns(**{self.RESULT_COMPUTED_FLAG: False}) @@ -200,7 +205,9 @@ async def async_process_data( python_schema=pf.get_execution_data_schema(), data_context=pf.data_context, ) - self._cache.store(data, output, var_dg, exec_dg) + self._cache.store( + data, output, var_dg, exec_dg, data_context_key=self.data_context_key + ) output = output.with_meta_columns(**{self.RESULT_COMPUTED_FLAG: True}) return tag, output @@ -361,7 +368,8 @@ def get_all_cached_outputs( ) -> "pa.Table | None": """Return all cached records from the result store for this pod.""" return self._cache.get_all_records( - include_system_columns=include_system_columns + self.data_context_key, + include_system_columns=include_system_columns, ) def process( diff --git a/src/orcapod/core/data_function.py b/src/orcapod/core/data_function.py index c84986d66..e99977e90 100644 --- a/src/orcapod/core/data_function.py +++ b/src/orcapod/core/data_function.py @@ -529,9 +529,14 @@ def canonical_function_name(self) -> str: return self._function_name def get_function_variation_data(self) -> dict[str, Any]: - """Raw data defining function variation - system computes hash""" + """Raw data defining function variation, stored with every result row. + + ``function_version`` is the full version string the function was declared + with (``"v3.7"``); only its major part is in the table path. + """ return { "function_name": self._function_name, + "function_version": self._version, "function_signature_hash": self._function_signature_hash, "function_content_hash": self._function_content_hash, "git_hash": self._git_hash, @@ -542,6 +547,7 @@ def get_function_variation_data_schema(self) -> Schema: return Schema( { "function_name": str, + "function_version": str, "function_signature_hash": bytes, "function_content_hash": bytes, "git_hash": str, @@ -1054,7 +1060,9 @@ def call( output_data = None if not skip_cache_lookup: module_logger.info("Checking for cache...") - output_data = self._cache.lookup(data) + output_data = self._cache.lookup( + data, data_context_key=self.data_context_key + ) if output_data is not None: module_logger.info(f"Cache hit for {data}!") return output_data @@ -1062,7 +1070,13 @@ def call( if output_data is not None: if not skip_cache_insert: var_dg, exec_dg = self._build_metadata_datagrams() - self._cache.store(data, output_data, var_dg, exec_dg) + self._cache.store( + data, + output_data, + var_dg, + exec_dg, + data_context_key=self.data_context_key, + ) output_data = output_data.with_meta_columns( **{self.RESULT_COMPUTED_FLAG: True} ) @@ -1081,7 +1095,9 @@ async def async_call( output_data = None if not skip_cache_lookup: module_logger.info("Checking for cache...") - output_data = self._cache.lookup(data) + output_data = self._cache.lookup( + data, data_context_key=self.data_context_key + ) if output_data is not None: module_logger.info(f"Cache hit for {data}!") return output_data @@ -1091,7 +1107,13 @@ async def async_call( if output_data is not None: if not skip_cache_insert: var_dg, exec_dg = self._build_metadata_datagrams() - self._cache.store(data, output_data, var_dg, exec_dg) + self._cache.store( + data, + output_data, + var_dg, + exec_dg, + data_context_key=self.data_context_key, + ) output_data = output_data.with_meta_columns( **{self.RESULT_COMPUTED_FLAG: True} ) @@ -1107,7 +1129,7 @@ def get_cached_output_for_data( Returns: The cached output data, or ``None`` if no entry was found. """ - return self._cache.lookup(input_data) + return self._cache.lookup(input_data, data_context_key=self.data_context_key) def record_data( self, @@ -1123,6 +1145,7 @@ def record_data( var_dg, exec_dg, skip_duplicates=skip_duplicates, + data_context_key=self.data_context_key, ) return output_data @@ -1139,5 +1162,6 @@ def get_all_cached_outputs( A PyArrow table of cached results, or ``None`` if empty. """ return self._cache.get_all_records( - include_system_columns=include_system_columns + self.data_context_key, + include_system_columns=include_system_columns, ) diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 3454c8856..49615d8df 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -33,12 +33,19 @@ from orcapod.core.datagrams.tag_data import EmptyData, Tag from orcapod.errors import ( EphemeralResultMissingError, + InputValidationError, PipelineJobRequiredError, - SchemaVersionError, StoreError, StoreReadError, StoreWriteError, ) +from orcapod.contexts.stored import ( + keep_rows_of_data_context, + require_same_data_context, + account_for_stale_tables, + find_stale_tables, + report_rows_of_other_data_contexts, +) from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.core_protocols import ( FunctionPodProtocol, @@ -56,6 +63,7 @@ ) from orcapod.system_constants import ( constants, + OLD_PIPELINE_DB_SCHEMA_VERSIONS, PIPELINE_DB_SCHEMA_VERSION, RESULT_DB_SCHEMA_VERSION, ) @@ -123,11 +131,6 @@ def _build_record_id_preimage( # record_and_forward() before downstream emission. _TAG_NODE_INPUT_REF = "_tag_node_input_ref" -# Module-level set of versioned pdb paths that have been checked for legacy schema. -# Keyed by the full versioned path (node_identity_path + (PIPELINE_DB_SCHEMA_VERSION,)). -# Populated on first access; prevents repeated table_exists checks across instances. -_checked_pdb_paths: set[tuple[str, ...]] = set() - def _executor_supports_concurrent( data_function: DataFunctionProtocol, @@ -600,7 +603,7 @@ def from_descriptor( node._label = descriptor.get("label") # From DataContextMixin - node._data_context = contexts.resolve_context( + node._data_context = contexts.resolve_saved_context( descriptor.get("data_context_key") ) from orcapod.config import DEFAULT_CONFIG, OrcapodConfig @@ -781,6 +784,12 @@ class FunctionJobNode(FunctionNodeBase): execution. """ + # Set once a read has reported the stored rows it skipped (rows of another data + # context); a class default so nodes rebuilt from a saved descriptor start unset. + _skipped_rows_reported: bool = False + # Set once stored rows of other generations have been accounted for. + _stale_tables_checked: bool = False + def __init__( self, function_pod: FunctionPodProtocol, @@ -840,10 +849,6 @@ def node_config(self) -> NodeConfig: @node_config.setter def node_config(self, value: NodeConfig) -> None: self._node_config = value - if self._cached_function_pod is not None: - self._cached_function_pod.set_ignore_schema(value.ignore_schema) - if self._ephemeral_cached_pod is not None: - self._ephemeral_cached_pod.set_ignore_schema(value.ignore_schema) @property def _versioned_pipeline_path(self) -> tuple[str, ...]: @@ -854,52 +859,70 @@ def _versioned_pipeline_path(self) -> tuple[str, ...]: """ return self.node_identity_path + (PIPELINE_DB_SCHEMA_VERSION,) - def _ensure_pdb_schema(self) -> None: - """Check once per versioned path that no legacy v0 schema table is present. + @property + def _stored_data_context_key(self) -> str: + """Key of the data context this node writes on its rows and reads back. - Detection flow: - 1. If the v1 versioned path already exists → already migrated, skip check. - 2. If v1 absent → check whether the unversioned (v0) path exists. - 3. If v0 exists AND the version is not in ``ignore_schema`` → raise. - 4. If v0 exists AND version is ignored → log a warning and proceed. - 5. If v0 also absent → first use, proceed normally. + A node loaded from a saved job has no function pod; its context then comes + from the descriptor. + """ + context = ( + self._data_context if self._function_pod is None else self.data_context + ) + return context.context_key - Results are cached per versioned path (module-level set) so the check - happens at most once per process per versioned path. + @property + def _delivers_side_effects(self) -> bool: + """Whether running this node is an external action. + + True for a function pod that takes the invocation context (``ctx_arg``): + its call is a side effect, and its completions are its function tables. + """ + return ( + self._function_pod is not None + and self._function_pod.ctx_arg_name is not None + ) + + def _account_for_stale_tables(self) -> None: + """Report or refuse, once per node object, stored rows of other generations. + + A pipeline table written by an earlier version of Orcapod (``pdb_v1``) holds + no ``__data_context`` column: its rows are not read as results and are left + unchanged, and their number is reported (ITL-944). A node that delivers side + effects fails instead, and for it the result tables and rows of another data + context in the current tables count too: none of them can be matched with + current inputs, so every side effect would be delivered again. Raises: - SchemaVersionError: When a v0 table is found and ``ignore_schema`` - does not include ``"v0"``. + SideEffectLogDataContextError: If this node delivers side effects and + such rows exist; nothing has been delivered or written then. """ - global _checked_pdb_paths - versioned_path = self._versioned_pipeline_path - if versioned_path in _checked_pdb_paths: - return - if self._pipeline_database is None: + if self._stale_tables_checked or self._pipeline_database is None: return - # v1 path exists → schema is current, mark as checked - if self._pipeline_database.table_exists(versioned_path): - _checked_pdb_paths.add(versioned_path) - return - # v1 absent — check for legacy v0 table at the unversioned path - if self._pipeline_database.table_exists(self.node_identity_path): - ignore = self._node_config.ignore_schema or () - if "v0" not in ignore: - node_path_str = "/".join(self.node_identity_path) - raise SchemaVersionError( - f"Pipeline DB at {self.node_identity_path!r} contains a legacy v0 schema table.\n" - "Run migration first:\n" - f" orcapod migrate pipeline-db {node_path_str}\n" - "To suppress this error and recompute all results instead, set:\n" - ' node.node_config = NodeConfig(ignore_schema=("v0",))' - ) - logger.warning( - "Pipeline DB at %r has a legacy v0 schema table; " - "proceeding without migration because ignore_schema includes 'v0'. " - "All results will be recomputed from scratch.", - self.node_identity_path, + context_key = self._stored_data_context_key + delivers = self._delivers_side_effects + found = find_stale_tables( + self._pipeline_database, + # Orcapod v0.1 stored the table at the path without a layout component. + tuple( + self.node_identity_path + (layout,) + for layout in OLD_PIPELINE_DB_SCHEMA_VERSIONS + ) + + (self.node_identity_path,), + current=(self._versioned_pipeline_path, context_key) if delivers else None, + ) + if delivers: + for cached_pod in (self._cached_function_pod, self._ephemeral_cached_pod): + if cached_pod is not None: + found.extend(cached_pod.stale_result_tables()) + try: + account_for_stale_tables( + found, delivers_side_effects=delivers, current_context=context_key ) - _checked_pdb_paths.add(versioned_path) + except StoreError as exc: + exc.add_context(node_label=self.label) + raise + self._stale_tables_checked = True # ------------------------------------------------------------------ # attach_databases @@ -941,6 +964,9 @@ def attach_databases( ``set_ephemeral_store(ephemeral_database)`` after ``attach_databases()``. """ + # Another store has its own tables of other generations to account for. + self._stale_tables_checked = False + self._skipped_rows_reported = False if pipeline_database is not None: if result_database is None: # Default result database is pipeline_database scoped to "_result" @@ -1037,7 +1063,7 @@ def from_descriptor( node._label = descriptor.get("label") # From DataContextMixin - node._data_context = contexts.resolve_context( + node._data_context = contexts.resolve_saved_context( descriptor.get("data_context_key") ) config_dict = descriptor.get("config") @@ -1263,9 +1289,18 @@ def execute( Returns: Materialized list of (tag, output_data) pairs, excluding ``None`` outputs and failed data. + + Raises: + InputValidationError: If the input stream is of another data context + than this node; raised before any row is processed, whatever the + error policy. """ from orcapod.pipeline.observer import NoOpObserver + # A whole-input condition, not a row failure: the error policy does not apply. + require_same_data_context( + input_stream.data_context_key, self._stored_data_context_key + ) node_label = self.label node_hash = self.content_hash().to_string() @@ -1376,6 +1411,9 @@ def _process_data_internal( Returns: A ``(tag, output_data)`` 2-tuple. """ + # A node that delivers side effects must not run on completions it cannot match. + self._account_for_stale_tables() + require_same_data_context(data.data_context_key, self._stored_data_context_key) # Guard: EmptyData inputs skip all computation and go straight to cache lookup. # EmptyData.content_hash() returns the cached_content_hash, so # result_cache.lookup(data) works transparently — it finds the downstream @@ -1519,6 +1557,9 @@ async def _async_process_data_internal( Returns: A ``(tag, output_data)`` 2-tuple. """ + # A node that delivers side effects must not run on completions it cannot match. + self._account_for_stale_tables() + require_same_data_context(data.data_context_key, self._stored_data_context_key) # Guard: same logic as sync _process_data_internal — see inline comment there. if isinstance(data, EmptyData): if self._cached_function_pod is not None: @@ -1725,7 +1766,7 @@ def add_pipeline_record( result keyed by the same hash (= the downstream's ``INPUT_DATA_HASH_COL``). """ pipeline_database = self._require_pipeline_database() - self._ensure_pdb_schema() + self._account_for_stale_tables() base_entry_id = self.compute_base_entry_id(tag, input_data) # Determine the next recomputation index by querying all existing rows @@ -1733,7 +1774,10 @@ def add_pipeline_record( # asyncio event loop this read-then-write sequence is uninterrupted. existing = pipeline_database.get_records_with_column_value( self._versioned_pipeline_path, - {_PIPELINE_BASE_ENTRY_ID_COL: base_entry_id}, + { + _PIPELINE_BASE_ENTRY_ID_COL: base_entry_id, + constants.DATA_CONTEXT_COL: self._stored_data_context_key, + }, ) if existing is None or existing.num_rows == 0: new_index = 0 @@ -1773,8 +1817,8 @@ def add_pipeline_record( ], type=pa.large_binary(), ), - f"{constants.META_PREFIX}input_data{constants.CONTEXT_KEY}": pa.array( - [input_data.data_context_key], type=pa.large_string() + constants.DATA_CONTEXT_COL: pa.array( + [self._stored_data_context_key], type=pa.large_string() ), f"{constants.META_PREFIX}computed": pa.array( [computed], type=pa.bool_() @@ -1946,7 +1990,7 @@ def _fetch_joined_records( if self._cached_function_pod is None or self._pipeline_database is None: return None - self._ensure_pdb_schema() + self._account_for_stale_tables() taginfo = self._pipeline_database.get_all_records( self._versioned_pipeline_path, record_id_column=_PIPELINE_ENTRY_ID_COL, @@ -1955,6 +1999,19 @@ def _fetch_joined_records( if taginfo is None: return None + # Rows written under another data context are not this node's current + # entries: they stay in the table untouched (ITL-944). Result rows are + # reached only through the pipeline rows kept here. + context_key = self._stored_data_context_key + stored_rows = taginfo.num_rows + taginfo = keep_rows_of_data_context(taginfo, context_key) + if taginfo.num_rows < stored_rows and not self._skipped_rows_reported: + self._skipped_rows_reported = True + report_rows_of_other_data_contexts( + self._pipeline_database.base_path + self._versioned_pipeline_path, + stored_rows - taginfo.num_rows, + ) + taginfo_columns = tuple(taginfo.column_names) taginfo_schema = taginfo.schema @@ -1991,6 +2048,9 @@ def _fetch_joined_records( self._cached_function_pod.record_path, record_id_column=constants.DATA_RECORD_ID, ) + if results is not None: + # The data context is taken from the pipeline rows kept above. + results = results.drop([constants.DATA_CONTEXT_COL]) if results is None: # Tag table has persistent entries but result DB is empty — data loss logger.warning( @@ -2034,6 +2094,7 @@ def _fetch_joined_records( ) ) if eph_results is not None: + eph_results = eph_results.drop([constants.DATA_CONTEXT_COL]) if results_schema is None: results_schema = eph_results.schema ephemeral_df = ephemeral_taginfo_df.join( @@ -2115,6 +2176,11 @@ def _fetch_joined_records( joined = arrow_utils.restore_schema_nullability( joined, taginfo_schema, results_schema ) + # In memory, each row names its data context in the datagram context column. + joined = joined.append_column( + pa.field(constants.CONTEXT_KEY, pa.large_string(), nullable=False), + pa.array([context_key] * joined.num_rows, type=pa.large_string()), + ) return _JoinedRecords( table=joined, taginfo_columns=taginfo_columns, @@ -2369,6 +2435,15 @@ async def async_execute( from orcapod.pipeline.observer import NoOpObserver + # A whole-input condition, not a row failure: the error policy does not apply. + if self._input_stream is not None: + try: + require_same_data_context( + self._input_stream.data_context_key, self._stored_data_context_key + ) + except InputValidationError: + await output.close() + raise node_label = self.label node_hash = self.content_hash().to_string() diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 0e6c4c188..a23f962c4 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -20,7 +20,7 @@ import logging from abc import abstractmethod from collections.abc import Collection, Iterator, Sequence -from typing import TYPE_CHECKING, Any, Literal +from typing import TYPE_CHECKING, Any, Literal, cast from orcapod import contexts from orcapod.channels import Channel, ReadableChannel, WritableChannel @@ -34,6 +34,14 @@ StoreReadError, StoreWriteError, ) +from orcapod.contexts.stored import ( + keep_rows_of_data_context, + require_same_data_context, + account_for_stale_tables, + find_stale_tables, + report_rows_of_other_data_contexts, + with_data_context, +) from orcapod.protocols.core_protocols import ( DataProtocol, StreamProtocol, @@ -42,7 +50,7 @@ ) from orcapod.protocols.core_protocols.operator_pod import OperatorPodProtocol from orcapod.protocols.database_protocols import ArrowDatabaseProtocol -from orcapod.system_constants import constants +from orcapod.system_constants import OPERATOR_DB_SCHEMA_VERSION, constants from orcapod.types import CacheMode, ColumnConfig, ContentHash, Schema from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -429,7 +437,7 @@ def from_descriptor( # From DataContextMixin from orcapod.config import DEFAULT_CONFIG, OrcapodConfig - node._data_context = contexts.resolve_context( + node._data_context = contexts.resolve_saved_context( descriptor.get("data_context_key") ) config_dict = descriptor.get("config") @@ -536,6 +544,37 @@ class OperatorJobNode(OperatorNodeBase): - **REPLAY**: skip computation, flow cached results downstream. """ + # Set once a read has reported the stored rows it skipped (rows of another data + # context); a class default so nodes rebuilt from a saved descriptor start unset. + _skipped_rows_reported: bool = False + # Set once a table found at the older layout has been accounted for. + _stale_tables_checked: bool = False + + @property + def _stored_table_path(self) -> tuple[str, ...]: + """Path of this node's stored table: the identity path plus the layout version. + + Every read and write of the operator-cache table uses this path. The identity + path itself is where Orcapod up to v0.2 stored the table (the old layout). + """ + return self.node_identity_path + (OPERATOR_DB_SCHEMA_VERSION,) + + def _account_for_stale_tables(self) -> None: + """Report, once per node object, a table found at the older layout. + + A table written by an earlier version of Orcapod holds no ``__data_context`` + column. Its rows are not read as results and are left unchanged; their number + is reported (ITL-944). + """ + if self._stale_tables_checked or self._pipeline_database is None: + return + self._stale_tables_checked = True + account_for_stale_tables( + find_stale_tables(self._pipeline_database, (self.node_identity_path,)), + delivers_side_effects=False, + current_context=self.data_context_key, + ) + def __init__( self, operator: OperatorPodProtocol, @@ -619,6 +658,9 @@ def attach_databases( pipeline_database: Database for pipeline records. cache_mode: Caching behaviour (OFF, LOG, or REPLAY). """ + # Another store has its own tables of other generations to account for. + self._stale_tables_checked = False + self._skipped_rows_reported = False self._pipeline_database = pipeline_database self._cache_mode = cache_mode @@ -709,7 +751,7 @@ def from_descriptor( # From DataContextMixin from orcapod.config import DEFAULT_CONFIG, OrcapodConfig - node._data_context = contexts.resolve_context( + node._data_context = contexts.resolve_saved_context( descriptor.get("data_context_key") ) config_dict = descriptor.get("config") @@ -808,8 +850,20 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: f"OperatorJobNode {self.label!r} has no pipeline database attached." ) output_table = stream.as_table( - columns={"source": True, "system_tags": True}, + columns={"source": True, "system_tags": True, "context": True}, + ) + # The inputs and the rows name their data context; all must be this node's. + for input_stream in self._input_streams: + require_same_data_context( + input_stream.data_context_key, self.data_context_key + ) + row_context_keys = cast( + "list[str]", + pc.unique(output_table.column(constants.CONTEXT_KEY)).to_pylist(), ) + for row_context_key in row_context_keys: + require_same_data_context(row_context_key, self.data_context_key) + output_table = output_table.drop([constants.CONTEXT_KEY]) # Always append the node content hash column first so that it is included # in the per-row HASH_COLUMN_NAME computation below. This makes record IDs @@ -848,10 +902,13 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: # the hash), so rows from different runs with identical output will have # distinct record IDs and both be stored. skip_duplicates=True still # deduplicates exact re-runs of the same node within a single run. + # The data context column is appended after the record hashes are computed: + # it names the generation of hashing that produced them and is not hashed. + self._account_for_stale_tables() try: pipeline_database.add_records( - self.node_identity_path, - output_table, + self._stored_table_path, + with_data_context(output_table, self.data_context_key), record_id_column=self.HASH_COLUMN_NAME, skip_duplicates=True, ) @@ -860,13 +917,43 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: raise except Exception as exc: raise StoreWriteError( - self.node_identity_path, node_label=self.label + self._stored_table_path, node_label=self.label ) from exc self._cached_output_table = output_table.drop( [self.HASH_COLUMN_NAME, constants.NODE_CONTENT_HASH_COL] ) + def _read_stored_records(self) -> "pa.Table | None": + """Read this node's stored rows written under the current data context. + + Every read of the whole stored table goes through here. Rows are narrowed to + the current data context (``__data_context``) and then to this node's content + hash (``_filter_by_content_hash``). Rows of another data context stay in the + table untouched, and a table at the older layout is not read (ITL-944). + + Returns: + The stored rows, with ``HASH_COLUMN_NAME`` and ``NODE_CONTENT_HASH_COL`` + still present and without the ``__data_context`` column, or ``None`` + when the table does not exist. + """ + assert self._pipeline_database is not None + self._account_for_stale_tables() + records = self._pipeline_database.get_all_records( + self._stored_table_path, record_id_column=self.HASH_COLUMN_NAME + ) + if records is None: + return None + stored_rows = records.num_rows + records = keep_rows_of_data_context(records, self.data_context_key) + if records.num_rows < stored_rows and not self._skipped_rows_reported: + self._skipped_rows_reported = True + report_rows_of_other_data_contexts( + self._pipeline_database.base_path + self._stored_table_path, + stored_rows - records.num_rows, + ) + return self._filter_by_content_hash(records.drop([constants.DATA_CONTEXT_COL])) + def _make_empty_table(self) -> "pa.Table": """Build a zero-row PyArrow table matching this node's full output schema. @@ -904,7 +991,7 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": return None if self._cache_mode != CacheMode.REPLAY: return None - records = self._pipeline_database.get_all_records(self.node_identity_path) + records = self._read_stored_records() if records is None: if self._operator is None: # Read-only (deserialized) node with no operator: cannot derive @@ -913,7 +1000,7 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": return None records_table = self._make_empty_table() else: - records_table = self._filter_by_content_hash(records) + records_table = records # Drop internal columns that must not surface to stream consumers. cols_to_drop = [ c @@ -923,7 +1010,10 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": if cols_to_drop: records_table = records_table.drop(cols_to_drop) tag_keys = self.keys()[0] - return ArrowTableStream(records_table, tag_columns=tag_keys) + # Stored rows are this node's: give them its data context, not the default. + return ArrowTableStream( + records_table, tag_columns=tag_keys, data_context=self.data_context + ) def get_cached_output(self) -> StreamProtocol | None: """Return cached output stream in REPLAY mode, else None. @@ -1016,11 +1106,10 @@ def _replay_from_cache(self) -> None: If no cached records exist yet, produces an empty stream with the correct schema (zero rows, correct columns). """ - records = self._pipeline_database.get_all_records(self.node_identity_path) + records = self._read_stored_records() if records is None: records = self._make_empty_table() else: - records = self._filter_by_content_hash(records) # Drop internal columns that must not surface to stream consumers. cols_to_drop = [ c @@ -1031,7 +1120,9 @@ def _replay_from_cache(self) -> None: records = records.drop(cols_to_drop) tag_keys = self.keys()[0] - self._cached_output_stream = ArrowTableStream(records, tag_columns=tag_keys) + self._cached_output_stream = ArrowTableStream( + records, tag_columns=tag_keys, data_context=self.data_context + ) self._update_modified_time() def run(self) -> None: @@ -1123,12 +1214,10 @@ def get_all_records( if self._pipeline_database is None: return None - results = self._pipeline_database.get_all_records(self.node_identity_path) + results = self._read_stored_records() if results is None: return None - results = self._filter_by_content_hash(results) - column_config = ColumnConfig.handle_config(columns, all_info=all_info) # Always drop internal row-level columns. diff --git a/src/orcapod/core/result_cache.py b/src/orcapod/core/result_cache.py index 89bd0311e..8387284cd 100644 --- a/src/orcapod/core/result_cache.py +++ b/src/orcapod/core/result_cache.py @@ -12,10 +12,23 @@ from datetime import datetime, timezone from typing import TYPE_CHECKING, Any -from orcapod.errors import SchemaVersionError, StoreReadError, StoreWriteError +from orcapod.errors import StoreReadError, StoreWriteError +from orcapod.contexts.stored import ( + StaleTable, + keep_rows_of_data_context, + require_same_data_context, + account_for_stale_tables, + find_stale_tables, + report_rows_of_other_data_contexts, + with_data_context, +) from orcapod.protocols.core_protocols import DataProtocol from orcapod.protocols.database_protocols import ArrowDatabaseProtocol -from orcapod.system_constants import constants, RESULT_DB_SCHEMA_VERSION +from orcapod.system_constants import ( + OLD_RESULT_DB_SCHEMA_VERSIONS, + RESULT_DB_SCHEMA_VERSION, + constants, +) from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -28,12 +41,6 @@ logger = logging.getLogger(__name__) -# Process-level cache of v1 result DB paths that have already been checked for -# legacy v0 schema. Populated on first access; prevents repeated table_exists -# calls for the same path within a single process. -_checked_rdb_paths: set[tuple[str, ...]] = set() - - class ResultCache: """Shared result caching backed by an ``ArrowDatabaseProtocol``. @@ -60,33 +67,39 @@ def __init__( result_database: ArrowDatabaseProtocol, record_path: tuple[str, ...], auto_flush: bool = True, + delivers_side_effects: bool = False, ) -> None: + """Create a result cache over one result table. + + Args: + result_database: The store holding the result table. + record_path: Path of the result table without its layout component. + auto_flush: Whether to flush the store after each stored result. + delivers_side_effects: Whether computing a result is an external action + (a function pod that takes the invocation context). Such a cache + fails, instead of computing again, when it finds results stored + under another data context or at an older table layout. + """ self._result_database = result_database self._record_path = record_path self._auto_flush = auto_flush - self._ignore_schema: tuple[str, ...] | None = None + self._delivers_side_effects = delivers_side_effects + # Set once a read has reported the stored rows it skipped. + self._skipped_rows_reported = False + # Set once stored results of other generations have been accounted for. + self._stale_tables_checked = False @property def result_database(self) -> ArrowDatabaseProtocol: """The underlying database.""" return self._result_database - @property - def base_record_path(self) -> tuple[str, ...]: - """The unversioned base record path (v0 path), without schema version suffix. - - Used by migration utilities that need to reference the legacy v0 table - path. The public ``record_path`` property returns the versioned v1 path. - """ - return self._record_path - @property def record_path(self) -> tuple[str, ...]: """The versioned path where records are stored. - Returns ``_record_path + (RESULT_DB_SCHEMA_VERSION,)`` — the actual - storage location of result records in the v1 schema. Use - ``base_record_path`` to access the unversioned path. + Returns ``_record_path + (RESULT_DB_SCHEMA_VERSION,)``: the actual storage + location of result records. The layout version is the last component. """ return self._versioned_record_path @@ -99,66 +112,60 @@ def set_auto_flush(self, on: bool = True) -> None: """Set auto-flush behavior.""" self._auto_flush = on - def set_ignore_schema(self, ignore_schema: tuple[str, ...] | None) -> None: - """Set which old schema versions to tolerate without raising ``SchemaVersionError``. + def stale_tables(self, data_context_key: str) -> list[StaleTable]: + """Find stored results of other generations: rows this cache will not use. - Args: - ignore_schema: Tuple of schema version strings to tolerate (e.g. - ``("v0",)``), or ``None`` to use the default (raise on any - old schema). - """ - self._ignore_schema = ignore_schema + Result tables at an older layout (``rdb_v1``, or no layout component), and, for a cache whose + function delivers side effects, rows of another data context in the current + table (ITL-944). The tables are only read. - def _ensure_rdb_schema(self) -> None: - """Check for a legacy v0 result DB on first access per path. + Args: + data_context_key: Key of the caller's data context. - Detection flow (runs at most once per v1 path per process): + Returns: + What ``find_stale_tables`` found; empty when there is nothing. + """ + return find_stale_tables( + self._result_database, + # Orcapod v0.1 stored the table at the path without a layout component. + tuple( + self._record_path + (layout,) + for layout in OLD_RESULT_DB_SCHEMA_VERSIONS + ) + + (self._record_path,), + current=(self._versioned_record_path, data_context_key) + if self._delivers_side_effects + else None, + ) - 1. If the v1 path is already in ``_checked_rdb_paths`` → return immediately. - 2. If the v1 table exists → mark checked and return. - 3. If the v0 path (bare ``_record_path``) has a table: - - ``"v0"`` in ``_ignore_schema`` → log info, continue. - - Otherwise → raise ``SchemaVersionError``. - 4. Neither path exists → fresh database, continue. - 5. Mark v1 path as checked. + def _account_for_stale_tables(self, data_context_key: str) -> None: + """Report or refuse, once per cache object, stored results of other generations. Raises: - SchemaVersionError: If a v0 table is detected and not ignored. + SideEffectLogDataContextError: If this cache's function delivers side + effects and such results exist; nothing has been computed then. """ - v1_path = self._versioned_record_path - if v1_path in _checked_rdb_paths: - return - if self._result_database.table_exists(v1_path): - _checked_rdb_paths.add(v1_path) + if self._stale_tables_checked: return - v0_path = self._record_path - if self._result_database.table_exists(v0_path): - ignore = self._ignore_schema or () - if "v0" not in ignore: - raise SchemaVersionError( - f"Result DB rows found at v0 schema path {'/'.join(v0_path)!r}.\n" - "Run migration first:\n" - f" orcapod migrate result-db {'/'.join(v0_path)}\n" - "To suppress this error and recompute all results instead, set:\n" - ' node.node_config = NodeConfig(ignore_schema=("v0",))' - ) - logger.info( - "Result DB v0 schema detected at %r — proceeding because " - "ignore_schema=%r", - v0_path, - ignore, - ) - # Only cache as checked after the raise-or-ignore decision passes. - _checked_rdb_paths.add(v1_path) + account_for_stale_tables( + self.stale_tables(data_context_key), + delivers_side_effects=self._delivers_side_effects, + current_context=data_context_key, + ) + self._stale_tables_checked = True def lookup( self, input_data: DataProtocol, additional_constraints: dict[str, Any] | None = None, + *, + data_context_key: str, ) -> DataProtocol | None: """Look up a cached output data for *input_data*. - The default match is by ``INPUT_DATA_HASH_COL`` only. + The default match is by ``INPUT_DATA_HASH_COL`` among the rows written under + the data context ``data_context_key``: a result stored under another data + context for the same input hash is not a match (ITL-944). *additional_constraints* can narrow the match further (e.g. by function variation hash for stricter cache invalidation). @@ -171,6 +178,8 @@ def lookup( include in the lookup query. Values may be ``bytes`` (for binary hash columns) or other scalar types (e.g. ``str`` for ``function_name``). + data_context_key: Key of the caller's data context. Only rows written + under it are matched, and the returned data carries it. Returns: The cached output data with ``RESULT_COMPUTED_FLAG: False`` @@ -178,7 +187,8 @@ def lookup( """ from orcapod.core.datagrams import Data - self._ensure_rdb_schema() + require_same_data_context(input_data.data_context_key, data_context_key) + self._account_for_stale_tables(data_context_key) RECORD_ID_COL = "_record_id" @@ -186,7 +196,14 @@ def lookup( constants.INPUT_DATA_HASH_COL: input_data.content_hash().to_prefixed_digest(), } if additional_constraints: + if constants.DATA_CONTEXT_COL in additional_constraints: + raise ValueError( + f"additional_constraints must not constrain " + f"{constants.DATA_CONTEXT_COL!r}: a lookup always matches the " + "caller's data context only." + ) constraints.update(additional_constraints) + constraints[constants.DATA_CONTEXT_COL] = data_context_key result_table = self._result_database.get_records_with_column_value( self._versioned_record_path, @@ -225,6 +242,7 @@ def lookup( result_table, record_uuid=record_uuid, meta_info={self.RESULT_COMPUTED_FLAG: False}, + data_context=data_context_key, ) def store( @@ -234,11 +252,14 @@ def store( variation_datagram: "DatagramProtocol", execution_datagram: "DatagramProtocol", skip_duplicates: bool = False, + *, + data_context_key: str, ) -> None: """Store an output data in the cache. Stores the output data data alongside function variation and - execution metadata (as Datagrams), input data hash, and a timestamp. + execution metadata (as Datagrams), input data hash, a timestamp and the + key of the data context that wrote the row (``__data_context``). Args: input_data: The input data (used for its content hash). @@ -247,10 +268,13 @@ def store( execution_datagram: Execution environment metadata as a Datagram. skip_duplicates: If True, silently skip if a record with the same ID already exists. + data_context_key: Key of the caller's data context, stored on the row. """ - self._ensure_rdb_schema() + require_same_data_context(input_data.data_context_key, data_context_key) + require_same_data_context(output_data.data_context_key, data_context_key) + self._account_for_stale_tables(data_context_key) - data_table = output_data.as_table(columns={"source": True, "context": True}) + data_table = output_data.as_table(columns={"source": True}) # Add variation and execution columns with prefixes. # Use a running counter for insertion position since add_column shifts indices. @@ -286,6 +310,7 @@ def store( constants.POD_TIMESTAMP, pa.array([datetime.now(timezone.utc)], type=pa.timestamp("us", tz="UTC")), ) + data_table = with_data_context(data_table, data_context_key) try: self._result_database.add_record( @@ -313,23 +338,38 @@ def store( ) from exc def get_all_records( - self, include_system_columns: bool = False + self, data_context_key: str, include_system_columns: bool = False ) -> "pa.Table | None": - """Return all cached records from the result store. + """Return the cached records written under the data context ``data_context_key``. + + Rows written under another data context are not current entries; they stay + in the table and are not returned (ITL-944). The ``__data_context`` column is + not part of the returned table. Args: + data_context_key: Key of the caller's data context. include_system_columns: If True, include system columns (e.g. record_id) in the result. Returns: A PyArrow table of cached results, or ``None`` if empty. """ - self._ensure_rdb_schema() + self._account_for_stale_tables(data_context_key) record_id_column = constants.DATA_RECORD_ID if include_system_columns else None result_table = self._result_database.get_all_records( self._versioned_record_path, record_id_column=record_id_column ) - if result_table is None or result_table.num_rows == 0: + if result_table is None: + return None + stored_rows = result_table.num_rows + result_table = keep_rows_of_data_context(result_table, data_context_key) + if result_table.num_rows < stored_rows and not self._skipped_rows_reported: + self._skipped_rows_reported = True + report_rows_of_other_data_contexts( + self._result_database.base_path + self._versioned_record_path, + stored_rows - result_table.num_rows, + ) + if result_table.num_rows == 0: return None - return result_table + return result_table.drop([constants.DATA_CONTEXT_COL]) diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index 13fa0cccc..13fedef7e 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -8,8 +8,17 @@ from orcapod.config import OrcapodConfig from orcapod.core.sources.base import RootSource from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.contexts.stored import ( + keep_rows_of_data_context, + require_same_data_context, + account_for_stale_tables, + find_stale_tables, + report_rows_of_other_data_contexts, + with_data_context, +) from orcapod.protocols.core_protocols import DataProtocol, SourceProtocol, TagProtocol from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.system_constants import SOURCE_DB_SCHEMA_VERSION, constants from orcapod.types import ColumnConfig, Schema from orcapod.utils.lazy_module import LazyModule @@ -50,6 +59,11 @@ class CachedSource(RootSource): HASH_COLUMN_NAME = "_record_hash" + # Set once a read has reported the cached rows it skipped. + _skipped_rows_reported: bool = False + # Set once a cache table found at the older layout has been accounted for. + _stale_tables_checked: bool = False + def __init__( self, source: SourceProtocol, @@ -110,7 +124,8 @@ def to_config( "inner_source": inner_source_config, "cache_database": cache_db_ref, "cache_path_prefix": list(self._cache_path_prefix), - "cache_path": list(self.cache_path), + # The path without its layout component, as ``cache_path`` takes it. + "cache_path": list(self._unversioned_cache_path), "source_id": self.source_id, # Note: identity fields (content_hash, pipeline_hash, etc.) intentionally omitted } @@ -180,8 +195,11 @@ def identity_structure(self) -> Any: # ------------------------------------------------------------------------- @property - def cache_path(self) -> tuple[str, ...]: - """Cache table path, scoped to the source's content hash.""" + def _unversioned_cache_path(self) -> tuple[str, ...]: + """The cache path without its layout component. + + This is where Orcapod up to v0.2 stored the cache table (the old layout). + """ if self._explicit_cache_path is not None: return self._explicit_cache_path return self._cache_path_prefix + ( @@ -189,6 +207,31 @@ def cache_path(self) -> tuple[str, ...]: f"node:{self._source.content_hash().to_string()}", ) + @property + def cache_path(self) -> tuple[str, ...]: + """Cache table path: scoped to the source's content hash, then the layout version. + + The layout version (``sdb_v1``) is always the last component, also when an + explicit ``cache_path`` was given. + """ + return self._unversioned_cache_path + (SOURCE_DB_SCHEMA_VERSION,) + + def _account_for_stale_tables(self) -> None: + """Report, once per source object, a cache table found at the older layout. + + A cache written by an earlier version of Orcapod holds no ``__data_context`` + column. Its rows are not read and are left unchanged; their number is + reported (ITL-944). + """ + if self._stale_tables_checked: + return + self._stale_tables_checked = True + account_for_stale_tables( + find_stale_tables(self._cache_database, (self._unversioned_cache_path,)), + delivers_side_effects=False, + current_context=self.data_context_key, + ) + # ------------------------------------------------------------------------- # Stream interface — delegate schema, materialize with cache # ------------------------------------------------------------------------- @@ -215,6 +258,7 @@ def _ingest_live_data(self) -> None: Raises if the source cannot provide data (e.g. an unbound ``SourceProxy``). """ + require_same_data_context(self._source.data_context_key, self.data_context_key) live_table = self._source.as_table( columns={"source": True, "system_tags": True} ) @@ -238,14 +282,44 @@ def _ingest_live_data(self) -> None: self.HASH_COLUMN_NAME, pa.array([h.encode() for h in record_hashes], type=pa.large_binary()), ) + # The data context column is appended after the record hashes are computed: + # it names the generation of hashing that produced them and is not hashed. + self._account_for_stale_tables() self._cache_database.add_records( self.cache_path, - live_with_hash, + with_data_context(live_with_hash, self.data_context_key), record_id_column=self.HASH_COLUMN_NAME, skip_duplicates=True, ) self._cache_database.flush() + def _read_cached_records(self) -> "pa.Table | None": + """Read the cached rows written under the current data context. + + Every read of the whole cache table goes through here. Rows of another data + context stay in the table untouched, and a cache at the older layout is not + read (ITL-944). + + Returns: + The cached rows without the record hash and ``__data_context`` columns, + or ``None`` when the cache table does not exist. + """ + self._account_for_stale_tables() + records = self._cache_database.get_all_records( + self.cache_path, record_id_column=self.HASH_COLUMN_NAME + ) + if records is None: + return None + stored_rows = records.num_rows + records = keep_rows_of_data_context(records, self.data_context_key) + if records.num_rows < stored_rows and not self._skipped_rows_reported: + self._skipped_rows_reported = True + report_rows_of_other_data_contexts( + self._cache_database.base_path + self.cache_path, + stored_rows - records.num_rows, + ) + return records.drop([self.HASH_COLUMN_NAME, constants.DATA_CONTEXT_COL]) + def _build_merged_stream(self) -> ArrowTableStream: """Ingest live data (if available), then return all cached records. @@ -262,12 +336,15 @@ def _build_merged_stream(self) -> ArrowTableStream: self._source.source_id, ) - all_records = self._cache_database.get_all_records(self.cache_path) + all_records = self._read_cached_records() if all_records is None: all_records = self._empty_table() tag_keys = self._source.keys()[0] - return ArrowTableStream(all_records, tag_columns=tag_keys) + # Cached rows are this source's: give them its data context, not the default. + return ArrowTableStream( + all_records, tag_columns=tag_keys, data_context=self.data_context + ) def _empty_table(self) -> pa.Table: """Build an empty Arrow table matching the source's output schema.""" @@ -323,5 +400,5 @@ def get_all_records( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> "pa.Table | None": - """Retrieve all stored records from the cache database.""" - return self._cache_database.get_all_records(self.cache_path) + """Retrieve the cached records that the current Arrow hasher wrote.""" + return self._read_cached_records() diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 0e471a15d..f6f9ca8be 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -165,19 +165,6 @@ def __init__(self, empty_data: object) -> None: ) -class SchemaVersionError(Exception): - """Raised when an old schema version is detected at a database path. - - Occurs when a table exists at the unversioned (v0) path but not at the - expected versioned path, and the node has not opted in to tolerating - the old schema via ``NodeConfig.ignore_schema``. - - To suppress this error, set ``node.node_config = NodeConfig(ignore_schema=("v0",))`` - and re-run. The node will recompute all results from scratch rather than - reading from the v0 cache. - """ - - class EphemeralResultMissingError(Exception): """Raised when a downstream pod has a cache miss for an ``EmptyData`` input. @@ -406,3 +393,70 @@ def __str__(self) -> str: "The table exists, so its rows are not treated as missing and the table " "was not overwritten." ) + + +class SideEffectLogDataContextError(StoreError): + """Raised when a side-effecting node finds completions logged under another data context. + + A node whose run is an external action finds what it already did by keys hashed + under its data context: a side-effect pod in its invocation log, a function pod + that takes the invocation context (``ctx_arg``) in its function tables. + Completions stored under another data context cannot be matched with current + inputs: tables written by Orcapod up to v0.2 (an older table layout, data context + ``std:v0.1:default``), or rows of another data context in the current tables + (ITL-944). Running the node would perform every side effect again. Unlike a + computed result, a side effect cannot be taken back, so the node raises this + error before it does anything and leaves the tables as they are. Other function, + operator and source tables do not fail this way: there, such rows are reported, + skipped and computed again. + + Use a fresh store for the node, or keep using the Orcapod version that wrote the + tables; migrating stored rows to the current data context is planned. + + Attributes: + record_paths: Paths, from the root of the store, of the tables holding such + completions. + record_path: The first of ``record_paths``. + found_context: Data context key of the completions that do not match. + current_context: Data context key of the node. + node_label: Label of the node. + """ + + def __init__( + self, + record_paths: tuple[tuple[str, ...], ...], + found_context: str, + current_context: str, + node_label: str | None = None, + ) -> None: + """Create the error for the tables of one node. + + Args: + record_paths: Paths of the tables holding completions that do not match. + found_context: Data context key of those completions. + current_context: Data context key of the node. + node_label: Label of the node, if known. + """ + self.record_paths: tuple[tuple[str, ...], ...] = tuple( + tuple(path) for path in record_paths + ) + self.record_path: tuple[str, ...] = self.record_paths[0] + self.found_context: str = found_context + self.current_context: str = current_context + super().__init__( + self.record_paths, found_context, current_context, node_label=node_label + ) + + def __str__(self) -> str: + """Describe the tables, both data contexts and the remedy.""" + node = f"Node {self.node_label!r}" if self.node_label else "This node" + tables = ", ".join(repr("/".join(path)) for path in self.record_paths) + return ( + f"{node} performs side effects, and its stored completions were logged " + f"under the data context {self.found_context!r} (in {tables}), but the " + f"node runs under {self.current_context!r}. They cannot be matched with " + "current inputs, so running the node would perform every side effect " + "again; nothing was delivered or written. Use a fresh store for this " + "node, or keep using the Orcapod version that wrote these tables; they " + "are left in place and can be migrated to the current data context later." + ) diff --git a/src/orcapod/hashing/arrow_hashers.py b/src/orcapod/hashing/arrow_hashers.py index 54c27a528..521fb969b 100644 --- a/src/orcapod/hashing/arrow_hashers.py +++ b/src/orcapod/hashing/arrow_hashers.py @@ -8,6 +8,7 @@ from orcapod.hashing.schema_cleaner import ( clean_schema_for_hashing, has_extension_metadata, + has_extension_type, ) from orcapod.hashing.visitors import SemanticHashingVisitor from orcapod.utils.arrow_utils import normalize_extension_columns @@ -30,8 +31,22 @@ class StarfixArrowHasher: Extension-typed columns without a registered hasher pass through with their full extension metadata intact. 2. **Starfix hashing** — ``ArrowDigester.hash_table`` produces a 35-byte - versioned SHA-256 digest that is byte-for-byte identical to the Rust - ``starfix`` crate output. + digest: the starfix hash format number ``00 00 02`` (hash format 2, + starfix 0.5) followed by a SHA-256 digest, byte-for-byte identical to + the Rust ``starfix`` crate output. The order of columns and of struct + fields is not part of the hash (starfix's order options stay off); row + order and list element order are. Extension identity + (``ARROW:extension:name`` and ``ARROW:extension:metadata``) is hashed + through starfix's ``include_metadata`` whenever the table has an + extension column, including pyarrow's built-in canonical extension types + such as ``arrow.uuid``. + + starfix raises ``ValueError`` for inputs it cannot hash: data of ``null``, + ``map``, interval and view types (whatever the row count); union and + run-end-encoded types; two schema nodes sharing a path (duplicate column + names, or a ``/`` in a name aliasing a struct child); and a non-nullable + field holding a null. Those errors propagate unchanged; they name the + column path and type. Parameters ---------- @@ -127,8 +142,14 @@ def _process_table_columns(self, table: "pa.Table | pa.RecordBatch") -> "pa.Tabl return normalize_extension_columns(intermediate) def hash_schema(self, schema: "pa.Schema") -> ContentHash: - """Hash an Arrow schema using the starfix canonical algorithm.""" - include_meta = has_extension_metadata(schema) + """Hash an Arrow schema using the starfix canonical algorithm. + + Extension-typed columns are normalized to their storage type with the + extension identity in field metadata, as ``hash_table`` does, so the + extension name and metadata are part of the hash. + """ + schema = normalize_extension_columns(schema.empty_table()).schema + include_meta = has_extension_metadata(schema) or has_extension_type(schema) if include_meta: schema = clean_schema_for_hashing(schema) digest = ArrowDigester.hash_schema(schema, include_metadata=include_meta) @@ -140,7 +161,9 @@ def hash_table(self, table: "pa.Table | pa.RecordBatch") -> ContentHash: table = pa.Table.from_batches([table]) processed_table = self._process_table_columns(table) - include_meta = has_extension_metadata(processed_table.schema) + include_meta = has_extension_metadata( + processed_table.schema + ) or has_extension_type(processed_table.schema) if include_meta: clean_schema = clean_schema_for_hashing(processed_table.schema) clean_table = pa.Table.from_arrays( diff --git a/src/orcapod/hashing/schema_cleaner.py b/src/orcapod/hashing/schema_cleaner.py index 09a55768f..a3021c811 100644 --- a/src/orcapod/hashing/schema_cleaner.py +++ b/src/orcapod/hashing/schema_cleaner.py @@ -162,6 +162,47 @@ def clean_schema_for_hashing(schema: pa.Schema) -> pa.Schema: ) +def _contains_extension_type(arrow_type: pa.DataType) -> bool: + """Return True if ``arrow_type`` is or contains an Arrow extension type.""" + if isinstance(arrow_type, pa.BaseExtensionType): + return True + if pa.types.is_struct(arrow_type): + return any( + _contains_extension_type(arrow_type.field(i).type) + for i in range(arrow_type.num_fields) + ) + if ( + pa.types.is_list(arrow_type) + or pa.types.is_large_list(arrow_type) + or pa.types.is_fixed_size_list(arrow_type) + ): + return _contains_extension_type(arrow_type.value_type) + if pa.types.is_map(arrow_type): + return _contains_extension_type( + arrow_type.key_type + ) or _contains_extension_type(arrow_type.item_type) + if pa.types.is_dictionary(arrow_type): + return _contains_extension_type(arrow_type.value_type) + return False + + +def has_extension_type(schema: pa.Schema) -> bool: + """Return True if any field of ``schema``, at any depth, has an extension type. + + Covers Python-defined ``pa.ExtensionType`` subclasses and pyarrow's built-in canonical + extension types (``arrow.uuid``, ``arrow.fixed_shape_tensor``, ...), which derive from + ``pa.BaseExtensionType`` only. starfix hashes an extension type as its storage type and + adds its ``ARROW:extension:*`` identity only when metadata is included. + + Args: + schema: The ``pa.Schema`` to inspect. + + Returns: + True if an extension type occurs anywhere in ``schema``; False otherwise. + """ + return any(_contains_extension_type(field.type) for field in schema) + + def has_extension_metadata(schema: pa.Schema) -> bool: """Return True if ``schema`` has any ``ARROW:extension:name`` key at any depth. diff --git a/src/orcapod/hashing/versioned_hashers.py b/src/orcapod/hashing/versioned_hashers.py index c75cf7ed5..1aa1c84f4 100644 --- a/src/orcapod/hashing/versioned_hashers.py +++ b/src/orcapod/hashing/versioned_hashers.py @@ -36,8 +36,10 @@ # hashes can be distinguished from newly-computed ones. _CURRENT_SEMANTIC_HASHER_ID = "semantic_v0.1" -# The hasher_id for the Arrow hasher. -_CURRENT_ARROW_HASHER_ID = "arrow_v0.1" +# The hasher_id for the Arrow hasher. Bumped whenever the hash of any table can change, +# whatever the cause; its number is not kept equal to the starfix hash format number. +# arrow_v0.1: starfix 0.4 (hash format 1). arrow_v0.2: starfix 0.5 (hash format 2, ITL-910). +_CURRENT_ARROW_HASHER_ID = "arrow_v0.2" # --------------------------------------------------------------------------- diff --git a/src/orcapod/logical_types/__init__.py b/src/orcapod/logical_types/__init__.py index d5b50341c..82fbea8e7 100644 --- a/src/orcapod/logical_types/__init__.py +++ b/src/orcapod/logical_types/__init__.py @@ -4,7 +4,7 @@ Python classes to their Arrow/Polars extension type representation. Built-in registrations (``LogicalPath``, ``LogicalUPath``, ``LogicalUUID``) are -wired into ``DataContext`` via ``contexts/data/v0.1.json``. Use +wired into ``DataContext`` via ``contexts/logical_types.json``. Use ``get_default_context().type_converter.register_python_class()`` to register new types, ``register_logical_type_factory()`` to add factories, and ``apply_logical_types()`` to re-wrap Arrow tables with their registered extension types. diff --git a/src/orcapod/logical_types/builtin_logical_types.py b/src/orcapod/logical_types/builtin_logical_types.py index 5c706e8b1..bc307102f 100644 --- a/src/orcapod/logical_types/builtin_logical_types.py +++ b/src/orcapod/logical_types/builtin_logical_types.py @@ -1,7 +1,7 @@ """Built-in LogicalType implementations for orcapod. Provides three built-in logical types registered into the default -``DataContext.logical_type_registry`` via ``contexts/data/v0.1.json``: +``DataContext.logical_type_registry`` via ``contexts/logical_types.json``: - ``LogicalPath``: maps ``pathlib.Path`` ↔ Arrow large_string extension ``"orcapod.path"`` - ``LogicalUPath``: maps ``upath.UPath`` ↔ Arrow large_string extension ``"orcapod.upath"`` diff --git a/src/orcapod/migrations/__init__.py b/src/orcapod/migrations/__init__.py deleted file mode 100644 index 3c190cd21..000000000 --- a/src/orcapod/migrations/__init__.py +++ /dev/null @@ -1,19 +0,0 @@ -"""Orcapod schema migration utilities. - -Public API: - ``migrate_pipeline_v0_to_v1`` — migrate a pipeline DB table from v0 to v1. - ``migrate_result_v0_to_v1`` — migrate a result DB table from v0 to v1. - ``migrate_node`` — convenience wrapper: migrate a ``FunctionJobNode`` in one call. - ``MigrationResult`` — dataclass summarising a migration run. -""" - -from orcapod.migrations.types import MigrationResult -from orcapod.migrations.result_db import migrate_result_v0_to_v1 -from orcapod.migrations.pipeline_db import migrate_pipeline_v0_to_v1, migrate_node - -__all__ = [ - "MigrationResult", - "migrate_pipeline_v0_to_v1", - "migrate_result_v0_to_v1", - "migrate_node", -] diff --git a/src/orcapod/migrations/pipeline_db.py b/src/orcapod/migrations/pipeline_db.py deleted file mode 100644 index 85338d83e..000000000 --- a/src/orcapod/migrations/pipeline_db.py +++ /dev/null @@ -1,402 +0,0 @@ -"""v0 → v1 migration for the Orcapod pipeline DB (pdb).""" - -from __future__ import annotations - -import logging -import time -from typing import TYPE_CHECKING, Any - -import pyarrow as pa - -from orcapod.hashing.defaults import get_default_arrow_hasher -from orcapod.migrations.types import MigrationResult -from orcapod.system_constants import PIPELINE_DB_SCHEMA_VERSION, constants -from orcapod.types import ContentHash - -if TYPE_CHECKING: - from orcapod.protocols.database_protocols import ArrowDatabaseProtocol - from orcapod.core.nodes.function_node import FunctionJobNode - -logger = logging.getLogger(__name__) - -# Column name used to expose the internal record ID when reading from the DB. -_RECORD_ID_COL = "__record_id" - -# v0 pdb rows used DATAGRAM_PREFIX ("_") for NODE_CONTENT_HASH_COL, giving the column -# name "_node_content_hash". The fix to ITL-533 changed the prefix to SYSTEM_COLUMN_PREFIX -# ("__"), so v1 rows use "__node_content_hash". The migration must rename the column as -# it converts the string value to binary. -_V0_NODE_HASH_COL = "_node_content_hash" - -# pdb columns whose values are ContentHash strings in v0 and must become binary in v1. -# NODE_CONTENT_HASH_COL here refers to the v1 name ("__node_content_hash"). -_PDB_HASH_COLS = frozenset( - { - constants.NODE_CONTENT_HASH_COL, - constants.INPUT_DATA_HASH_COL, - constants.OUTPUT_DATA_HASH_COL, - } -) - - -def migrate_pipeline_v0_to_v1( - pipeline_db: "ArrowDatabaseProtocol", - pipeline_path: tuple[str, ...], - result_db: "ArrowDatabaseProtocol", - result_path: tuple[str, ...], - *, - dry_run: bool = False, - batch_size: int = 500, - progress: bool = True, - track_skipped: bool = True, -) -> MigrationResult: - """Migrate a pipeline DB table from v0 schema to v1 schema. - - Reads records from ``pipeline_path`` (v0, no suffix), converts - ContentHash columns (``__node_content_hash``, ``__input_data_hash``, - ``__output_data_hash``) from ``large_string`` to ``large_binary``, - and writes the transformed rows to ``pipeline_path + ("pdb_v1",)``. - ``__node_content_hash`` is retained in v1 rows for per-node isolation. - - **Backfill strategy for ``__input_data_hash``:** when the column value is - ``None`` in the pdb row (older writes may lack it), the migration falls - back to the rdb via a per-batch ``get_records_by_ids`` lookup keyed by - ``DATA_RECORD_ID``. Rows where both the pdb value is ``None`` and the rdb - lookup fails are counted as ``rows_unresolvable`` and written with a - ``null`` ``__input_data_hash``. - - **``__output_data_hash`` is not backfilled** — the rdb does not store this - value, so rows where it is ``None`` in v0 will remain ``null`` in v1 - (these are not counted as unresolvable). - - Rows already present at the v1 path are skipped (idempotent re-runs). - - Args: - pipeline_db: The database containing the v0 pipeline table. - pipeline_path: Bare v0 pipeline path tuple (no ``pdb_v1`` suffix). - result_db: The database containing the v0 result table. - result_path: Bare v0 result path tuple (no ``rdb_v1`` suffix). - dry_run: If ``True``, read and count rows but write nothing. - batch_size: Number of rows to process per batch. - progress: If ``True``, log progress at INFO level. - track_skipped: If ``True`` (default), scan the v1 table upfront to - detect already-migrated rows (idempotent re-run support). Set - to ``False`` to skip this scan on the first run of a large table - where no v1 rows exist yet; ``rows_skipped`` will be reported as - ``0`` in that case. - - Returns: - ``MigrationResult`` summarising the run. - """ - v1_path = pipeline_path + (PIPELINE_DB_SCHEMA_VERSION,) - start = time.monotonic() - - # Read v0 records with their internal UUIDs exposed as _RECORD_ID_COL. - v0_table = pipeline_db.get_all_records( - pipeline_path, record_id_column=_RECORD_ID_COL - ) - if v0_table is None or v0_table.num_rows == 0: - logger.info("No v0 records found at %r — nothing to migrate.", pipeline_path) - return MigrationResult( - rows_total=0, - rows_migrated=0, - rows_skipped=0, - rows_unresolvable=0, - elapsed_s=time.monotonic() - start, - dry_run=dry_run, - ) - - rows_total = v0_table.num_rows - - # Collect IDs already at v1 for idempotency (skipped when track_skipped=False). - existing_ids: set[bytes] = set() - if track_skipped: - v1_existing = pipeline_db.get_all_records( - v1_path, record_id_column=_RECORD_ID_COL - ) - if v1_existing is not None and _RECORD_ID_COL in v1_existing.schema.names: - existing_ids = { - bytes(r) - for r in v1_existing.column(_RECORD_ID_COL).to_pylist() - if r is not None - } - - if progress: - logger.info( - "migrate_pipeline_v0_to_v1: found %d rows at v0 path %r", - rows_total, - pipeline_path, - ) - - rows_migrated = 0 - rows_skipped = 0 - rows_unresolvable = 0 - arrow_hasher = get_default_arrow_hasher() - - for batch_start in range(0, rows_total, batch_size): - batch = v0_table.slice(batch_start, batch_size) - - # Build a per-batch rdb index for rows where __input_data_hash is None. - # Uses get_records_by_ids to avoid loading the entire rdb into memory. - input_hash_col = constants.INPUT_DATA_HASH_COL - data_id_col = constants.DATA_RECORD_ID - needed_ids: set[bytes] = set() - for row in batch.to_pylist(): - if row.get(input_hash_col) is None: - data_id = row.get(data_id_col) - if data_id is not None: - needed_ids.add(bytes(data_id)) - - rdb_index: dict[bytes, dict] = {} - if needed_ids: - rdb_batch = result_db.get_records_by_ids( - result_path, needed_ids, record_id_column=_RECORD_ID_COL - ) - if rdb_batch is not None: - for row in rdb_batch.to_pylist(): - rid = row.get(_RECORD_ID_COL) - if rid is not None: - rdb_index[bytes(rid)] = row - - transformed, batch_unresolvable = _transform_pdb_batch( - batch, rdb_index, arrow_hasher - ) - rows_unresolvable += batch_unresolvable - - # Skip rows whose recomputed record ID is already at v1 (idempotent re-runs). - # The recomputed ID (not the original v0 ID) is the authoritative key in v1. - if existing_ids and _RECORD_ID_COL in transformed.schema.names: - mask = pa.array( - [ - (rid is None or bytes(rid) not in existing_ids) - for rid in transformed.column(_RECORD_ID_COL).to_pylist() - ], - type=pa.bool_(), - ) - new_rows = transformed.filter(mask) - rows_skipped += transformed.num_rows - new_rows.num_rows - else: - new_rows = transformed - - if new_rows.num_rows == 0: - continue - - if not dry_run: - pipeline_db.add_records( - v1_path, - new_rows, - record_id_column=_RECORD_ID_COL, - skip_duplicates=True, - ) - rows_migrated += new_rows.num_rows - if progress: - logger.info( - "migrate_pipeline_v0_to_v1: %d/%d rows processed", - min(batch_start + batch_size, rows_total), - rows_total, - ) - - if not dry_run: - pipeline_db.flush() - - elapsed = time.monotonic() - start - return MigrationResult( - rows_total=rows_total, - rows_migrated=rows_migrated, - rows_skipped=rows_skipped, - rows_unresolvable=rows_unresolvable, - elapsed_s=elapsed, - dry_run=dry_run, - ) - - -def _transform_pdb_batch( - batch: pa.Table, - rdb_index: dict[bytes, dict], - arrow_hasher: Any, -) -> tuple[pa.Table, int]: - """Transform a batch of v0 pdb rows into v1 format. - - Per-row transformations (applied in order): - - 1. Rename ``_node_content_hash`` (v0, ``DATAGRAM_PREFIX``) to - ``__node_content_hash`` (v1, ``SYSTEM_COLUMN_PREFIX``) and convert from - ``large_string`` → ``large_binary`` (no rdb fallback; retained in v1 for - per-node isolation). - 2. Convert ``__input_data_hash`` from ``large_string`` → ``large_binary`` - (falls back to rdb index when the pdb value is ``None``). - 3. Convert ``__output_data_hash`` from ``large_string`` → ``large_binary`` - (no rdb fallback; ``None`` stays ``null``). - 4. Recompute ``__pipeline_base_entry_id`` and ``__record_id`` using the new - preimage (``system_tag_cols + INPUT_DATA_HASH_COL``, without - ``NODE_CONTENT_HASH_COL``). For rows where ``__input_data_hash`` cannot be - resolved, step 4 is skipped entirely, and those rows retain their original - v0 values for ``__pipeline_base_entry_id`` and ``__record_id``. - - Args: - batch: Arrow table slice of v0 pdb rows (with ``_RECORD_ID_COL`` as first - column, as returned by ``get_all_records(record_id_column=...)``). - rdb_index: Dict mapping rdb record-ID bytes to row dicts. - arrow_hasher: ``ArrowHasherProtocol`` used for recomputing the hash columns. - - Returns: - Tuple of (transformed Arrow table, count of unresolvable rows). - """ - node_hash_col = constants.NODE_CONTENT_HASH_COL - input_hash_col = constants.INPUT_DATA_HASH_COL - output_hash_col = constants.OUTPUT_DATA_HASH_COL - data_id_col = constants.DATA_RECORD_ID - sys_tag_prefix = constants.SYSTEM_TAG_PREFIX - - rows = batch.to_pylist() - unresolvable = 0 - out_rows: list[dict] = [] - - for row in rows: - new_row = dict(row) - - # 1. Convert node_content_hash from string → binary and rename from v0 - # column name (_node_content_hash) to v1 column name (__node_content_hash). - # Pop the old v0 name; fall back to the new name for idempotent re-runs. - val = new_row.pop(_V0_NODE_HASH_COL, None) - if val is None: - val = new_row.get(node_hash_col) - if val is not None and isinstance(val, str): - new_row[node_hash_col] = ContentHash.from_string(val).to_prefixed_digest() - elif val is not None: - # Already binary (e.g. idempotent re-run on a partially migrated row). - new_row[node_hash_col] = val - - # 2. Convert __input_data_hash from string → binary. - val = new_row.get(input_hash_col) - if val is not None and isinstance(val, str): - new_row[input_hash_col] = ContentHash.from_string(val).to_prefixed_digest() - elif val is None: - # Missing in pdb — fall back to rdb index. - data_id = row.get(data_id_col) - if data_id is not None: - data_id_bytes = bytes(data_id) - rdb_row = rdb_index.get(data_id_bytes) - if rdb_row is not None: - raw = rdb_row.get(input_hash_col) - if raw is not None and isinstance(raw, str): - new_row[input_hash_col] = ContentHash.from_string( - raw - ).to_prefixed_digest() - else: - unresolvable += 1 - - # 3. Convert __output_data_hash from string → binary. - val = new_row.get(output_hash_col) - if val is not None and isinstance(val, str): - new_row[output_hash_col] = ContentHash.from_string(val).to_prefixed_digest() - - # 4. Recompute __pipeline_base_entry_id and __record_id using new preimage. - input_hash_bytes = new_row.get(input_hash_col) - if input_hash_bytes is not None: - sys_tag_cols = sorted(c for c in row if c.startswith(sys_tag_prefix)) - preimage_arrays: dict[str, pa.Array] = {} - for col in sys_tag_cols: - preimage_arrays[col] = pa.array([row.get(col)], type=pa.large_string()) - preimage_arrays[input_hash_col] = pa.array( - [input_hash_bytes], type=pa.large_binary() - ) - preimage = pa.table(preimage_arrays) - - new_base_entry_id = arrow_hasher.hash_table(preimage).to_prefixed_digest() - new_row["__pipeline_base_entry_id"] = new_base_entry_id - - recomp_idx = new_row.get("__pipeline_recomputation_index") or 0 - preimage_with_idx = preimage.append_column( - "__pipeline_recomputation_index", - pa.array([recomp_idx], type=pa.int32()), - ) - new_row[_RECORD_ID_COL] = arrow_hasher.hash_table( - preimage_with_idx - ).to_prefixed_digest() - - out_rows.append(new_row) - - transformed = pa.Table.from_pylist(out_rows, schema=_v1_pdb_schema(batch)) - return transformed, unresolvable - - -def _v1_pdb_schema(v0_batch: pa.Table) -> pa.Schema: - """Derive the v1 pdb Arrow schema from a v0 batch. - - Replaces all ContentHash columns with ``large_binary`` equivalents and - renames ``_node_content_hash`` (v0 ``DATAGRAM_PREFIX`` name) to - ``__node_content_hash`` (v1 ``SYSTEM_COLUMN_PREFIX`` name). - ``__node_content_hash`` is retained in v1 for per-node isolation (not - dropped). All other columns retain their original types. - - Args: - v0_batch: A v0 pdb Arrow table (used to read non-hash column types). - - Returns: - Arrow schema for the v1 pdb table. - """ - fields = [] - for field in v0_batch.schema: - if field.name == _V0_NODE_HASH_COL: - # Renamed in v1: skip here; the v1 name is added via _PDB_HASH_COLS below. - continue - if field.name in _PDB_HASH_COLS: - fields.append(pa.field(field.name, pa.large_binary(), nullable=True)) - else: - fields.append(field) - # Ensure all hash columns (using v1 names) exist in the schema. - existing_names = {f.name for f in fields} - for col in _PDB_HASH_COLS: - if col not in existing_names: - fields.append(pa.field(col, pa.large_binary(), nullable=True)) - return pa.schema(fields) - - -def migrate_node( - node: "FunctionJobNode", - *, - dry_run: bool = False, - batch_size: int = 500, - progress: bool = True, -) -> MigrationResult: - """Convenience wrapper: migrate a single ``FunctionJobNode``'s pipeline DB. - - Extracts the pipeline DB, pipeline path, result DB, and result path - directly from the node and delegates to ``migrate_pipeline_v0_to_v1()``. - - The ``result_path`` used is the unversioned v0 path (``base_record_path``), - NOT the versioned v1 path returned by ``record_path``. - - Args: - node: The ``FunctionJobNode`` whose pipeline DB to migrate. - dry_run: If ``True``, read and count rows but write nothing. - batch_size: Rows to process per batch. - progress: If ``True``, log progress at INFO level. - - Returns: - ``MigrationResult`` summarising the run. - - Raises: - RuntimeError: If the node has no pipeline database attached. - """ - if node._pipeline_database is None: - raise RuntimeError( - f"Node {node.label!r} has no pipeline database — cannot migrate." - ) - cached_pod = node._cached_function_pod - if cached_pod is None: - raise RuntimeError( - f"Node {node.label!r} has no cached function pod — cannot locate result DB." - ) - return migrate_pipeline_v0_to_v1( - pipeline_db=node._pipeline_database, - pipeline_path=node.node_identity_path, - result_db=cached_pod._cache.result_database, - # Use the public base_record_path (unversioned v0 path being migrated FROM), - # not the versioned record_path (which points to the v1 destination). - result_path=cached_pod._cache.base_record_path, - dry_run=dry_run, - batch_size=batch_size, - progress=progress, - ) diff --git a/src/orcapod/migrations/result_db.py b/src/orcapod/migrations/result_db.py deleted file mode 100644 index 5a4710dae..000000000 --- a/src/orcapod/migrations/result_db.py +++ /dev/null @@ -1,183 +0,0 @@ -"""v0 → v1 migration for the Orcapod result DB (rdb).""" - -from __future__ import annotations - -import logging -import time -from typing import TYPE_CHECKING - -import pyarrow as pa - -from orcapod.migrations.types import MigrationResult -from orcapod.system_constants import RESULT_DB_SCHEMA_VERSION, constants -from orcapod.types import ContentHash - -if TYPE_CHECKING: - from orcapod.protocols.database_protocols import ArrowDatabaseProtocol - -logger = logging.getLogger(__name__) - -# Column name used to expose the internal record ID when reading from the DB. -_RECORD_ID_COL = "__record_id" - -# rdb columns that hold orcapod-produced ContentHash values and must be -# converted from large_string (v0) to large_binary (v1). -_HASH_COLS = ( - constants.INPUT_DATA_HASH_COL, - f"{constants.PF_VARIATION_PREFIX}function_signature_hash", - f"{constants.PF_VARIATION_PREFIX}function_content_hash", -) - - -def migrate_result_v0_to_v1( - result_db: "ArrowDatabaseProtocol", - result_path: tuple[str, ...], - *, - dry_run: bool = False, - batch_size: int = 500, - progress: bool = True, - track_skipped: bool = True, -) -> MigrationResult: - """Migrate a result DB table from v0 schema to v1 schema. - - Reads records from ``result_path`` (v0, no suffix), converts all - orcapod-produced ``ContentHash`` columns from ``large_string`` to - ``large_binary`` (using ``ContentHash.to_prefixed_digest()``), and writes - the transformed rows to ``result_path + ("rdb_v1",)``. - - Rows already present at the v1 path are skipped (idempotent re-runs). - - Args: - result_db: The database containing the v0 table. - result_path: Bare v0 path tuple (no ``rdb_v1`` suffix). - dry_run: If ``True``, read and count rows but write nothing. - batch_size: Number of rows to process per batch. - progress: If ``True``, log progress at INFO level. - track_skipped: If ``True`` (default), scan the v1 table upfront to - detect already-migrated rows (idempotent re-run support). Set - to ``False`` to skip this scan on the first run of a large table - where no v1 rows exist yet; ``rows_skipped`` will be reported as - ``0`` in that case. - - Returns: - ``MigrationResult`` summarising the run. - """ - v1_path = result_path + (RESULT_DB_SCHEMA_VERSION,) - start = time.monotonic() - - # Read v0 records with their internal UUIDs exposed as _RECORD_ID_COL. - v0_table = result_db.get_all_records(result_path, record_id_column=_RECORD_ID_COL) - if v0_table is None or v0_table.num_rows == 0: - logger.info("No v0 records found at %r — nothing to migrate.", result_path) - return MigrationResult( - rows_total=0, - rows_migrated=0, - rows_skipped=0, - rows_unresolvable=0, - elapsed_s=time.monotonic() - start, - dry_run=dry_run, - ) - - rows_total = v0_table.num_rows - - # Collect IDs already at v1 for idempotency check (skipped when track_skipped=False). - existing_ids: set[bytes] = set() - if track_skipped: - v1_existing = result_db.get_all_records( - v1_path, record_id_column=_RECORD_ID_COL - ) - if v1_existing is not None and _RECORD_ID_COL in v1_existing.schema.names: - existing_ids = { - bytes(r) - for r in v1_existing.column(_RECORD_ID_COL).to_pylist() - if r is not None - } - - if progress: - logger.info( - "migrate_result_v0_to_v1: found %d rows at v0 path %r", - rows_total, - result_path, - ) - - rows_migrated = 0 - rows_skipped = 0 - - # Process in batches. - for batch_start in range(0, rows_total, batch_size): - batch = v0_table.slice(batch_start, batch_size) - - # Filter rows already at v1. - if existing_ids and _RECORD_ID_COL in batch.schema.names: - mask = pa.array( - [ - bytes(rid) not in existing_ids - for rid in batch.column(_RECORD_ID_COL).to_pylist() - ], - type=pa.bool_(), - ) - new_rows = batch.filter(mask) - rows_skipped += batch.num_rows - new_rows.num_rows - else: - new_rows = batch - - if new_rows.num_rows == 0: - continue - - # Convert hash columns from string → binary. - transformed = _convert_hash_cols(new_rows) - - if not dry_run: - result_db.add_records( - v1_path, - transformed, - record_id_column=_RECORD_ID_COL, - skip_duplicates=True, - ) - rows_migrated += new_rows.num_rows - if progress: - logger.info( - "migrate_result_v0_to_v1: %d/%d rows processed", - min(batch_start + batch_size, rows_total), - rows_total, - ) - - if not dry_run: - result_db.flush() - - elapsed = time.monotonic() - start - return MigrationResult( - rows_total=rows_total, - rows_migrated=rows_migrated, - rows_skipped=rows_skipped, - rows_unresolvable=0, # rdb migration has no unresolvable rows - elapsed_s=elapsed, - dry_run=dry_run, - ) - - -def _convert_hash_cols(table: pa.Table) -> pa.Table: - """Convert all v0 large_string ContentHash columns to v1 large_binary. - - Args: - table: Arrow table with v0-format hash columns. - - Returns: - New table with hash columns converted to ``large_binary``. - """ - for col_name in _HASH_COLS: - if col_name not in table.schema.names: - continue - col_idx = table.schema.names.index(col_name) - string_vals = table.column(col_name).to_pylist() - binary_vals = pa.array( - [ - ContentHash.from_string(s).to_prefixed_digest() - if s is not None - else None - for s in string_vals - ], - type=pa.large_binary(), - ) - table = table.set_column(col_idx, col_name, binary_vals) - return table diff --git a/src/orcapod/migrations/types.py b/src/orcapod/migrations/types.py deleted file mode 100644 index a717c0210..000000000 --- a/src/orcapod/migrations/types.py +++ /dev/null @@ -1,27 +0,0 @@ -"""Migration result types for Orcapod schema migrations.""" - -from __future__ import annotations - -from dataclasses import dataclass - - -@dataclass -class MigrationResult: - """Summary of a completed schema migration run. - - Attributes: - rows_total: Total rows found at the v0 path. - rows_migrated: Rows successfully written to the v1 path. - rows_skipped: Rows already present at the v1 path (idempotent re-run). - rows_unresolvable: Rows whose result data was unreachable (e.g. ephemeral - result expired); ``__output_data_hash`` written as ``None`` for these. - elapsed_s: Wall-clock seconds elapsed during the migration. - dry_run: ``True`` if the run was a dry run (no writes performed). - """ - - rows_total: int - rows_migrated: int - rows_skipped: int - rows_unresolvable: int - elapsed_s: float - dry_run: bool diff --git a/src/orcapod/pipeline/graph.py b/src/orcapod/pipeline/graph.py index 89471066d..28a86289e 100644 --- a/src/orcapod/pipeline/graph.py +++ b/src/orcapod/pipeline/graph.py @@ -6,6 +6,7 @@ from pathlib import Path from typing import TYPE_CHECKING, Any +from orcapod.contexts import resolve_saved_context from orcapod.core.nodes import ( FunctionNode, GraphNode, @@ -315,7 +316,9 @@ def load(cls, path: str | Path) -> "Pipeline": name=node_name, tag_schema=tag_schema, data_schema=data_schema, - data_context=descriptor.get("data_context_key"), + data_context=resolve_saved_context( + descriptor.get("data_context_key") + ), ) # Restore label from descriptor if set explicitly stored_label = descriptor.get("label") diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 854eab5ba..83a5bf264 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -19,12 +19,25 @@ from orcapod.core.nodes.function_node import _build_record_id_preimage from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER -from orcapod.errors import StoreError, StoreReadError, StoreWriteError +from orcapod.errors import ( + StoreError, + StoreReadError, + StoreWriteError, +) +from orcapod.contexts.stored import ( + account_for_stale_tables, + find_stale_tables, + require_same_data_context, + with_data_context, +) from orcapod.invocation import ( InvocationContext, InvocationHashConfig, ) -from orcapod.system_constants import TRACKING_DB_SCHEMA_VERSION +from orcapod.system_constants import ( + OLD_TRACKING_DB_SCHEMA_VERSIONS, + TRACKING_DB_SCHEMA_VERSION, +) from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -145,6 +158,7 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: pod_name=self._pod.label, run_id=None, arrow_hasher=self._pod.data_context.arrow_hasher, + data_context_key=self._pod.data_context_key, ctx_arg_name=self._pod.ctx_arg_name, ) if result is not None: @@ -189,6 +203,7 @@ def _execute_side_effect_row( pod_name: str, run_id: str | None, arrow_hasher: Any, + data_context_key: str, ctx_arg_name: str = "ctx", pipeline_database: ArrowDatabaseProtocol | None = None, table_path: tuple[str, ...] | None = None, @@ -217,12 +232,15 @@ def _execute_side_effect_row( pod_name: Label of the pod. run_id: Pipeline run identifier (or ``None`` in standalone mode). arrow_hasher: The ``arrow_hasher`` from the pod's data context. + data_context_key: Key of the pod's data context, stored on the log row. pipeline_database: Attached DB (or ``None`` for standalone mode). table_path: Path tuple for the invocation log table. Returns: ``(tag, data)`` to emit downstream, or ``None`` to drop the row. """ + require_same_data_context(data.data_context_key, data_context_key) + # 1. Build the preimage — delegates to the shared helper used by FunctionJobNode. # Appends the recomputation index (fixed at 0; side effects never recompute). preimage = _build_record_id_preimage(tag, data).append_column( @@ -268,6 +286,7 @@ def _execute_side_effect_row( record_id=record_id, record_id_hash_bytes=record_id_hash.to_prefixed_digest(), run_id=run_id, + data_context_key=data_context_key, ) return (tag, data) except StoreError as exc: @@ -292,6 +311,7 @@ def _write_invocation_row( record_id: bytes, record_id_hash_bytes: bytes, run_id: str | None, + data_context_key: str, ) -> None: """Write one success row to the side-effect invocation log table. @@ -307,6 +327,8 @@ def _write_invocation_row( record_id_hash_bytes: Binary prefixed digest of the record-ID hash (stored for human inspection via ``ContentHash.from_prefixed_digest``). run_id: Pipeline run identifier (or ``None`` for standalone mode). + data_context_key: Key of the data context whose hasher produced + ``record_id``; stored on the row as ``__data_context``. """ executed_at = datetime.datetime.now(datetime.timezone.utc) record = pa.table( @@ -323,7 +345,7 @@ def _write_invocation_row( pipeline_database.add_record( table_path, record_id, - record, + with_data_context(record, data_context_key), skip_duplicates=True, ) except StoreReadError: @@ -714,6 +736,7 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: pod_name=self._pod.label, run_id=None, arrow_hasher=self._pod.data_context.arrow_hasher, + data_context_key=self._pod.data_context_key, ctx_arg_name=self._pod.ctx_arg_name, pipeline_database=None, table_path=None, @@ -788,6 +811,45 @@ def __init__( ) self._pipeline_database: ArrowDatabaseProtocol | None = None self._table_path: tuple[str, ...] | None = None + self._log_contexts_checked = False + + def _refuse_log_of_another_data_context(self) -> None: + """Refuse an invocation log holding deliveries of another data context. + + A delivery already made is found by a record id hashed under this node's + data context, so deliveries logged under another one would not be found and + every side effect would be delivered again (ITL-944). Two cases: a log at an + older table layout (written by Orcapod up to v0.2), and rows of another data + context in the current log. Checked once per node instance and attached + database; the logs are only read. + + Raises: + SideEffectLogDataContextError: If such deliveries are found; nothing has + been delivered or written then. + """ + if ( + self._log_contexts_checked + or self._pipeline_database is None + or self._table_path is None + ): + return + current = self._pod.data_context_key + found = find_stale_tables( + self._pipeline_database, + tuple( + self._table_path[:-1] + (layout,) + for layout in OLD_TRACKING_DB_SCHEMA_VERSIONS + ), + current=(self._table_path, current), + ) + try: + account_for_stale_tables( + found, delivers_side_effects=True, current_context=current + ) + except StoreError as exc: + exc.add_context(node_label=self.label) + raise + self._log_contexts_checked = True # ------------------------------------------------------------------ # DB attachment @@ -808,6 +870,7 @@ def attach_databases( or ``None`` to detach. """ self._pipeline_database = pipeline_database + self._log_contexts_checked = False if pipeline_database is not None: self._table_path = self.node_uri + ( f"schema:{self.pipeline_hash().to_string()}", @@ -836,7 +899,12 @@ def execute( Returns: List of ``(tag, data)`` tuples — the pass-through rows. + + Raises: + SideEffectLogDataContextError: If the invocation log holds deliveries + logged under another data context; nothing is delivered then. """ + self._refuse_log_of_another_data_context() results = [] for tag, data in input_stream.iter_data(): result = _execute_side_effect_row( @@ -848,6 +916,7 @@ def execute( pod_name=self._pod.label, run_id=run_id, arrow_hasher=self._pod.data_context.arrow_hasher, + data_context_key=self._pod.data_context_key, ctx_arg_name=self._pod.ctx_arg_name, pipeline_database=self._pipeline_database, table_path=self._table_path, @@ -898,6 +967,7 @@ async def async_execute( ) try: + self._refuse_log_of_another_data_context() async def process_one(tag: TagProtocol, data: DataProtocol) -> None: try: @@ -910,6 +980,7 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: pod_name=self._pod.label, run_id=run_id, arrow_hasher=self._pod.data_context.arrow_hasher, + data_context_key=self._pod.data_context_key, ctx_arg_name=self._pod.ctx_arg_name, pipeline_database=self._pipeline_database, table_path=self._table_path, diff --git a/src/orcapod/system_constants.py b/src/orcapod/system_constants.py index 204dd9180..721dd7b7c 100644 --- a/src/orcapod/system_constants.py +++ b/src/orcapod/system_constants.py @@ -20,9 +20,24 @@ BLOCK_SEPARATOR = "::" ENV_INFO = "env_info" IS_EPHEMERAL_COL = "is_ephemeral" -PIPELINE_DB_SCHEMA_VERSION = "pdb_v1" -RESULT_DB_SCHEMA_VERSION = "rdb_v1" -TRACKING_DB_SCHEMA_VERSION = "tdb_v1" +# Table layout versions: the last component of every stored table's path. A layout +# version moves whenever the bookkeeping columns of that table kind change. v0.3 moved +# all of them, because every table gained the ``__data_context`` column (ITL-944). +PIPELINE_DB_SCHEMA_VERSION = "pdb_v2" +RESULT_DB_SCHEMA_VERSION = "rdb_v2" +TRACKING_DB_SCHEMA_VERSION = "tdb_v2" +OPERATOR_DB_SCHEMA_VERSION = "odb_v1" +SOURCE_DB_SCHEMA_VERSION = "sdb_v1" +# Layout components earlier versions of Orcapod wrote, newest first. A table found at +# one of them (or at the path without a layout component) is an old-layout table. +OLD_PIPELINE_DB_SCHEMA_VERSIONS = ("pdb_v1",) +OLD_RESULT_DB_SCHEMA_VERSIONS = ("rdb_v1",) +OLD_TRACKING_DB_SCHEMA_VERSIONS = ("tdb_v1",) +STORED_DATA_CONTEXT = "data_context" +# The data context of everything Orcapod wrote before v0.3. Its manifest is not shipped: +# stored rows that carry this key, and tables without a data context column, belong to it. +PRE_V1_DATA_CONTEXT_VERSION = "v0.1" +PRE_V1_DATA_CONTEXT_KEY = "std:v0.1:default" class SystemConstant: @@ -53,6 +68,11 @@ def SOURCE_PREFIX(self) -> str: def CONTEXT_KEY(self) -> str: return f"{self._global_prefix}{DATAGRAM_PREFIX}{DATA_CONTEXT_KEY}" + @property + def DATA_CONTEXT_COL(self) -> str: + """Column of every stored table naming the data context that wrote the row.""" + return f"{self._global_prefix}{SYSTEM_COLUMN_PREFIX}{STORED_DATA_CONTEXT}" + @property def POD_ID_PREFIX(self) -> str: return f"{self._global_prefix}{SYSTEM_COLUMN_PREFIX}{POD_ID_PREFIX}" diff --git a/src/orcapod/types.py b/src/orcapod/types.py index 01629608b..b302de6ac 100644 --- a/src/orcapod/types.py +++ b/src/orcapod/types.py @@ -361,15 +361,9 @@ class NodeConfig: Persistent cache hits are still served when available. Raises ``RuntimeError`` at execution time if ``True`` but no ephemeral store has been injected via ``set_ephemeral_store()``. - ignore_schema: Tuple of schema version strings that this node will - tolerate without raising ``SchemaVersionError``. ``None`` (default) - means no old schema is tolerated — any detected v0 table raises - ``SchemaVersionError``. Pass ``("v0",)`` to suppress the error - and allow the node to recompute all results from scratch. """ is_result_ephemeral: bool | None = None - ignore_schema: tuple[str, ...] | None = None def merge(self, other: "NodeConfig") -> "NodeConfig": """Return a new ``NodeConfig`` with ``other``'s non-``None`` fields overriding self. @@ -396,11 +390,6 @@ def merge(self, other: "NodeConfig") -> "NodeConfig": if other.is_result_ephemeral is not None else self.is_result_ephemeral ), - ignore_schema=( - other.ignore_schema - if other.ignore_schema is not None - else self.ignore_schema - ), ) diff --git a/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md new file mode 100644 index 000000000..50317251e --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md @@ -0,0 +1,249 @@ +# ITL-910 — Adopt starfix 0.5.0 (hash format 2) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Move Orcapod's Arrow hashing onto starfix 0.5.0 (hash format 2, prefix `00 00 02`), bump the +Arrow hash label `arrow_v0.1` → `arrow_v0.2`, and regenerate the golden hashes once, explicitly. + +**Architecture:** `StarfixArrowHasher` (`src/orcapod/hashing/arrow_hashers.py`) keeps calling +`ArrowDigester.hash_table` / `hash_schema` with the default (order-independent) options; only the +dependency, the label and the tests/docs/golden data change, plus whatever the new starfix rejections +require. No `ordered_columns` / `ordered_struct_fields` (developer decision 5). No legacy 0.4 mode. + +**Tech Stack:** starfix 0.5.0 (pure Python, `pyarrow>=19`), pyarrow 23.0.1 (locked; Orcapod floor already +`>=20`, so no pyarrow floor change), pytest, uv. + +Branch: `eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orcapod` (Linear gitBranchName). + +--- + +## Status log (for a successor) + +- 2026-11-11: investigation done (below). Plan sent to coordinator via `ask` — WAIT for the answer + before Task 4 (golden regeneration) and before any migration-related code. + +## Investigation results + +### starfix 0.5.0 API (installed wheel, README, `arrow_digester.py`) + +- Same entry points as 0.4: `ArrowDigester(schema, ...)`, `update`, `finalize`, `hash_schema`, + `hash_record_batch`, `hash_table`, `hash_array`. New keyword-only options `ordered_columns`, + `ordered_struct_fields` (not used here). `include_metadata` unchanged. +- Every hash starts `00 00 02`; all values differ from 0.4. +- `ValueError` (was `NotImplementedError` in 0.4) for data of Null/Map/Interval/Utf8View/BinaryView, + any row count; `ValueError` for ListView/Union/RunEndEncoded; new `ValueError`s: ambiguous node + paths (duplicate names, `"a/b"` aliasing struct `a` child `b`), non-nullable field with a visible null, + duplicate metadata keys with `include_metadata`. `TypeError` for wrong container type (Orcapod + converts RecordBatch → Table first, unaffected). +- FixedSizeList data now hashes (any depth), distinct from `List`/`LargeList` of the same values. +- Canonical extension types (`arrow.uuid`, `arrow.fixed_shape_tensor`) now hash. + +### Orcapod callers + +`grep -rn starfix src`: only `src/orcapod/hashing/arrow_hashers.py` (`hash_schema`, `hash_table`). +Everything else (sources' default `source_id`, record ids, `__input_data_hash`, base/pipeline entry +ids, OperatorNode record hashes, `Data`/`Tag` content hashes, `ArrowTableHandler` for `pa.Table`, +pandas DataFrame/Series handlers) goes through `StarfixArrowHasher`. + +### Behaviour changes visible to an Orcapod user (probe: `ArrowTableSource(t, tag_columns=["k"])`, +`hasher.hash_table`, `hasher.hash_schema`, 0.4 vs 0.5) + +| Input | 0.4 | 0.5 | +|---|---|---| +| `fixed_size_list` column (top level, in struct, in list, with null slot) | `NotImplementedError` (source fails) | hashes; source constructs | +| homogeneous `tuple[T, T]` pod output (maps to `fixed_size_list`) | fails to hash | works | +| `arrow.uuid`, `arrow.fixed_shape_tensor` extension columns | `NotImplementedError` | hashes | +| `hash_schema` with null / view / interval types | `NotImplementedError` | hashes | +| **zero-row** table with a `map` column | hashes | `ValueError` | +| non-nullable field with a visible null | hashes silently | `ValueError: Non-nullable field 'x' contains nulls` | +| duplicate column names (raw hasher, e.g. a `pa.Table` value) | hashes | `ValueError: Ambiguous node path` (sources already failed with `KeyError`) | +| column `"a/b"` next to struct `a{b}` | hashes | `ValueError: Ambiguous node path 'a/b'` | +| `map` (rows), `null`, view, interval data | `NotImplementedError` | `ValueError` (same rejection, new class) | +| `CachedSource._build_merged_stream` | `except NotImplementedError` silently served from cache when live ingestion hit an unhashable type | now raises (the hashing error is no longer mistaken for "source cannot provide data") | + +ITL-848 collisions through the public interface (`ArrowTableSource.iter_data()` row content hashes): +nested list lengths `[[[1]],[[2],[3]]]` vs `[[[1],[2]],[[3]]]` collide on 0.4, differ on 0.5; null struct vs +`{"a": None}` collide on 0.4, differ on 0.5; Bool column hash depends on chunking on 0.4, not on 0.5; +non-nullable `[True]` vs `[True, False, False]` collide on 0.4, differ on 0.5. Slice offsets are canonical +on both. + +### Where the label lives + +- `src/orcapod/hashing/versioned_hashers.py:40` `_CURRENT_ARROW_HASHER_ID` +- `src/orcapod/contexts/data/v0.1.json` `arrow_hasher._config.hasher_id` (+ changelog list) +- `src/orcapod/contexts/data/schemas/context_schema.json:178` (example value) +- tests: `tests/test_hashing/test_starfix_arrow_hasher.py` (label + three pinned format-1 digests), + `tests/test_hashing/test_pandas_handlers.py` (label), `tests/test_core/operators/test_materialize_source_info.py` + (three pinned format-1 digests), `tests/test_core/operators/test_async_sync_equivalence.py` + (`PINNED_ROW_HASHES`, `PINNED_STREAM_CONTENT_HASH`), + `tests/test_hashing/golden/cases.py` (docstring), golden data files. +- docs: `docs/reference/hashing.md` §3/§4 worked-example tables (regenerate with + `superpowers/scripts/hash_audit_example.py`); `notebooks/tutorials/01_introduction_to_orcapod.ipynb` + (outputs). Historical plans/specs under `superpowers/` are left as they are. +- Stored names: no table path embeds an Arrow hash under the default `table_scope="pipeline_hash"`. + With `table_scope="content_hash"`, the path embeds the node content hash, which moves when a source's + content hash moves. + +### Golden estimate (simulated: starfix 0.5 overlay + label patched to `arrow_v0.2`) + +| File | Entries | Changed | Unchanged | Keys added/removed | +|---|---|---|---|---| +| arrow.json | 53 | 53 | 0 | 0 | +| datagram.json | 171 | 40 | 131 | 0 | +| pipeline.json | 305 | 68 | 237 | 0 | +| schema.json | 92 | 22 | 70 | 0 | +| semantic.json | 64 | 1 (`value.dataframe.basic`, pandas via StarfixArrowHasher) | 63 | 0 | + +Every changed entry is an Arrow-derived value (row data hashes, record/entry ids, source content hashes +and default source ids, `pa.Table`/DataFrame hashes). No pipeline hash, schema hash, system-tag column +name or table path moved. + +### Migration finding (needs a decision) + +Default `table_scope` paths do not move, so a v0.3 job pointed at a v0.2 store writes into the same +tables. Observed with `DeltaTableDatabase` and the docs' `bmi_demo` job: v0.3 `run()` recomputes both rows +(old record ids miss), and a later read without `run()` returns **4 rows for 2 inputs** (v0.2 + v0.3 mixed); +before the first v0.3 run, a read returns the v0.2 rows. Stored rows carry the label +(`__input_data_hash`, `__pipeline_base_entry_id` start `b"arrow_v0.1:"` vs `b"arrow_v0.2:"`). OperatorNode +REPLAY reads all rows matching the node content hash, so a v0.3 LOG-mode write next to v0.2 rows can +duplicate too. + +### Full suite under starfix 0.5 overlay (label unchanged): 201 failed / 7400 passed / 96 skipped + +- 186 golden entries (all Arrow-derived, see above). +- Pinned format-1 digests: `test_starfix_arrow_hasher.py` (5 tests), `test_materialize_source_info.py` (1), + `test_async_sync_equivalence.py::test_issue_example_types_and_pinned_hashes`, + `test_struct_layout_identity.py::TestStructHashesUnchanged` (4 params). +- `test_group_by.py::test_falls_back_to_record_id_when_key_covers_all_tags`: member order comes from + record ids (uuid5 of the Arrow-hashed default `source_id`), so `["a","b"]` becomes `["b","a"]`. Arrow-derived. +- `test_array_v02_store.py` (2): the ITL-888 claim "an array-free pod reuses its v0.2 results" stops being + true: every function node recomputes (input data hashes move) into the same tables (mixing finding). +- `test_async_sync_equivalence.py::test_arrow_hasher_hash_schema_with_extension_type` (strict xfail, ITL-846) + now XPASSes: starfix 0.5 accepts a live extension type in `hash_schema`, but hashes it as its storage + type, so `hash_schema(Path column) == hash_schema(large_string column)`. +- Same identity loss in `hash_table` for pyarrow's canonical extension types (`arrow.uuid`, + `arrow.fixed_shape_tensor`): they are `pa.BaseExtensionType`, not `pa.ExtensionType`, so + `normalize_extension_columns` skips them, `include_metadata` stays off, and an `arrow.uuid` table hashes + equal to a `fixed_size_binary(16)` table. (On 0.4 they raised.) + +## Tasks (after approval) + +### Task 1: failing tests first +- [ ] New `tests/test_hashing/test_starfix_format2.py` (public interface: `ArrowTableSource`, `iter_data`, + `content_hash`; the hasher itself via `resolve_context(None).arrow_hasher` for table-level checks): + label `arrow_v0.2` and digest prefix `00 00 02`; the four ITL-848 cases above; Bool chunking; slice/offset + and `combine_chunks` canonicality; `fixed_size_list` column hashes and differs from the same values as + `large_list`; `ArrowTableSource` with `fixed_size_list` columns constructs, keeps the type, and runs through + a function pod; pod returning `tuple[float, float]`; rejections with `ValueError` (zero-row map, + non-nullable null, duplicate names via `pa.Table` value hash, `"a/b"` alias); `arrow.uuid` column hashes. +- [ ] Update `test_starfix_arrow_hasher.py` / `test_pandas_handlers.py` label assertions to `arrow_v0.2`. +- [ ] Run, confirm failures on starfix 0.4. WIP commit. + +### Task 2: bump +- [ ] `pyproject.toml`: `starfix>=0.5.0,<0.6` (decision 6 floor); `uv lock`; report what moved in `uv.lock`. +- [ ] Label: `versioned_hashers.py`, `v0.1.json` (+ changelog line: "Hard cut (v0.3): arrow_v0.2 = starfix + hash format 2 (starfix 0.5) …"), `context_schema.json` example. +- [ ] `StarfixArrowHasher` docstring: format 2, prefix `00 00 02`, rejections. +- [ ] Re-pin the format-1 digests in `test_starfix_arrow_hasher.py` and `test_materialize_source_info.py` + (list old/new in the PR). +- [ ] Remove the generator header's "Known gaps (ITL-848)" items that are now hashable (fixed-size list, + `arrow.uuid`); add golden cases for them (adding entries is allowed). + +### Task 3: migration handling — per coordinator decision (see ask) + +### Task 4: regenerate golden (approved regeneration) +- [ ] `uv run python tests/test_hashing/golden/generate_golden_hashes.py`; compare with + `scratchpad/scripts/cmp.py` against main's files; every changed value must be Arrow-derived. + +### Task 5: docs +- [ ] CHANGELOG `[0.3.0] — unreleased` → Breaking changes and migration: starfix format 2 entry. +- [ ] `docs/reference/hashing.md` worked-example tables, prefix note; notebook outputs. +- [ ] DESIGN_ISSUES: one appended entry (resolved, Fix note). + +### Task 6: verification +- [ ] Full suite on 3.11, 3.12, 3.13, one at a time; ruff check/format; basedpyright exit code. +- [ ] Perf: 100k-row table hash, 0.4 vs 0.5; golden-test wall time before/after. + +### Task 7: PR +- [ ] Merge origin/main, push, PR (`Fixes ITL-910`, mentions ITL-848, ITL-927), CodeRabbit rounds. + +## D1 = D (developer, 2026-11-11): fail explicitly on rows hashed under another label + +Design (tests first): +- New public `orcapod.errors.StoreHashLabelError(StoreError)` (always fails the run, whatever + `error_policy`): attributes `record_path`, `found_label`, `current_label`; message names the table, + both labels and the remedy (fresh store, or keep the version that wrote it; migration tool planned). + No opt-out (`ignore_schema` does not disable it). +- Helpers in `src/orcapod/hashing/stored_hash_labels.py` return what they find (no raising): + `foreign_arrow_label(values, current_label)` for `b"