diff --git a/CHANGELOG.md b/CHANGELOG.md index 878a6247b..ed235c033 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,308 @@ ### Breaking changes and migration +#### Upgrading from 0.2.0 + +`0.3.0` is the first release on PyPI since `0.1.4`; `0.2.0` was a tag only. This section lists +every change a `0.2.0` user meets, grouped, each with what to do. The entries below it give +the details. + +**Coming from `0.1.4`** (the previous PyPI release): also read the API changes in the +`## [0.2.0]` section, but skip its database migration steps. `orcapod migrate`, +`migrate_node` and `ignore_schema` are removed in 0.3, and a table without a layout +component (as `0.1.4` wrote) is counted and reported like any other old table, never read +(ITL-944). A `0.1.4` database is therefore treated exactly like a `0.2.0` one, as described +below. + +**Stored data: nothing written by 0.2.0 is reused.** + +- Every hash moved once in v0.3 (hash labels, ITL-945; starfix hash format 2, ITL-910; + logical type identifiers, ITL-935; user classes, ITL-972; NumPy arrays, ITL-981; pandas + values, ITL-897; the v1 hash definition, ITL-1154), and every table has a new layout and a + `__data_context` column (ITL-944). + The tables `0.2.0` wrote are not read as results and are never modified. Given the same + database, the first v0.3 run computes every input again and writes new tables beside the + old ones, at new paths. Each node reports the old tables it finds, by name (ITL-954). To do: nothing, beyond accepting one full recomputation; keep `0.2.0` + to read the old tables. A migration tool is planned after v0.3 (ITL-913, ITL-914, ITL-1048). +- A database written by a development build of v0.3 (installed from git before `0.3.0`) is + discarded, not migrated: every v0.3 hash was defined anew before the release, and the + record-id column is now text (ITL-1154). Its results are computed again. **A side-effect + node run by such a build delivers every input again**, and so does one given a new + database. To do: run such a node only where a repeated delivery is acceptable. +- Side-effecting nodes (`@side_effect_pod`, `@sink_pod`, `@tap_pod`, + `@function_pod(ctx_arg=...)`, `@side_effect_function_pod`) that find `0.2.0` tables raise + `SideEffectLogDataContextError` before delivering anything (ITL-944, ITL-954). Keep `0.2.0` + for such a node; a fresh database performs every side effect again. +- A job file saved by `0.2.0` with a database names it under `bindings.store`, and + `PipelineJob.load(path)` raises `ValueError` for it: pass `database=` to `load`, or rename + the key to `bindings.database` (ITL-1082). Once a `0.2.0` job or pipeline file loads, its + nodes are under the current data context, with a warning, and show no rows until the job + runs (ITL-944, ITL-945). A saved file + whose schema text names a logical type by name only is rejected: save a new job from the + original pipeline (ITL-904). `0.2.0` cannot load a file saved by `0.3.0` (ITL-1026). +- `std:v0.1:default` no longer resolves (`ContextResolutionError`), and a full data context + key must match its manifest (ITL-944). +- Removed with no replacement: `SchemaVersionError`, `NodeConfig(ignore_schema=...)`, + `orcapod.migrations` and the `orcapod migrate` command (ITL-944). Remove `ignore_schema` + from your code. +- `CachedSource(cache_path=...)` stores under `cache_path + ("sdb_v1",)`; pass the path + without the layout component (ITL-944). +- From `0.3.0` on, released hash values are protected by CI: a golden hash file a release + listed never changes, and new golden values go into new files, in any release. A change to + a released golden value, a logical type's own identifier change included, needs a new data + context version, shipped only in a minor (before 1.0) or major release (ITL-946; see + `docs/reference/golden-hashes.md`). To do: nothing; hashes written by `0.3.0` stay valid + for every release of the `std:v1:default` data context. + +**Requirements.** + +- `pyarrow>=22.0.0` (was 20.0.0) and `pandas<3` (ITL-897); `starfix>=0.5.0,<0.6` (ITL-910); + the `spiraldb` extra needs `pyspiral>=0.16.33` (ITL-926). + +**Renamed or removed names (no aliases).** + +- `store` is `database` everywhere: `PipelineJob(database=...)`, `from_pipeline`, `bind`, + `load` and `job.database` (ITL-1082); `set_ephemeral_store` is `set_ephemeral_database`; + `StoreError`, `StoreReadError`, `StoreWriteError` and `StoreContentWarning` are + `OrcapodDatabaseError`, `OrcapodDatabaseReadError`, `OrcapodDatabaseWriteError` and + `OrcapodDatabaseContentWarning`; `orcapod.utils.store_errors` is + `orcapod.utils.database_errors`; `describe_store_table` is `describe_database_table` + (ITL-1087). A job file with `bindings.store` raises `ValueError` on load: pass + `database=` or rename the key (ITL-1082). `PipelineJob.set_ephemeral_database(...)` now + works on a job with a side-effect node, which it skips (it raised `AttributeError`; + ITL-1147). +- What `head()` returns is `StreamView` (was `StreamHead`), with `ViewStatus` (was + `HeadStatus`) and `StreamViewProtocol`, in `orcapod.core.streams.view` (ITL-1081). The old + `StreamBase.view()` is removed: use `head(n)` or `as_polars_df()` (ITL-991). +- `resolve_field` and `FieldNotResolvableError` are removed; read values through the source's + stream (ITL-1078). +- Hash labels are `_v1`: `semantic_v1`, `arrow_v1`, `directory_v1` (ITL-945). A + hash inside a name (function `uri[1]`, `schema:`, `node:` path components) is the bare + digest: compare with `to_hex()`, not `to_string()` (ITL-945). + +**Running a job.** + +- `PipelineJob.run()` returns a `RunSummary`, not the job: write `job.run()`, not + `job = job.run()` (ITL-997). +- A job node shows the current run only. Before a run in this process it is empty; read what + is stored with `node.as_source()` (or `get_all_records()`), or run the job (ITL-1095). +- `run()` is state-driven: a node that holds a current run is not run again, so running an + unchanged job a second time computes, reads and delivers nothing. A source whose data + changes in memory needs a new source object, bound with `job.bind(sources=...)`; call + `clear_cache()` on a node to run it again (ITL-1095). Reading or running a job's + side-effect node no longer delivers every row again: deliver through `job.run()`; to + repeat the deliveries of a side-effect pod declared `track_completion=False`, call + `clear_cache()` on its node first (ITL-1091, ITL-1095). +- An operator stream's `run()` takes no arguments; job nodes' `run()` returns a `StreamView` + (ITL-1081). +- Displaying a stream, source or node in a notebook never computes it: call `as_table()` or + run the job (ITL-991). +- A database write or read failure fails the run whatever the `error_policy` (ITL-878, + ITL-884). The async orchestrator honours `error_policy="fail_fast"` and raises the failing + node's own exception, never an `ExceptionGroup`: replace `except*` / `except ExceptionGroup` + with `except SomeError` (ITL-879). +- Appending rows whose column type differs from a stored Delta table's raises + `InputValidationError` instead of casting (ITL-884). +- A connector database (`ConnectorArrowDatabase` over SQLite, PostgreSQL or SpiralDB) cannot + hold a function node in `0.3.0`: the job refuses the node when it is built, naming it, + with `UnstorableColumnError`, where it used to run the function and fail at commit + (ITL-1162, ITL-1164). Use `DeltaTableDatabase` or `InMemoryArrowDatabase` for a job's + database. + +**Operators.** + +- `GroupBy` keeps only its `by` tags; a non-key tag is dropped, no longer turned into a data + column. Offer the tag as data before grouping + (`DerivedSource(origin, tags_as_data={...})`), or drop the parameter that read it + (ITL-1070). +- `SemiJoin` matches on the tag columns both inputs share; shared data columns are ignored, + with a `SemiJoinIgnoredColumnWarning`. Rename or drop the column on the right input to + silence it (ITL-1088, ITL-829). +- The fluent `map_tags` / `map_data` keep unmapped columns by default; pass + `drop_unmapped=True` for the old behaviour (ITL-1027). The renames dict is their only + positional argument: `drop_unmapped`, `strict` and `label` are keyword-only, so + `stream.map_tags(names, True)` raises `TypeError`; and `map_data(name_map={...})` no + longer names the dict: pass it by position (ITL-1160, ITL-1171). +- `map_tags` / `map_data` refuse a name that is not a column of the kind they rename, with + `InputValidationError` (for example `map_tags` of a column that became data after + `group_by`); it used to rename nothing. Fix the name, use the other method or the new + `map_columns`, or pass `strict=False` (ITL-1160). A new name a stream would read as a + system column (`_tag_...`, `_source_...`, `_context_key`) is refused (ITL-1160). +- A name that is a tag in one input and data in another is refused when the graph is built + (`Join`, `MergeJoin`, `Pick`/`Index` `out=`, a pod output named like an input tag); rename + with `map_tags` or `map_data` (ITL-940). A tag column named with the `__` prefix is refused + (ITL-1080). +- Column selections are identified by the set of names: a pipeline that spelled them as a + tuple, a set, or an unsorted list gets new tables (ITL-942). + +**Function pods.** + +- `pod(a, b)` now drops the data columns the function does not take (an automatic + `SelectDataColumns` after the join). Use `pod(a, b, strict=True)` or `pod.process(...)` to + keep rejecting extra columns (ITL-807). +- A function's return value must match its declared output type; a mismatch raises + `OutputTypeError` (numeric narrowing, float for int, a set for a list, a plain tensor for + `-> torch.nn.Parameter`). Convert explicitly in the function (ITL-895, ITL-871). +- A default parameter value the hasher cannot hash (an `Enum` member, a dataclass instance, + `Path`, `datetime`, `Decimal`) raises `TypeError` when the pod is created: use a hashable + default or supply the value as a column (ITL-953). +- `FunctionPod.from_fn` and `side_effect_function_pod` default to version `v0.0` (was + `v1.0`); pass `version="v1.0"` to keep `/v1/` in paths (ITL-942). +- An ordinary `@dataclass` or pydantic model is identified by its exact class + (`module.Class`): a class defined again (a re-run notebook cell) is a different type. Use + `orcapod.dataclass` for a class identified by its name and fields. Changing a class's + fields makes a new type with new tables; a recursive class raises `RecursiveTypeError` + (ITL-972). +- A `set[T]` of a primitive (`set[int]`, `set[str]`) is stored as the native set container + (Arrow extension type `set[int64]`) and reads back as a `set` on every path: a downstream + pod, `node.as_source()`, `DerivedSource` and a job loaded in another process, where it + read as a `list` before (ITL-611). `list[T]` is unchanged. + +**Sources and binding.** + +- Binding a source to a job slot checks column types, not just names; a different type, or a + nullable column for a non-null slot, raises `SourceSpecMismatchError`. A joined or merged + stream, or one without system tags, is refused too: wrap it with + `ArrowTableSource(stream.as_table(), tag_columns=[...])` (ITL-1112). +- Every table has large Arrow types (`large_string`, `large_binary`, `large_list`): compare + with `pa.large_string()`, not `pa.string()` (ITL-1122). +- In-memory sources without a `source_id` are named `:arrow_v1:` + (`dict:arrow_v1:...`), and `node.as_source()` is `derived:semantic_v1:`. Pass an + explicit `source_id` to keep a name stable (ITL-957). +- Default source ids of file and table sources name the kind of source: + `csv:`, `delta_table:` (was the folder name), + `sqlite_table::main.`, `postgresql_table:..
`, + `spiraldb_table:..
@
`, and + `polling_source:semantic_v1:` (ITL-1154, ITL-1157, ITL-926). A `DBTableSource` used + directly over a connector that gives no database address requires a `source_id`. A source + id or column name holding `::`, and an empty source id, are refused. Pass `source_id=` to + keep an earlier id. +- `record_id_column` is `record_id_columns` (one name or several, in key order; `()` for the + row position) on every source (ITL-1154). A database or SpiralDB table identifies its rows + by its primary key by default; a database table without one is refused unless + `record_id_columns` is given (a SQLite table that relied on `rowid` passes + `record_id_columns="rowid"`); a `PollingSource` requires it (ITL-1158). A key value must be + a string, an integer, a boolean, a timestamp or a date; a null or repeated key is refused + when the source is built. +- The record-id system tag holds the row identifier as text (`row_`, or the canonical JSON + of the key columns), no longer a `uuid5`, and a source-info token is + `::::` (ITL-1154). Update code that parsed these values. +- `SpiralDBTableSource` reads no rows when it is built; a table without a key schema is + refused; reads are capped by `max_rows` (ITL-926, ITL-186). +- Every source built from a table takes the keyword-only option `tag_only_columns`. In + `0.3.0` every tag is plain (a tag only), and the one accepted value is `True`, the default; + any other value raises `ValueError`. In v0.4 a declared tag becomes referential by default + (also the data column of the same name): to keep today's behaviour then, pass + `tag_only_columns=True` now. Hashes, source ids and saved files are unchanged by it + (ITL-1159). + +**Values that are refused now.** + +- A `None` inside a container where a read could not give it back (`list[list[int] | None]`, + `list[Path | None]`, a field `x: UUID | None`) raises `ValueError` at declaration; a source + given a `None` its declared type does not allow raises `UnsupportedValueError` (ITL-1144). + Declare the inner type without `| None`, or wrap it in a dataclass. +- `frozenset[T]` is not supported: declare `set[T]` (ITL-611). +- A logical type or native container as a dict key or value, at any depth + (`dict[str, set[int]]`, `dict[str, Path]`, `list[dict[str, set[int]]]`), raises + `NestedTypeNotStorableError` (a `ValueError`) where it is declared: a pod's parameter or + output, a source schema, a dataclass or pydantic field. It was stored as a plain list and + read back as one. Hold the value in a dataclass field, or declare the plain type + (`dict[str, list[int]]`) (ITL-1150). +- A dict or set value is stored in canonical order; a NaN key or element, keys or elements + that cannot be compared, or two keys that convert to one stored value raise + `CanonicalOrderError` naming the column (ITL-905). +- NumPy arrays: masked arrays, object dtypes, `StringDType`, `longdouble`, unit-less + datetimes, field titles, overlapping structured fields and very large headers are refused + at write (ITL-888, ITL-981). +- pandas values storage would change or not read back are refused at write, with + `ValueError` (ITL-1054, ITL-897). +- An Arrow table with a null in a non-nullable field, a zero-row `map` column, or two + columns at one path (duplicate names) raises `ValueError` naming the column; `null`, + `map`, interval, view, union and run-end-encoded data raise `ValueError` instead of + `NotImplementedError` (ITL-910). + +**For authors of extensions.** + +- A custom Arrow hasher must implement `ArrowHasherProtocol.hash_table_ordered` (ITL-897); a + custom logical type not derived from `BaseLogicalType` must implement + `readable_extension_metadata()` (ITL-935); a type handler that returned a plain + `{"__type__": ...}` dict returns `Encoded(tag, fields)` (ITL-942). + `StarfixArrowHasher.hash_table_ordered` returns the label `arrow_ordered_v1`, and a nested + hash enters another hash with its label (ITL-1154). +- An operator's `async_execute` requires `input_pipeline_hashes` (ITL-906), and + `FunctionNodeProtocol.async_execute` takes a keyword-only `error_policy` (ITL-879). + +**Documentation.** + +- The tutorial notebooks in `notebooks/tutorials/` are removed: follow the tutorial in + `docs/tutorial/` (eleven short parts, with exercises and solutions), and see the new pages + *Pipelines and Jobs* and *File and Path Types* (PLT-4864). + +#### Other breaking changes + +- `AsyncPipelineOrchestrator(error_policy="fail_fast")` now stops the run at the first row whose function raises and raises that row's own exception, as `SyncPipelineOrchestrator` does (ITL-879). Before, the async orchestrator ignored the policy: the run skipped the row, computed the others and returned. An async run with `fail_fast` that used to finish now raises; pass `error_policy="continue"` (the default) to keep the old behaviour. Rows run concurrently, so the row raised is the first to fail, which is the first in input order only with `max_concurrency=1`. Every stored row is a correct result of its input row and the failing row is never stored; which other rows are stored before the run stops is not specified (see `docs/concepts/function-pods.md`, "Stopping at the first failed row"). +- An async run raises the failing node's own exception, never the task group's `ExceptionGroup` (ITL-879). This covers a side-effect pod with `on_error="raise"`, an operator that rejects its inputs, and any other node failure, as store errors already were. When several nodes fail before the run stops, one exception is raised and the others are added to it as notes (`error.__notes__`): a store error (`StoreError`) first, then a stored row that cannot be decoded as its class (`TypeIdentityError`), as before; among the other failures, the one of the node that comes first in the pipeline. Replace `except* SomeError` or `except ExceptionGroup` around `job.run()` / `job.run_async()` / `AsyncPipelineOrchestrator.run()` with `except SomeError`. +- `FunctionNodeProtocol.async_execute`, `FunctionJobNode.async_execute` and the function pods' `async_execute` take a keyword-only `error_policy` (default `"continue"`), as `execute` already did. A class implementing `FunctionNodeProtocol` must accept it. +- Saved pipeline/job schema text now includes the full logical type identifier as + `extension`, including nested logical types. Loading validates it + against the registered type; incompatible or malformed identifiers raise + `LogicalTypeIdentifierError` naming the node and column. Name-only extension text from + older saved files is rejected with an earlier-version error. Read those files with the + version that wrote them, and save a new job from the original pipeline under this version. + Stored table layouts, value decoding and hashes are unchanged. Unregistered logical types + are reported by node and column; affected operators load as read-only stubs. (ITL-904) +- **What a v0.2 store looks like to v0.3: old tables are found by name, not by hash.** v0.3 + computes other hashes than v0.2, so a v0.2 table is at a path v0.3 does not compute. Each + function and side-effect node lists the tables stored under its job and function or pod + name (`//`, `/_result//`, + `/side_effect_function//`, `/side_effects//`) and treats + every table that is not at a current layout as written by an earlier version. Function + nodes report them in one warning per node (no number of rows: the tables are not + read); side-effecting nodes raise `SideEffectLogDataContextError`. A job run adds one + total of the old tables that none of its nodes reports: operator caches, cached sources + stored under the job, and nodes that left the job. Because tables are found by name, an + old table of another function or pod with the same name in the same job, or of another + node of the same function, counts too; the warning and the error name the tables. + (ITL-954) +- A store that cannot list its tables (a Delta store on `gs://` or `az://` without the + matching `fsspec` package, a third-party database without `list_table_paths`) is not + searched by name. A side-effecting node then logs one warning that it could not check + for completions logged before v0.3, and runs. (ITL-954) +- **`PipelineJob.run()` returns a `RunSummary`, not the job.** Results are read from the + job itself, as before (`job.nodes[...]`). Code written `job = job.run()` becomes + `job.run()`; code that kept the return value as the job (`completed = job.run()`, + `completed.nodes[...]`) uses the job instead. `run_async()` returns the same summary. + (ITL-997) +- The `content_hash` of a `DerivedSource` built with `tags_as_data` now includes the mapping (ITL-1000). Before, two mappings that gave the same schema (two tag columns of one type copied to one name) and shared an explicit `source_id` had the same `content_hash` although the copy held different values; the sorted mapping items are now part of the identity. The `content_hash` moves whenever `tags_as_data` is given, with the default `source_id` as well as an explicit one, because the mapping is appended in both cases. The `source_id` (the default id was already built from the mapping), record IDs, source-info tokens and `pipeline_hash` do not move, and a `DerivedSource` without `tags_as_data` (a job node's stored records) is unchanged. `tags_as_data` was added in 0.3 and has not been released, so no released store holds a hash that moves. A store written by an unreleased 0.3 build with a `tags_as_data` `DerivedSource` gives that source, and the nodes downstream of it, a new node identity: those pipeline records are recomputed, and the old rows stay in place. The golden values `schema:derived_source_tags_as_data.{default_source_id,named_source_id,record_id_column}.content_hash` changed with it. +- `resolve_field(record_id, field_name)` is removed from `SourceProtocol`, `RootSource` and `SourceProxy`, and `orcapod.errors.FieldNotResolvableError` is removed with it. The method was added with `RootSource` (February 27, 2026) and its only implementation was dropped on March 13, 2026; since then every source raised `NotImplementedError`, so it never worked on a shipped source and nothing called it. There is no replacement: read a source's values through its stream (`as_table()`, `iter_data()`). +- `PipelineJob` takes its database as `database`, not `store`: `PipelineJob(name=..., database=db)`, `PipelineJob.from_pipeline(..., database=db)`, `job.bind(database=db)`, `PipelineJob.load(path, database=db)` and the `job.database` property. There is no `store` alias. Migrate with `store=` to `database=` on those calls and `job.store` to `job.database`. The saved job file writes the database config under `bindings.database` (was `bindings.store`); the format version `orcapod_pipeline_job_version` stays `0.2.0`, a version no release has shipped. `PipelineJob.load(path)` of a file that still has `bindings.store` raises `ValueError` naming both keys instead of loading without its database: pass `database=` to `load` (it overrides the saved block, so such a file then loads), or rename the key in the file and save it again. The old key is never read. +- `run()` on a job's side-effect node no longer delivers every side effect again on every call (ITL-1091). It does the work `job.run()` does for that node: with `track_completion=True` (the default) it delivers only the rows whose delivery the invocation log does not record, and with `track_completion=False` it delivers every row again, as a second `job.run()` does. Code that called `node.run()` or `node.as_table()` on a job's side-effect node to deliver again must call `job.run()` (with `track_completion=False` to repeat deliveries) or use the side-effect pod outside a job, whose stream still delivers on every read. +- A job node shows the current run; `node.as_source()` shows everything stored for it (developer, November 13, 2026; ITL-1095). Before a run in this process, a function or operator job node is empty: `as_table()`, `iter_data()`, the dataframe methods, `flow()`, `head()` and the notebook preview return zero rows with the node's columns, and `head()` (status `not_computed`) says the node has not run and points to `run()` and to `node.as_source()`. Before, a function node built over a store that already held results, or loaded with `PipelineJob.load`, returned every row stored under its key, and an operator node in `CacheMode.REPLAY` returned every logged run. The same holds after `clear_cache()`, after a run that raised, and while the node is stale. Migration: read stored results with `node.as_source()` (or `get_all_records()`), or run the job first; a run over a store that holds every result computes nothing. +- A function or operator job node that has not run in this process is stale (`is_stale` is `True`; it has no modified time), and a job node is stale also when any function or operator job node it depends on is. Binding a source to a job's source slot (`job.bind(sources=...)`) marks the slot modified, so every node fed by it is stale and empty until it runs again. +- A loaded operator node no longer raises from `as_table()` or `iter_data()`: it is empty, like every node that has not run. Its stored view (`as_source()`, `get_all_records()`) raises instead when the job file does not record its run's content hash and the store holds nothing under the hash the loaded job computes (ITL-1043). +- A job node's `run()` is state-driven and runs what the node depends on (developer, November 13, 2026; ITL-1095). A node that holds a completed run and is not stale runs nothing: `run()` returns the rows it holds. Otherwise every job node it depends on that holds no current run runs first, each once, as `job.run()` would run that part of the job, then the node; the run's writes are committed when it ends (before, `node.run()` read an upstream node's stored rows, or nothing, and left its last pipeline record and a `LOG` operator's output uncommitted). `run()` of a node whose function pod is a placeholder, or that was loaded without one, raises unless `load_only=True`; a `CACHE_ONLY` node's `run()` raises (its input is unavailable): read its stored results with `as_source()`. A job node's `run()` takes one keyword-only argument, `load_only`; every other stream's `run()` still takes none. +- A side-effect job node follows the same rule: it is stale before a run, and while it holds a current run neither its `run()`, nor a downstream node's `run()`, nor `job.run()` delivers anything; to deliver again without `track_completion`, call `clear_cache()` on it first. A load-only run passes its rows on and delivers nothing. +- `job.run()` and `job.run_async()` follow the same rule as `node.run()` (developer, November 14, 2026): a node that holds a current run is not run again, and its rows feed the nodes after it. Running an unchanged job a second time now computes nothing, reads and writes nothing in the store, reads no source rows to process them and delivers no side effect, even without `track_completion` (before, every `job.run()` re-read every source and re-processed every node, and delivered every untracked side effect again). Nodes run when they have not run in this process, were cleared, are below a cleared node or a node that ran again, or below a source bound again or whose own modified time moved. A re-run notices new data in a source whose object is unchanged through the source's start-of-run check (ITL-1102); a source that cannot see its data change (one built over data held in memory: a `DictSource` copies the list it is given) needs a new source object, bound with `job.bind(sources=...)`. The summary reports a node that was not run as held: "rowwise: 6 rows: held from the previous run" (`NodeRunSummary.held`). If the commit at the end of a run fails, the nodes it ran are cleared and the nodes it held keep their rows. +- If a run raises or is interrupted, or the commit at the end of `job.run()` or `node.run()` fails, every node the run ran is cleared (not run, stale) before the error propagates; nodes it held keep their rows. A run that left failed rows (`error_policy="continue"`) is never current: the next run computes only the failed rows and reads the rest from the store. +- Every table Orcapod presents or stores has the large Arrow variants of string, binary and list types (`large_string`, `large_binary`, `large_list`, recursively), with no exception (ITL-1122). A `string`, `binary` or `list` column of an Arrow table you give `ArrowTableSource` (or a raw `ArrowTableStream`), or of an Arrow schema you pass as `schema=` to `CSVSource` or `DataFrameSource` or as `data_schema=` to `DictSource`, now comes back as `large_string`, `large_binary` or `large_list`. Names, nullability, widths and logical types are kept. Code that compared an Arrow type to `pa.string()` compares it to `pa.large_string()` now. No hash moves: large and non-large variants hash identically, so row hashes, content hashes, pipeline hashes, record ids and default source ids stay as they were, and a Delta table written with either variant accepts rows of the other. +- A `None` inside a container is refused where a read could not give it back (ITL-1144). A list or set element, a dict value or a class field declared `X | None`, where `X` is a container (`list`, `set`, `dict`) or a logical type other than your own class (`Path`, `UPath`, `UUID`, `numpy.ndarray`, `torch.Tensor`, pandas and Polars values), now raises `ValueError` when the type is declared, as a function's parameter or return value, a source's `data_schema`, or a field of a dataclass or pydantic model. Examples: `list[set[int] | None]`, `list[list[int] | None]`, `dict[str, list[int] | None]`, `list[Path | None]`, `set[UUID | None]`, a field `x: UUID | None` or `x: list[int] | None`. The error names the column (or `Class.field`), the type and ITL-1144. Such a `None` was stored correctly, but read back as an empty container (`[]`, `set()`, `{}`), with a different row hash, or the read failed with a `TypeError`: through `as_source()`, a `DictSource`, a loaded job and a downstream function. An in-memory pod-to-pod chain that never stored or re-read the values did keep the `None`; it is refused too. Declare the inner type without `| None`, or hold the value in a dataclass and make that optional (`list[MyData | None]`). Still accepted, unchanged: a `None` inside a container where the inner type is a scalar (`list[int | None]`, `dict[str, float | None]`, a field `x: int | None`) or your own class (`list[MyData | None]`), and a `None` at the top level of any column (`list[Path] | None`). No hash, stored table or golden value changes. +- Reading a stored column that holds such a `None` raises `ValueError` naming the column, instead of returning a wrong value or a `TypeError` (ITL-1144). A null key/value entry of a stored dict raises too, instead of being dropped. This covers an Arrow table given to `ArrowTableSource` with a null inner list or struct, and `GroupBy` or `Batch` over a nullable list, set, `Path` or `UUID` column, whose output rows read as Python values now raise. `GroupBy` and `Batch` over a nullable scalar column (`list[int | None]`) are unchanged. Drop the nulls first (`DropNulls` on that column) or keep the value in a dataclass. +- A source given a `None` where its declared type does not allow one now refuses instead of storing an empty container or a wrong value (ITL-1144). A `None` for a list, set or dict whose declared type is not `| None`, inside a column's value (`[None, [1]]` for `list[list[int]]`, `{"k": None}` for `dict[str, list[int]]`, a dataclass field `xs: list[int]` given `None`) or as the column's own value, was stored as `[]`, `set()` or `{}`. A `None` for a logical type declared without `| None` reached the type's encoder: a `Path` was stored as the string `'None'` and read back as `Path('None')`, a `list[UUID]` as `[]`, and a `UUID` failed with an `AttributeError`; a user class raised `TypeError` and now raises the same `UnsupportedValueError`. `DictSource`, `ListSource` and any other source built from Python values now raise `UnsupportedValueError` (a `ValueError`) naming the column and the declared type. A function output already refused such a value (`OutputTypeError`). A `None` at a position declared `| None` is unchanged, and no hash, stored table or golden value of a valid value changes. +- Every v0.3 hash is defined anew before its first release by one rule: a hash always carries its label, except where it is written into a name, whose grammar fixes which algorithm produced it (ITL-1154, revising ITL-945). A nested hash enters a structure hash as `{"__type__": "hash", "label": ..., "digest": ...}`; a value-hashed cell of a logical-type column enters a table hash as `::
`, `postgresql_table:..
`, `spiraldb_table:..
@
` (ITL-1157). A `DBTableSource` used directly, over a connector that gives no database address, has no default: it requires a `source_id`, because the table name alone does not say which database the table is in. A source id or a column name holding `::` is refused, and so is an empty source id. +- A root source's content hash includes its key columns, the resolved `record_id_columns` in key order (`()` for the position; the primary key of a database or SpiralDB table by default), so two sources with one `source_id` that name their rows differently are two sources: they no longer share a cache (`CachedSource`) or a node's stored records. The pipeline hash, the source id (given or default), the record ids and the source-info tokens are unchanged. This applies to every root source, including `ListSource`, `PollingSource`, `SpiralDBTableSource` and `DerivedSource` with `tags_as_data`. +- A store written by a development build of v0.3 is discarded, not migrated: the v0.3 table layouts were redefined before their first release (the record-id column is now text). +- These change 849 golden values of data context `v1`, which no release has shipped: v0.3.0 is its first release. Nothing needs migrating beyond what v0.3 already requires: data written by v0.1.4 or v0.2 is already stale under v0.3, and data written by a development build of v0.3 is recomputed (a moved entry id or path is not found; nothing is lost or read wrongly). **Side effects are delivered again:** a side-effect node finds its completed deliveries by entry id, so deliveries logged by a development build of v0.3 are not recognized and a run delivers them again. A new store does not avoid this: it holds no log at all, so a run delivers every input again too. Run a side-effect node that was run by a development build of v0.3 only where a repeated delivery is acceptable. +- `map_tags` / `MapTags` and `map_data` / `MapData` refuse, by default, a mapping key that is not a column of the kind they rename (ITL-1160). Before, such a key was accepted and renamed nothing, which hid mistakes such as `map_tags` of a column that had become data after `group_by`. The call now raises `InputValidationError` naming the missing names and the columns of that kind; a name that is a column of the other kind says so (`'age' is a data column: use map_data`). The check runs when the operator is called on a stream, before any row is processed, and on the async path against the first row. A raw stream's meta (`__`) column is not a data column to rename. Migration: fix the name, use the other method (or `map_columns`), or pass `strict=False` to keep the old behaviour. +- `strict` is part of the identity of `MapTags` and `MapData`, as it is of the column selection operators: an operator's identity is its full configuration (developer decision, ITL-1160). Their identity is now `(class name, name_map, drop_unmapped, strict)`, and it is saved in `to_config()`. This moves the content and pipeline hash of every `MapTags` and `MapData` step and of everything downstream of one, and with them the paths of the tables that are scoped by those hashes. +- The stream methods `map_tags`, `map_data` and `map_columns` take the renames dict as their only positional argument, and `drop_unmapped`, `strict` and `label` are keyword-only (ITL-1160, ITL-1171). `stream.map_tags(name_map, True)` and `stream.map_tags(name_map, False, "label")` raise `TypeError`: write `drop_unmapped=True`, `label="label"`. `stream.map_data(name_map={...})` no longer names the dict (it reads as a rename of a column called `name_map` and raises `TypeError`): pass the dict by position. The operator classes `MapTags`, `MapData` and `MapColumns` keep their arguments. + #### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883) **The array format and every array hash changed in v0.3.** A store written by v0.2 that holds @@ -73,6 +375,1001 @@ first `InputValidationError`, with each other failure attached as a note and log stay queued for the next flush. A flush that fails only for other reasons still raises `RuntimeError`, with the first failure as its cause. +#### `SpiralDBTableSource` reads no rows when it is built, and reads in batches (ITL-186) + +A Spiral source used to read its whole table when it was built, which made large tables +unusable (about 4.6 GB for an 8-million-row table). It is now built from the table's metadata +alone, in about a second, and reads rows only when asked: in batches of at most `batch_rows` +rows (default 8,192) for `iter_data()` and pipeline jobs, and all at once for `as_table()` and +the dataframe methods. See "Reading a Spiral table" in `docs/concepts/sources.md`. + +What changes for existing code: + +- **Default `source_id`.** It is now `/.
@`, for + example `edi-tutorial-v1-841226/default.session@table_Jo7J0lpkLcOM` (it was + `//
`). SpiralDB's own table id makes it unique: a table dropped + and created again under the same name is a different source. For a Spiral source built + without `source_id`, the `content_hash`, every record ID and every source-info token + change. `pipeline_hash` does not. +- **Record IDs come from the primary key.** A row's token is the Arrow hash of its key + columns (for example `arrow_v1:000002...`), no longer its position (`row_0`). Record IDs + and source-info tokens of every Spiral source change, also with an explicit `source_id`; + `content_hash` and `pipeline_hash` of a source with an explicit `source_id` do not. + Pipeline records stored earlier for a Spiral source are not found again: the rows are + recorded once more, and no result is recomputed, because function results are found by + the row's data. +- **`record_id_column` is removed** from `SpiralDBTableSource`: the primary key identifies + the row. +- **A table without a key schema is refused**, also when `tag_columns` is given. +- **`SpiralDBTableSource` is no longer a `DBTableSource`.** +- **A read is capped at `max_rows` rows** (default 100,000); a larger read raises + `SourceTooLargeError`. Pass `limit=`, `where=`, or `max_rows=None`. +- **A saved job** stores the table id. Loading it raises `ValueError` if the saved table name + now leads to a table with another id. Loading reads no rows. + +#### A set of a primitive is stored as the native set container; its rows hash differently (ITL-611) + +A column declared `set[T]`, where `T` has no logical type (`set[int]`, `set[str]`, +`set[bytes]`, `set[datetime]`), is now stored under the Arrow extension type +`set[]` (for example `set[int64]`, metadata +`{"category": "set", "element_kind": "arrow"}`: the element type is in the name and is read +from the storage, `large_list`) instead of a plain `large_list`. This moves: + +- the stored Arrow type of such a column (at the top level, as `set[T] | None`, and as the + element of a list: `list[set[int]]` is stored as `list[set[int64]]`); +- the row hash of every row that holds such a column, and with it the input hash of a function + pod that takes such a column (its results are computed again), and the default `source_id` + and content hash of an in-memory source with such a column. + +Type, schema and pipeline hashes do not move, nor does anything about `list[T]` or about a set +of a logical type. No released version wrote such a row: only stores written by development +builds of v0.3 hold a set of a primitive as a plain list. Such a store is not refused: a node +serves its stored values as lists, and a run that adds a row raises when the stored and the new +form are combined. Delete the stored tables of such nodes and run again. + +#### `SemiJoin` keys on the shared tag columns only (ITL-1088) + +`SemiJoin` used every name its two inputs share, tag or data, as a match key. A left row was +kept only when its tags **and** its values in every shared data column equalled those of some +right row, and a name that was a tag on one side and a data column on the other was a key too. +So the operator read data values, which only filters may do. The match keys are now exactly the +columns that are tag columns of both inputs. Any other shared name (data on both sides, or a +tag on one side and data on the other) is ignored, and the right input's data columns are +never read. + +```python +left = DictSource(data=[{"id": 1, "y": 1}, {"id": 2, "y": 2}], tag_columns=["id"]) +right = DictSource(data=[{"id": 1, "y": 1}, {"id": 2, "y": 9}], tag_columns=["id"]) +SemiJoin()(left, right) +# v0.2.0: only id 1 (id and y must both match) +# now: id 1 and id 2 (only the shared tag id decides), and a SemiJoinIgnoredColumnWarning +# naming 'y' +``` + +- **What changes.** If the two `SemiJoin` inputs share a data column name, the result now keeps + every left row whose shared tags match, whatever the data values. Before, rows whose shared + data values differed were dropped, so the new result always contains the old one. Filtering a + stream by a subset of itself (every data name shared, equal values) gives the same rows as + before. A shared data column whose type or nullability differs no longer raises, because it + is not a key. +- **The warning.** Each ignored shared name is listed in a `SemiJoinIgnoredColumnWarning` + (new, in `orcapod.errors`), once per operator and set of names. Silence it by renaming or + dropping the column on the right input before the semi join, or with + `warnings.filterwarnings("ignore", category=SemiJoinIgnoredColumnWarning)`. This behaviour is + provisional in v0.3 and may become an error. +- **Unchanged.** `SemiJoin`'s identity and pipeline hash do not change. The output schema is + still the left input's. With no shared column at all, every left row is still returned. +- **Stored operator caches.** A cached `SemiJoin` node keeps its table path, so what it stored + before the upgrade is still there. With caching off it computes the new result. With + `CacheMode.LOG` it computes the new result on its next run, passes it downstream, and adds + the rows it had not stored yet. With `CacheMode.REPLAY`, and for a pipeline loaded from a + file, it serves the stored rows, which are the old, smaller result. To get the new result + there, run the pipeline once with the operator cache in `LOG` mode, or clear the operator's + cache. + +#### pandas values: a strict, content-based hash; column order is identity (ITL-897, ITL-887) + +A `pandas.DataFrame` or `pandas.Series` value now hashes under its own label, +`pandas_dataframe_v1` / `pandas_series_v1`, instead of as an Arrow table (`arrow_v1`). +Storage and hashing are separate: a value is stored with pyarrow's own conversion (unchanged), +and the hash is defined by Orcapod and is about the content of the value. The hash is strict: +everything about a value that can be changed is part of its identity, except a property +storage does not restore (developer, November 11 and 12, 2026). + +Before, these pairs of values shared a hash, so a cached pod could be handed the result +computed for the other one (for example the wrong `df.columns[0]`): + +| Property | Before | Now | +|---|---|---| +| Column order | not hashed | hashed | +| MultiIndex level order (frame and Series) | not hashed | hashed | +| Struct field order in a `pd.ArrowDtype` struct column | not hashed | hashed | +| `columns.name` / `columns.names` | not hashed | hashed | +| Dtype of the column axis, per level (an `int64` versus an `object` Index of the same labels) | not hashed | hashed | +| Column label type (`0` versus `"0"`), MultiIndex columns versus flat labels | not hashed | hashed | +| Series name type (`0` versus `"0"`); a frame versus its one-column Series | not hashed | hashed | +| The exact Arrow type of a non-nested `pd.ArrowDtype` column (`binary` versus `large_binary`, `null` versus `large_string`) | not hashed | hashed, as storage restores it (a nested type, and a decimal, which storage restores as `object`, are not distinguished; ITL-1054 decides nested types) | +| pandas dtype where the Arrow values coincide (`Int64` versus `int64`, `boolean` versus `bool`, `string` versus `object`) | not hashed | hashed | +| Categorical versus string; the category list (unused categories included), category order, `ordered` (ITL-887) | not hashed | hashed | +| `attrs` | not hashed | hashed, as the JSON object storage keeps | +| Values, row order, index values, index names and dtype, column dtype, Series name, `-0.0` versus `0.0` | hashed | hashed | +| `flags`, `RangeIndex` versus an equal `int64` index, an index's `freq`, unused MultiIndex levels, `string[python]` versus `string[pyarrow]`, key order of dicts in an `object` column | not hashed | never hashed: storage does not restore them (a `dict` is unordered) | +| Type of an index level name (`1` versus `"1"`) | hashed | never hashed: storage writes an index name as text, so both read back `"1"` | +| Types of column labels of mixed type (`0` beside `"b"`) | not hashed | never hashed: storage writes labels of mixed type as text | +| A masked, string or Arrow dtype of an index level (`Int64` versus `int64`) | not hashed | never hashed: storage restores an index level as NumPy | +| The dtype of a categorical's categories where storage changes it (`object` categories `[1, 2]` versus `int64`) | not hashed | never hashed: storage restores the categories from the Arrow dictionary | + +A value read back from a store now always hashes like the value that was written. Before, an +`object` column of non-strings (`[1, 2]` reads back `int64`) hashed differently after the +round trip; such a column is now hashed in the form storage restores. The same holds for an +`object` column of lists or dicts (a list `[1, None]` reads back `[1.0, nan]`) and for index +level names (the name `1` reads back `"1"`). + +Storing a pandas value that storage would corrupt now raises `ValueError`: column labels +and index names that collide once written as text (the index name `1` with a column `"1"`; +integer names of MultiIndex levels), and distinct labels that read back equal (the labels +`True` and `False`). Before, such a value was written and read back without its index (the +levels came back as columns) or with duplicate labels. + +`attrs` must be JSON. `attrs` with an entry JSON cannot write (a `set`, a tuple as a key) +are refused by the hash and by storage with `ValueError` naming the key, on every supported +pyarrow version. Before, pyarrow dropped the whole `attrs` with a warning or raised +`TypeError`, depending on its version. A key that is not text is still accepted and is +written as text (`{1: "x"}` reads back `{"1": "x"}`). + +The hash is `sha256(descriptor + b"\x00" + D)`: `D` is the Arrow hasher's ordered digest +(starfix 0.5 with `ordered_columns` and `ordered_struct_fields`) of the value's Arrow table +with positional field names, and the descriptor is canonical JSON in an Orcapod-owned grammar +of `attrs`, typed labels, `columns.names`, the column-axis dtype, index levels and pandas +dtypes. The definition and +the grammar are in `docs/reference/hashing.md`, section 10. A label or dtype outside the +grammar raises `ValueError` naming it. + +#### pandas logical types are named `orcapod.pandas.dataframe` / `orcapod.pandas.series` (ITL-897) + +The Arrow extension names of the two pandas logical types gain the `orcapod.` prefix (they +were `pandas.dataframe` / `pandas.series`), and the metadata is `{"ignore":[],"version":1}`: +the version, and the list of properties the data context configures the value hash to +ignore. Nothing is ignored by default. `hash_ignore` in the logical type registration file +can name `column_order`, `struct_field_order` and `attrs` for a frame, and `series_name`, +`struct_field_order` and `attrs` for a Series; two lists are two logical types, and a column +stored under another list is refused on read. The old names +are not registered any more: a column stored under them is refused on read, never read as +the current types or as raw bytes. + +**What moves.** The hash of every pandas value, and with it the data hash and the cache keys +of every row that holds a pandas value. Because the name is part of the logical type +identifier, the type hash of `pd.DataFrame` / `pd.Series`, the schema hash, the pipeline hash +and so the table paths and system-tag column names of every node with a pandas column move +too, and of every node downstream of one. The stored bytes of a pandas value are unchanged. +v0.3 already recomputes everything a v0.2 store holds (ITL-910), and v0.2 tables are still +found by name, reported and left unchanged, so this adds no migration step for a released +store. A store written by an unreleased v0.3 development build computes nodes with a pandas +column once more, in new tables; the older tables stay in place, unchanged, and are not +reported. + +#### Public interface: `ArrowHasherProtocol.hash_table_ordered` (ITL-897) + +`ArrowHasherProtocol` gains a required method, `hash_table_ordered(table)`, implemented by +`StarfixArrowHasher`: the same procedure as `hash_table`, with column order and struct field +order at every depth part of the digest (`struct_field_order=False` leaves struct field +order out). `hash_table` is unchanged and stays order +independent (streams, datagrams, rows). A custom Arrow hasher must add the method. +`PANDAS_DATAFRAME_HASH_LABEL` and `PANDAS_SERIES_HASH_LABEL` in +`orcapod.hashing.pandas_value_hash` define the two labels, and +`PANDAS_IDENTITY_PROPERTIES` there lists every property and whether it is part of the hash. + +#### pyarrow 22.0.0 or newer is required (ITL-897) + +The minimum pyarrow version is raised from 20.0.0 to 22.0.0. pyarrow 20 and 21 do not +restore `DataFrame.attrs` when a frame is read back, so a frame with `attrs` would hash +differently after it was stored; from 22.0.0 `attrs` is restored. The locked version +(23.0.1) is unchanged. + +#### A `dict` or `set` value hashes the same whatever order it was built in (ITL-905) + +A `dict[K, V]` value is stored as `large_list>` and a plain `set[T]` +(element type without a logical type) as `large_list`. Both were stored in Python +iteration order, so `{"a": 1, "b": 2}` and `{"b": 2, "a": 1}` had different row hashes, +and the same `set[str]` had a different row hash in every process (`PYTHONHASHSEED`). +Orcapod now stores dict entries sorted by key and set elements sorted, wherever a Python +value becomes Arrow; see the hashing reference, section 1e, for the order. A NaN key or +element, values that cannot be compared, and two keys that convert to the same stored +value raise `CanonicalOrderError` (a `ValueError`) naming the column. A datagram built from +Python values (and values set with `update` or `with_columns`) holds its dicts in the same +order, so a newly computed pod output and one served from the cache iterate alike. Dicts +nested in dataclass or pydantic instances are the exception: a fresh instance keeps insertion +order, while a read-back instance uses key order. An Arrow table is still hashed exactly as +stored: a key/value list that arrives in Arrow form is never re-sorted. + +Set values passed to `update` after Arrow materialization, or to `with_columns` with an +explicit Arrow list field, are also stored in canonical order. The existing Arrow field +(width, nullability and metadata) is preserved; actual list values keep their supplied order. + +Which hashes move because of this change, on top of the v0.3 hash step (ITL-910, ITL-945), +which moves every stored hash: only those of values holding a dict whose insertion order was +not key order, or a `set[T]` of two or more elements whose iteration order was not sorted +(for `str` and `bytes` that order changed with every process, so it had no stable value). +For such a row, its data hash moves, and with it the function pod cache key, the pipeline +record and entry ids, and the default `source_id` (with its per-row record ids) of a +source built without an explicit `source_id`. Schema hashes, `pipeline_hash` values and +explicit `source_id`s do not move for this reason, and a dict already built in key order is +stored byte-for-byte as before if every nested unordered container is also canonical. +In the golden hash suite exactly two entries moved: +`pipeline:index_pick.index.row[{"subject":"s1"}].data_hash` and `...s2...`, whose rows hold +`meta={"k": ..., "j": ...}`. + +What a `0.2.0` store sees (checked against a Delta store written by `0.2.0`): nothing +changes because of this fix. v0.3 reads no table `0.2.0` wrote (every table is at a new path, +see "Data context `std:v1:default`", ITL-944), so every input is computed again whether its +dict was in key order or not, and no row stored in `0.2.0`'s insertion order is ever served. +The `0.2.0` tables stay where they are, untouched. Every dict and plain set a v0.3 store +holds from a pod output or a source built from Python values is in canonical order. + +#### The async orchestrator names system-tag columns as the sync orchestrator does (ITL-906, ITL-885) + +`MergeJoin`, `Batch` and `GroupBy` put their inputs' pipeline hashes into the names of the +system-tag columns they write. When one of these operators read an input that is not a +source (an operator's or a function pod's output), `AsyncPipelineOrchestrator` used a +different hash than `SyncPipelineOrchestrator`, so the same graph got different system-tag +column names, and different record ids, depending on which orchestrator ran it. The async +orchestrator now uses the input's pipeline hash, and its names and record ids are the ones +the sync orchestrator has always produced. Nothing a sync run produces changes, and graphs +where these operators read a source directly are unaffected. + +Every operator's `async_execute` now requires the keyword argument `input_pipeline_hashes`, +one pipeline hash per input, as the operator node always passes it; a missing argument raises +`TypeError`. The operators that use the hashes (`Join`, `MergeJoin`, `Batch`, `GroupBy`, +`PolarsFilter`, and any operator on the default barrier path) raise `ValueError` when the count +does not match the inputs; the per-row operators and `SemiJoin` take the argument without +using it. Before, these +operators fell back without it to the schema-only hash, and `Join`'s async path to names built +from the input positions (`::0`, `::1`, ...), neither of which the sync path uses. Code that +calls an operator's `async_execute` directly must pass the inputs' pipeline hashes. + +This also fixes ITL-885: a store written by one orchestrator can now be extended by the +other. Before, the function node downstream of such an operator wrote two sets of +system-tag columns into one pipeline table, and the second run failed while storing its +results. + +Migration: stores written by v0.2 are not affected, because v0.3 never reads a table +written by v0.2. Only a store written by an async run of an unreleased v0.3 build, for a +graph with `MergeJoin`, `Batch` or `GroupBy` after an operator or a function pod, holds the +old names: a later run on that store fails when it writes the pipeline table of the function +node downstream. Delete that table, or use a fresh store, and run again. + +#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910) + +**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1` +to `arrow_v1`.** starfix 0.4 (hash format 1) lost information for some inputs, so different +data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and +`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean +columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash +format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept. + +- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default + `source_id`s and the `record_id`s and source content hashes built from them, function + input-data hashes, pipeline base and entry ids, operator and cached-source record hashes, + invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The + hash label in front of each (`arrow_v1:000002...`) shows the new format. Pipeline hashes, + schema hashes, system-tag column names and table paths under the default + `table_scope="pipeline_hash"` do not move for this reason; they move in v0.3 because of + the hash label rule (ITL-945, below). Paths that embed a content hash + (`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or + `Batch` orders members by `record_id`, the order of members can change too. +- **Results stored by 0.2.0 are not reused.** Every table v0.3 writes is at a new path and + every row names its data context; see "Data context `std:v1:default`" below (ITL-944) for + what a v0.2 store looks like to v0.3. +- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator + carrying one works, and its type is kept. A `fixed_size_list[n]` hashes differently from a + `large_list` with the same values. pyarrow's canonical extension types (`arrow.uuid`, + `arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid` + column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and + `StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846). +- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed + silently: a non-nullable field holding a null; a zero-row table with a `map` column; two + schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a + column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`, + interval, `string_view` and `binary_view` columns, union and run-end-encoded types are + still refused, now with `ValueError` instead of `NotImplementedError`. A source built from + a table with such columns raises `ValueError` naming every refused column and its type. +- Column order and struct field order are still not part of an Arrow hash. + +#### A Spiral source's default `source_id` is `//
` (ITL-926) + +`SpiralDBTableSource` used the bare table name as its default `source_id`. Tables with the +same name in different projects (the EDI sample projects all share their table names) then +had the same `content_hash`, record IDs and source-info tokens. In one store, a job on one +project read back the other project's pipeline records along with its own (cached results +were never mixed up: they are keyed on the row's data). The default is now the project id, +the dataset and the table name, for example `edi-sample-v1-855632/default/session`. + +For a Spiral source built without `source_id`, the `content_hash`, every record ID and every +source-info token change, so pipeline records stored by 0.2 for such a source are not found +again. The rows are recorded once more and no result is recomputed, because function results +are keyed on the data. For example, table `subject` (key `subject_id`, one row) in project +`edi-sample-v1-855632`: + +- `source_id`: `subject` becomes `edi-sample-v1-855632/default/subject`. +- `content_hash`: `semantic_v0.1:0380725acb90...e7f4f6` becomes + `semantic_v0.1:3a659ed610bb...29a93`. +- Row 0's record ID: `18d1efa53fbc52b6a40f406eca35ca81` becomes + `096c4e22981c52daa2b088bffe729ee7`. + +`pipeline_hash` does not change (it depends only on the schema), nor does anything for a +source given an explicit `source_id`. To keep 0.2 identities, pass the old value: +`SpiralDBTableSource(project, "subject", source_id="subject")`. Jobs saved by 0.2 store +their `source_id` explicitly and keep it when loaded. + +#### The Delta store keeps every column's Arrow type; rebuild a Delta store written by an earlier v0.3 build (ITL-928) + +Delta Lake has no unsigned integer, no `timestamp` in seconds, milliseconds or nanoseconds, no zone other than UTC, and no `date64`, `time` or `duration` type. `DeltaTableDatabase` used to let the Delta writer convert such a column, so a read gave back another type: `uint32` as `int32` (and a value above the signed maximum failed the write), `timestamp[s]`, `[ms]` and `[ns]` as `timestamp[us]` (losing nanosecond digits), `timestamp[us, tz=America/New_York]` as UTC; `date64`, `time` and `duration` columns could not be stored at all. A tag read back with another type changes the record ids computed from it, so on a Delta store a re-run over such a tag wrote the downstream nodes' pipeline rows a second time (a fresh job then read 6 rows for 3). + +Now such a column's original Arrow type is recorded in the Delta field's metadata, an unsigned integer is written as the signed integer of the same width bit for bit, and `timestamp[ns]`, `date64`, `time` and `duration` are written as their integer value; every read casts the column back to its recorded type. Nested columns (`list`, a struct with a `timestamp[ms]` field) are covered. Columns of every other type, and logical-type columns, are stored as before. + +A Delta store written by an earlier unreleased v0.3 build has no recorded types: its tables keep reading back Delta's types, a re-run over such a tag still writes duplicate pipeline rows, rows it already duplicated stay, a function node whose rows mix rows read from it with newly computed rows raises `InputValidationError` naming the column and both types, and appending a `timestamp[ns]` column to one of its tables is refused (the stored `timestamp[us]` differs from the new integer form). Rows with an unsigned column are still appended to such a table's signed column, as before, and read back signed; a value the signed type cannot hold is refused with `InputValidationError` and nothing is written. Nothing is repaired automatically: rebuild such a store by running the pipeline into a new store. Stores holding none of these column types are not affected. No content hash, pipeline hash, record id or table path changes. + +#### Logical type identifiers: every built-in type writes one, and all of it is hashed (ITL-935) + +A logical type is now identified by its Arrow extension name **and** its Arrow extension +metadata, and the whole identifier is part of the column's data type +(`docs/concepts/logical-type-identifiers.md`). The identifier belongs to whoever implements +the type and changes only when an existing value would be stored or hashed differently. + +- **Built-in types write an identifier.** `orcapod.path`, `orcapod.upath`, `orcapod.uuid`, + `orcapod.file`, `orcapod.directory`, `pandas.dataframe` and `pandas.series` now write the + extension metadata `{"version":1}` (they wrote none); `orcapod.ndarray` already did. + `list[T]` and `set[T]` of a logical type (the native containers) have no version of their + own, by design: their metadata carries the element's name and new metadata, so their + identifier changes exactly when the element's does. +- **The whole identifier is hashed.** A type hash, a schema hash and a function signature + used to contain the logical type's name only. They now contain + `;` (`LogicalTypeIdentifier.text`). The hashed cell + of a value-hashed type (UUID, file, directory, array, pandas) starts with the same text, + where it used to start with the name alone, written with `:` for `.`. +- **Fixed: `[]` and `[None]` hashed the same in a list or set of a value-hashed type.** The + hashed cell of a `list[T]` / `set[T]` column (T a UUID, file, directory, array or pandas + type) joined its element hashes with a separator and wrote nothing for a null element, so + an empty list, `[None]` and `[None, None]` shared one hash. The cell now holds the element + count and, per element, a null marker or a length-prefixed element hash. +- **What moves, once, in v0.3.** For every node whose schema or function signature has a + column of a logical type (including dataclass and pydantic columns): its schema, function + and pipeline hashes, hence the paths of its stored tables, and the hashes of the rows that + hold such values. Together with the other v0.3 hash changes this means no result stored by + v0.2 is reused; nothing stored is changed or deleted. **Nodes with only plain types are + not affected by this change.** +- **Reading.** A stored column is read only if its stored metadata is one the registered + type declares readable (`readable_extension_metadata()`, new on `LogicalTypeProtocol`, + with a default in `BaseLogicalType`). Otherwise `LogicalTypeIdentifierError` (a + `ValueError`) names the column, the stored identifier and what this installation reads. + Before, a column read as plain storage was wrapped as the registered type whatever its + stored metadata said. +- **Data written before v0.3 stays readable.** The seven types above read columns stored + with empty metadata as version 1 (the layout is the same), for example a Delta table + written by v0.2 and opened as a source. +- **For authors of logical types.** `make_arrow_extension_type` takes `also_reads`: older + metadata values the type still decodes. An extension name may not contain `;` + (`register_logical_type` raises `ValueError`). A custom logical type that does not derive + from `BaseLogicalType` must implement `readable_extension_metadata()`. + +#### Wrong hashes fixed: values that collided or depended on data, process or spelling (ITL-942) + +A hash-site audit (ITL-938) found hashes that were **wrong**: two different things shared a +hash, or one thing hashed differently depending on its data, the process or how it was +written. A wrong hash is a bug that overrides hash stability, so these values move. Each fix +changes only the values that were wrong. + +| Fix | Before | Now | What moves | +|---|---|---|---| +| Batch / GroupBy source-id fold | members concatenated with no separator: `["ab", "c"]` folded like `["a", "bc"]`, and a `None` member like `""` | SHA-256 of the canonical JSON list of members, in order | the `_tag_source_id` value of every row after a `Batch` or `GroupBy`, and the entry ids downstream of it. Column names and paths do not move. Source-info columns are unchanged | +| Arrow hashing of nested types | the hashed type was rebuilt from the first non-null value, so nested nullability counted for an empty or all-null column and not for a column with data | the hashed type is the declared type: struct and list fields keep their nullability, at every depth, with or without data | row hashes of struct or list columns with non-nullable nested fields, and of all-null or zero-row `Path`, `File`, `Directory` and `UUID` columns. Key values only | +| Set order in structure hashes | items sorted by `str()`, so `{1, "1"}` hashed differently per process (`PYTHONHASHSEED`) | items sorted by their canonical JSON text | structure hashes of sets whose `str()` order differs from their JSON order (mixed types, sets of tuples) | +| Dict keys in structure hashes | keys turned into `str()`: `{1: "x"}` equalled `{"1": "x"}`, and `{1: "a", "1": "b"}` lost a key | a dict with only `str` keys is unchanged; any other dict hashes as a sorted list of `[key, value]` pairs | structure hashes of dicts with non-`str` keys | +| Dicts that spell out an encoding | `{"__type__": "tuple", "items": [1, 2]}` hashed like `(1, 2)` (also set, namedtuple, generic alias, union) | only the hasher and type handlers write `"__type__"` objects; a dict with a `"__type__"` key hashes as a pair list | structure hashes of dicts with a `"__type__"` key. Handlers that returned a plain `{"__type__": ...}` dict must return `orcapod.hashing.semantic_hashing.Encoded(tag, fields)` | +| Namedtuple field order | fields sorted by name: `Point(x=1, y=2)` hashed like a `Point` declared `(y, x)` holding `y=2, x=1`, although the tuples differ | fields hashed in declared order | structure hashes of namedtuples | +| Column selection operators | `SelectTagColumns`, `SelectDataColumns`, `DropTagColumns`, `DropDataColumns` hashed `["x"]`, `("x",)`, `{"x"}`, `["x", "x"]` differently, and `["y", "x"]` unlike `["x", "y"]`, although the output is the same | identity is the set of names, hashed as a sorted list | pipelines that spelled the columns as a tuple, a set, or an unsorted or repeating list: their hashes **and table paths** move. Sorted lists (the usual spelling) keep their hashes | +| Record-id system tag type | `fixed_size_binary[16]` out of a source, `large_binary` after a `Tag`, so one row had two operator-cache keys (`cache_mode` LOG/REPLAY, sync run versus async orchestrator) | `large_binary` everywhere | operator-cache and cached-source row keys, and the content hash of a raw stream holding source system tags. On Delta Lake the stored column type does not change: Delta has a single binary type | +| Default function version | `v0.0` through `@function_pod`, `v1.0` through `FunctionPod.from_fn` and `side_effect_function_pod` | `v0.0` for every entry point | **table paths** of pods built with `FunctionPod.from_fn` or `side_effect_function_pod` without an explicit version: from `.../v1/...` to `.../v0/...` | +| Value-hashed logical types with no value (follow-up to the nested-type fix, found on the combined v0.3 branch, ITL-967) | a column of a value-hashed logical type (UUID, `File`, `Directory`, array, `DataFrame`, `Series`, and lists or sets of them) that had zero rows or held only nulls hashed like a plain `large_binary` column and like every other such type | the hashed field keeps the logical type's identifier (extension name and metadata), with or without values | the Arrow hash of every table with such a column, with data too: row data hashes, default `source_id`s and what derives from them | + +**Migration.** A pod built with `FunctionPod.from_fn` or `side_effect_function_pod` that relied +on the old default keeps the major-version component of its paths (`.../v1/...`) by passing +`version="v1.0"`. That alone does not make v0.3 find tables written by v0.2: every v0.3 path +also differs in its hashes and its layout component (ITL-945, ITL-944). A pipeline that spelled +selected columns as a tuple or a set gets new tables for that operator and everything +downstream; no stored row is changed or deleted. Every other fix changes values only, never a +path or a stored schema. + +Reviewed and kept by decision: `MergeJoin(m1, m1)` (a stream joined with itself) has a +different pipeline hash from `MergeJoin(m1, m2)` with the same schema, because a pipeline hash +covers the pipeline's structure; two inputs with the same source identity count as one. +Function default values in the function-value hash are tracked in ITL-953. See +`docs/reference/hashing.md`. + +#### Data context `std:v1:default`: a `__data_context` column on every table, and every table at a new path (ITL-944) + +**v0.3 writes every stored table at a new path, and records on every row which generation of +hashing wrote it.** A store written by v0.2 is not migrated and not modified: v0.3 does not +read its tables as results, and computes every input again into its own tables. + +- **One data context key for the whole of Orcapod's hashing.** The default data context is + now `std:v1:default` (manifest `orcapod/contexts/data/v1.json`). The version increments + when, and only when, a released hash value changes, for any reason (ITL-946). Everything + stored before v0.3 (rows that carry `std:v0.1:default`, and tables without the new column) + is the earlier generation. A manifest is frozen at the + release of the library version that contains it, and may be edited in place only while + that release is prepared: `orcapod/contexts/released_manifests.json` pins the digest of + each released one without its `golden` history, and a test fails when a pinned manifest is + edited (ITL-946). `v1.json` is pinned when v0.3.0 is tagged. +- **Manifest contents.** Besides the wiring of the hashers, a manifest now has + `released_in` (the library version that froze it; `null` while open), `previous` (the data + context it follows), `definitions` (a name for each definition that lives in code), + `libraries` (the libraries the core hashes depend on, with versions), `golden` (the folder + and the append-only history of the golden hash files, ITL-946) and `changes` (what differs + from `previous`). +- **Logical types moved out of the manifest.** The registered logical types and the value + handlers that hash their Python values are now in + `orcapod/contexts/logical_types.json`, which is not frozen: a logical type is versioned by + its own identifier, not by the data context. What the default context registers is + unchanged. `JSONDataContextRegistry` and `contexts.create_registry` take + `logical_types_file` to use another registration file; a manifest that lists its logical + types inline is still built as written. +- **New column `__data_context`** (`large_string`) on every row of the function pipeline + table, the function result table, the operator-cache table, the cached-source table and + the side-effect invocation log. It **replaces** `__input_data_context_key` in the pipeline + table and `_context_key` in the result table. In memory nothing is renamed: + `ColumnConfig(context=True)` still shows `_context_key`. +- **New column `__pf_var_function_version`** in the function result table: the full version + string the function was declared with (`"v3.7"`). Only the major part is in the table + path, and until now no stored column held the version. +- **New table layouts, so new paths.** The last component of every table path moved: + + | Table | v0.2 | v0.3 | + |---|---|---| + | Function pipeline table | `…/schema:/pdb_v1` | `…/schema:/pdb_v2` | + | Function result table | `…/_result//rdb_v1` | `…/_result//rdb_v2` | + | Side-effect invocation log | `…/schema:/tdb_v1` | `…/schema:/tdb_v2` | + | Operator-cache table | `/schema:` | `/schema:/odb_v1` | + | Cached-source table | `source/node:` | `source/node:/sdb_v1` | + + The layout component is always the last one: `CachedSource(..., cache_path=("my", "cache"))` + now stores at `("my", "cache", "sdb_v1")`, and `CachedSource.cache_path` returns that full + path. Pass the path without the layout component as `cache_path`. +- **Reads and lookups use rows of the current data context only.** A result lookup matches + on the input hash and the data context; every read of a whole table (a node's `as_table()`, + `iter_data()`, `get_all_records()`, `as_source()`, `get_all_cached_outputs()` of a cached + function, a stored operator node, a `CachedSource`) leaves rows of another data context + out, leaves them unchanged, and logs one warning per table with their number (logger + `orcapod.contexts.stored`). +- **What a v0.2 store looks like to v0.3.** + - Its tables are at the old layout. v0.3 never reads them as results and never writes to + them; they stay byte for byte as v0.2 left them and stay readable with v0.2. + - The first v0.3 run **computes every input again**, also for functions and data that did + not change, and stores the rows in the new tables. Before that run, reading a node + returns no rows. + - When v0.3 finds a table at an old layout, it logs one warning with the number of rows: + `Found 3 rows in stored table 'my_job/.../pdb_v1' written by an earlier version of + Orcapod (older table layout, data context std:v0.1:default). They are not used and are + left unchanged; they can be migrated later.` + - **Side-effecting nodes fail instead of delivering again.** A side effect is an external + action, and a completion stored by v0.2 cannot be matched with current inputs. Both + kinds of side-effecting node are covered: a side-effect pod (`@side_effect_pod`, + `@sink_pod`, `@tap_pod`), whose completions are in its invocation log, and a function + pod that takes the invocation context (`@function_pod(ctx_arg=...)`, + `@side_effect_function_pod`), whose completions are in its function tables. Such a + node that finds its tables at an old layout (or completions of another data context in + its current tables) raises `orcapod.errors.SideEffectLogDataContextError` (a + `StoreError`, so it fails the run whatever the `error_policy`) before it delivers or + writes anything. The error names the tables. Other function, operator and source nodes + of the same job are not refused. Keep using v0.2 for the node. A fresh store is not + a safe way around the error: it holds no completions, so on it the node performs every + side effect again; use one only when a repeated delivery is acceptable. The tables of + an earlier version are found by listing the store (ITL-954): on a store that cannot + list its tables the node logs one warning that it could not check, and runs, so the + refusal cannot be relied on there. + - `orcapod.contexts.stored_data_contexts(store, record_path)` counts the rows of a table + by data context, for example `{"std:v1:default": 3}`; a table without the column reports + its rows under `std:v0.1:default`. +- **`std:v0.1:default` is no longer shipped.** `orcapod/contexts/data/v0.1.json` is removed + (it will be kept in `orcapod-sanctuary`), so `resolve_context("std:v0.1:default")` and + `resolve_context("v0.1")` raise `ContextResolutionError` naming it as the data context of + Orcapod before v0.3. A pipeline or job file saved by v0.2 still loads: each node that + names `std:v0.1:default` is loaded under the current data context with a warning, and + shows no rows until v0.3 has run (its v0.2 tables are at the old layout). +- **Removed: the v0 to v1 migration machinery.** `orcapod.errors.SchemaVersionError`, + `NodeConfig(ignore_schema=...)` (and `NodeConfig.ignore_schema`), the package + `orcapod.migrations` (`migrate_pipeline_db`, `migrate_result_db`, `MigrationResult`), the + command `orcapod migrate` (`pipeline-db`, `result-db`), `ResultCache.base_record_path`, + `ResultCache.set_ignore_schema` and `CachedFunctionPod.set_ignore_schema`. They converted + tables without a layout component to `pdb_v1` / `rdb_v1`, which v0.3 does not read. A + table without a layout component is now counted and reported like any other old layout, + where it used to raise `SchemaVersionError`. Remove `ignore_schema` from your code; the + migration to the v0.3 layouts is a later tool (ITL-913, ITL-914). +- **An input of another data context is refused.** A node, cache or cached source that + would store a row for an input whose data context differs from its own raises + `InputValidationError` naming both keys, before anything is computed, delivered or + written. For a function node this is checked for the whole input, so it fails the run + whatever the `error_policy`. Only one data context is shipped, so this needs a custom + context registry to happen. It applies to nodes, caches and cached sources used + directly. Inside a `PipelineJob` it is not reached: a job wraps every input in a source + node of the default data context, so an input of another data context is not seen as + such there (a known gap, ITL-951). `ResultCache.store` refuses an output of another + data context in the same way. +- **A full data context key must match its manifest.** `resolve_context` selects a manifest + by the version part of a key, and used to return it whatever the rest of the key said + (`"custom:v1:production"` gave `std:v1:default`). A full key that is not the manifest's + key now raises `ContextResolutionError`. +- **Hash values.** The content hash of a stream that has no producer (a bare + `ArrowTableStream`) covers its `_context_key` column, so it changes with the key. No other + hash value changes because of this entry. +- **Removed (introduced earlier in this unreleased cycle, never released):** + `orcapod.hashing.stored_hash_labels` and `orcapod.errors.SideEffectLogHashLabelError`. + Rows are told apart by `__data_context`, not by the label inside a hash. +- **Internal API.** `ResultCache.lookup`, `ResultCache.store` and + `ResultCache.get_all_records` take the caller's `data_context_key`. + + **Migration:** none is needed to run the function, operator and source nodes of a v0.2 + store under v0.3, at the price of computing everything again; side-effect nodes fail + on such a store, unless the store cannot list its tables (then the node warns and + runs), and on a fresh store they perform every side effect again. A tool that converts old-layout tables is planned (ITL-913, ITL-914); until + then they are kept, untouched, for it. + +#### Hash label rule: labels renamed, out of hash inputs, bare digests inside names (ITL-945) + +**Every structure hash, every table path and every key moves in v0.3.** One rule now holds +for every hash label (developer, November 11, 2026; `docs/reference/hashing.md`, "The hash +label rule"): + +- **Labels are renamed to `_v1`.** `semantic_v0.1` becomes `semantic_v1`, the Arrow + label is `arrow_v1` (it was `arrow_v0.1` in v0.2), `merkle_sha256` becomes `directory_v1`; + `ndarray_v1` and `polars_expr_v1` keep their names; the file content hash keeps `sha256`. + A label names the algorithm that produced a digest and changes only when that algorithm + would give an existing input a different digest. +- **A label never enters a hash computation.** A nested hash now contributes its digest only: + as `{"__type__": "hash", "digest": ""}` inside a structure hash, as + `b";::" + digest` for an extension-typed cell inside an Arrow + hash (the name and metadata are the logical type identifier of ITL-935), and as the + raw digest in the entry-id and operator-node record-hash preimages and in the run snapshot + hash. Renaming a label later changes no digest. Until now `"
`, `postgresql_table:..
`, `spiraldb_table:..
@
` (ITL-1157). A `DBTableSource` used directly, over a connector that gives no database address, has no default: it requires a `source_id`, because the table name alone does not say which database the table is in. A source id or a column name holding `::` is refused, and so is an empty source id. -- A root source's content hash includes its key columns, the resolved `record_id_columns` in key order (`()` for the position; the primary key of a database or SpiralDB table by default), so two sources with one `source_id` that name their rows differently are two sources: they no longer share a cache (`CachedSource`) or a node's stored records. The pipeline hash, the source id (given or default), the record ids and the source-info tokens are unchanged. This applies to every root source, including `ListSource`, `PollingSource`, `SpiralDBTableSource` and `DerivedSource` with `tags_as_data`. -- A store written by a development build of v0.3 is discarded, not migrated: the v0.3 table layouts were redefined before their first release (the record-id column is now text). -- These change 849 golden values of data context `v1`, which no release has shipped: v0.3.0 is its first release. Nothing needs migrating beyond what v0.3 already requires: data written by v0.1.4 or v0.2 is already stale under v0.3, and data written by a development build of v0.3 is recomputed (a moved entry id or path is not found; nothing is lost or read wrongly). **Side effects are delivered again:** a side-effect node finds its completed deliveries by entry id, so deliveries logged by a development build of v0.3 are not recognized and a run delivers them again. A new store does not avoid this: it holds no log at all, so a run delivers every input again too. Run a side-effect node that was run by a development build of v0.3 only where a repeated delivery is acceptable. - -### Changed - -- A generic alias with no `__origin__` is refused (`TypeError`) instead of hashed by its `repr`, which is not stable. No `typing` or `collections.abc` alias lacks an origin on Python 3.11 to 3.13. diff --git a/changelog.d/ITL-1159.md b/changelog.d/ITL-1159.md deleted file mode 100644 index bd421726e..000000000 --- a/changelog.d/ITL-1159.md +++ /dev/null @@ -1,21 +0,0 @@ -### Added - -- Every source built from a table takes the keyword-only option `tag_only_columns`: - `ArrowTableSource`, `DictSource`, `DataFrameSource`, `CSVSource`, `DeltaTableSource`, - `DBTableSource`, `SQLiteTableSource`, `PostgreSQLTableSource` and `SpiralDBTableSource` - (ITL-1159). In v0.3 every tag is plain (a tag only, never also a data column), and the option - has one accepted value, `True` (all tags plain), which is also the default. A source built - with `tag_only_columns=True` is the source built without it: the same content and pipeline - hashes, source id, record ids, rows, stored paths and saved config. Any other value (`False`, - an empty collection, column names, even all of them) raises `ValueError`, because referential - tags arrive in v0.4. In v0.4 a declared tag becomes referential by default (a tag that is also - the data column of the same name) and `tag_only_columns` keeps a tag plain, so code that must - keep today's behaviour can pass `tag_only_columns=True` now. `ListSource`, `DerivedSource`, - `CachedSource` and `PollingSource` do not take the option. No hash or golden value changes; - the golden fixtures that build a source with tag columns now pass `tag_only_columns=True` - explicitly. -- A saved pipeline or job file does not record `tag_only_columns`: v0.3 writes no such key, and - both file format versions stay `0.2.0`. Plain tags are decided by - the format version: every source in a file at `0.1.0` or `0.2.0` has all its tags plain. The - format that brings referential tags (v0.4) writes `tag_only_columns` for every source, and its - reader refuses a source config without it, so a v0.3 file reads back plain in v0.4. diff --git a/changelog.d/ITL-1160.md b/changelog.d/ITL-1160.md deleted file mode 100644 index 699d0cf2d..000000000 --- a/changelog.d/ITL-1160.md +++ /dev/null @@ -1,14 +0,0 @@ -### Breaking changes and migration - -- `map_tags` / `MapTags` and `map_data` / `MapData` refuse, by default, a mapping key that is not a column of the kind they rename (ITL-1160). Before, such a key was accepted and renamed nothing, which hid mistakes such as `map_tags` of a column that had become data after `group_by`. The call now raises `InputValidationError` naming the missing names and the columns of that kind; a name that is a column of the other kind says so (`'age' is a data column: use map_data`). The check runs when the operator is called on a stream, before any row is processed, and on the async path against the first row. A raw stream's meta (`__`) column is not a data column to rename. Migration: fix the name, use the other method (or `map_columns`), or pass `strict=False` to keep the old behaviour. -- `strict` is part of the identity of `MapTags` and `MapData`, as it is of the column selection operators: an operator's identity is its full configuration (developer decision, ITL-1160). Their identity is now `(class name, name_map, drop_unmapped, strict)`, and it is saved in `to_config()`. This moves the content and pipeline hash of every `MapTags` and `MapData` step and of everything downstream of one, and with them the paths of the tables that are scoped by those hashes. -- The stream methods `map_tags`, `map_data` and `map_columns` take the renames dict as their only positional argument, and `drop_unmapped`, `strict` and `label` are keyword-only (ITL-1160, ITL-1171). `stream.map_tags(name_map, True)` and `stream.map_tags(name_map, False, "label")` raise `TypeError`: write `drop_unmapped=True`, `label="label"`. `stream.map_data(name_map={...})` no longer names the dict (it reads as a rename of a column called `name_map` and raises `TypeError`): pass the dict by position. The operator classes `MapTags`, `MapData` and `MapColumns` keep their arguments. - -### Added - -- `stream.map_columns(name_map)` and the operator `MapColumns` rename tag and data columns in one step (ITL-1160). A renamed tag stays a tag and a renamed data column stays data, with its source-info column; system tags pass through. `drop_unmapped=True` drops every tag and data column not in the mapping. As with `map_tags` and `map_data`, a name that is neither a tag nor a data column is refused unless `strict=False`, a new name taken by a column that is not renamed, or given to two columns, is refused, a `__` new name is refused when the operator is built, and a result with no data column is refused. Like `SelectColumns`, it carries no data meta (`__`) column of the input. Its identity is `("MapColumns", name_map, drop_unmapped, strict)`. -- `map_tags`, `map_data` and `map_columns` take the renames as keyword arguments too: `stream.map_data(age="age_weeks")` builds the same step, with the same hashes, as `stream.map_data({"age": "age_weeks"})` (ITL-1171). Give one dict or keyword arguments, not both (`TypeError`). `drop_unmapped`, `strict` and `label` are options, never renames: a column with one of those names, or a name that is not a Python identifier, is renamed through the dict. An old or new name that is not a string raises `TypeError` in both forms. - -### Fixed - -- `MapTags`, `MapData` and `MapColumns` refuse, when built, a new name that a stream reads as a system column: one that starts with the system-tag prefix (`_tag_`) or the source-info prefix (`_source_`), or the context key column (`_context_key`) (ITL-1160). Before, such a rename was accepted and the column then vanished from the stream or the stream failed on read with `No data columns found`. diff --git a/changelog.d/ITL-1162.md b/changelog.d/ITL-1162.md deleted file mode 100644 index cfcd589f6..000000000 --- a/changelog.d/ITL-1162.md +++ /dev/null @@ -1,11 +0,0 @@ -### Fixed - -- A `ConnectorArrowDatabase` over `SQLiteConnector` or `PostgreSQLConnector` now refuses a table it cannot store before anything of it is written, with `UnstorableColumnError` naming the connector, the column, its Arrow type and what to use instead (ITL-1162). A connector stores a column only when it reads it back as written: on SQLite `bool`, `int64`, `double`, `large_string` and `large_binary`; on PostgreSQL also `int32`, `float`, `date32`, `timestamp[us]` and `timestamp[us, tz=UTC]`. Before, a function node on such a database ran its function and then failed at commit (`sqlite3.ProgrammingError: ... type 'list' is not supported`), timestamps were stored as text and read back as strings, and narrower integers and floats read back widened, failing the next write. A job now refuses a node whose stored table such a database cannot hold (every function node; a side-effect node on SQLite) when the job is built, before anything runs, naming the node. The connectors no longer fall back to a `TEXT` column for a type they have no column type for. -- `PostgreSQLConnector` now refuses a column name longer than 63 bytes (UTF-8), which PostgreSQL truncates, so it would read back as another name (ITL-1162). Before, such a column was stored under the truncated name. A table name longer than 63 bytes is refused too, with `UnstorableTableNameError`, at every entry that creates, writes, reads or checks a table, so a truncated name is never sent: two tables whose names share their first 63 bytes would otherwise silently be one. `SQLiteConnector` refuses a table name starting with `sqlite_`, which SQLite reserves. -- A `ConnectorArrowDatabase` over `SpiralDBConnector` now refuses, by the same rule, a column SpiralDB cannot store: a value column whose name starts with `__` (the `__record_id` key is allowed) or any column whose name contains `.`, and a timestamp column with a time zone, which SpiralDB reads back without it (ITL-1164). Before, a function node on such a database ran and then failed at commit (`Column name '__input_data_hash' has reserved prefix`). A job refuses every function node, side-effect node, operator cache and cached source on such a database when it is built. - -### Known limitations - -- A connector job database cannot hold function nodes in v0.3.0: neither SQL connector (SQLite, PostgreSQL) nor SpiralDB. A function node's result table holds list-of-struct executor columns, which neither SQL connector stores, and `__`-prefixed value columns, which SpiralDB reserves (ITL-1164). A side-effect node is refused on SQLite (its log's UTC timestamp) and on SpiralDB, and an operator cache or cached source on SpiralDB (`__data_context`). Use a Delta database (`DeltaTableDatabase`) or `InMemoryArrowDatabase`. Storing such columns on connector databases is planned for a later release (ITL-1162, ITL-1164). -- On PostgreSQL, an operator cache with caching on is refused in v0.3.0 when it is written (before anything is stored): its system-tag column names (`_tag_source_id::` and a 64-character hash) are longer than PostgreSQL's 63-byte limit. Storing them under a mapped name is planned for a later release (ITL-1166). -- On PostgreSQL in v0.3.0, a table whose Orcapod path gives a name longer than 63 bytes is refused. Every node table and cached source path holds a 64-character hash, so only tables written directly through `ConnectorArrowDatabase` under short paths fit; reading an existing table with `PostgreSQLTableSource` is unaffected. Mapping long names is planned for a later release (ITL-1166). diff --git a/changelog.d/ITL-186.md b/changelog.d/ITL-186.md deleted file mode 100644 index 99b35203a..000000000 --- a/changelog.d/ITL-186.md +++ /dev/null @@ -1,57 +0,0 @@ -### Breaking changes and migration - -#### `SpiralDBTableSource` reads no rows when it is built, and reads in batches (ITL-186) - -A Spiral source used to read its whole table when it was built, which made large tables -unusable (about 4.6 GB for an 8-million-row table). It is now built from the table's metadata -alone, in about a second, and reads rows only when asked: in batches of at most `batch_rows` -rows (default 8,192) for `iter_data()` and pipeline jobs, and all at once for `as_table()` and -the dataframe methods. See "Reading a Spiral table" in `docs/concepts/sources.md`. - -What changes for existing code: - -- **Default `source_id`.** It is now `/.
@`, for - example `edi-tutorial-v1-841226/default.session@table_Jo7J0lpkLcOM` (it was - `//
`). SpiralDB's own table id makes it unique: a table dropped - and created again under the same name is a different source. For a Spiral source built - without `source_id`, the `content_hash`, every record ID and every source-info token - change. `pipeline_hash` does not. -- **Record IDs come from the primary key.** A row's token is the Arrow hash of its key - columns (for example `arrow_v1:000002...`), no longer its position (`row_0`). Record IDs - and source-info tokens of every Spiral source change, also with an explicit `source_id`; - `content_hash` and `pipeline_hash` of a source with an explicit `source_id` do not. - Pipeline records stored earlier for a Spiral source are not found again: the rows are - recorded once more, and no result is recomputed, because function results are found by - the row's data. -- **`record_id_column` is removed** from `SpiralDBTableSource`: the primary key identifies - the row. -- **A table without a key schema is refused**, also when `tag_columns` is given. -- **`SpiralDBTableSource` is no longer a `DBTableSource`.** -- **A read is capped at `max_rows` rows** (default 100,000); a larger read raises - `SourceTooLargeError`. Pass `limit=`, `where=`, or `max_rows=None`. -- **A saved job** stores the table id. Loading it raises `ValueError` if the saved table name - now leads to a table with another id. Loading reads no rows. - -### Added - -- `SpiralDBTableSource(..., where={column: value}, limit=n, asof=version)`: read part of a - table. SpiralDB applies the restriction before any row is transferred. Restrictions are - not part of the source's identity (ITL-186). -- `SpiralDBTableSource(..., max_rows=, batch_rows=)`, the `row_count_estimate` and `table_id` - properties, and `orcapod.errors.SourceTooLargeError` (ITL-186). -- A `SpiralDBTableSource` shown in a notebook (its `head()`) reads only its first 10 rows, - in one scan to which `max_rows` does not apply, and its header says how many rows the - table has, for example `first 10 rows of about 63,349` (ITL-186). -- `SpiralDBConnector.scan_batches(table, where=, limit=, asof=)`, `get_arrow_schema`, - `get_spiral_table_id`, `get_table_version` and `get_row_count_estimate`: a restricted scan - and metadata-only reads (ITL-186). - -### Known limitations - -- A pipeline job keeps every row it processes in memory (about 5 KB per row and node), so a - Spiral table of millions of rows can be iterated but not yet run through a job; the - `max_rows` cap guards against it (ITL-186). -- A Spiral row's key is hashed with one call per row: about 0.2 ms per row, 18 s per 100,000 - rows (ITL-186). -- A Spiral source does not poll for new rows within a run and has no watermark: a new run - scans the (restricted) table again and computes only rows it has not seen (ITL-186). diff --git a/changelog.d/ITL-578.md b/changelog.d/ITL-578.md deleted file mode 100644 index 8ce8bdfef..000000000 --- a/changelog.d/ITL-578.md +++ /dev/null @@ -1,27 +0,0 @@ -### Added - -#### `polars.DataFrame` and `polars.Series` values (ITL-578 #286) - -A function pod can take or return a `pl.DataFrame` or `pl.Series`; before, declaring one -raised `TypeError: No LogicalType or LogicalTypeFactory registered`. The logical types -`orcapod.polars.dataframe` and `orcapod.polars.series` store the value with Polars' own -`write_ipc` in `large_binary`, and a value reads back with exactly the schema it was stored -with (column order, struct field order, every dtype parameter). Chunking and sortedness flags -are not stored. `Object`, `Int128`, `UInt128` and Polars extension dtypes raise `ValueError` -naming the column when the value is stored. - -The content hash (the cache key) is labelled `polars_dataframe_v1` / `polars_series_v1` and -is strict: it covers everything a pod can observe as the value's logical content, that is -row count, column names and order, dtypes (struct field order, Enum categories, a -`Categorical`'s `Categories`), a Series' name, and values, where a null struct differs from -a struct of nulls. Chunking, sortedness flags and physical `Categorical` codes are not part -of it. A data context can leave out `column_order` and `struct_field_order` for a frame, and -`series_name` and `struct_field_order` for a Series, through the `hash_ignore` lists of -`contexts/logical_types.json`. See *Polars DataFrame and Series values* in the hashing -reference. Storing a frame in a SQLite store is not supported yet (ITL-877). - -### Fixed - -- A data context file (`contexts/logical_types.json` or a manifest) in which a JSON object - repeats a key is refused with `ContextValidationError` naming the file and the key. Before, - the last of the repeated keys silently replaced the others (ITL-578). diff --git a/changelog.d/ITL-611.md b/changelog.d/ITL-611.md deleted file mode 100644 index 1779e7990..000000000 --- a/changelog.d/ITL-611.md +++ /dev/null @@ -1,49 +0,0 @@ -### Breaking changes and migration - -#### A set of a primitive is stored as the native set container; its rows hash differently (ITL-611) - -A column declared `set[T]`, where `T` has no logical type (`set[int]`, `set[str]`, -`set[bytes]`, `set[datetime]`), is now stored under the Arrow extension type -`set[]` (for example `set[int64]`, metadata -`{"category": "set", "element_kind": "arrow"}`: the element type is in the name and is read -from the storage, `large_list`) instead of a plain `large_list`. This moves: - -- the stored Arrow type of such a column (at the top level, as `set[T] | None`, and as the - element of a list: `list[set[int]]` is stored as `list[set[int64]]`); -- the row hash of every row that holds such a column, and with it the input hash of a function - pod that takes such a column (its results are computed again), and the default `source_id` - and content hash of an in-memory source with such a column. - -Type, schema and pipeline hashes do not move, nor does anything about `list[T]` or about a set -of a logical type. No released version wrote such a row: only stores written by development -builds of v0.3 hold a set of a primitive as a plain list. Such a store is not refused: a node -serves its stored values as lists, and a run that adds a row raises when the stored and the new -form are combined. Delete the stored tables of such nodes and run again. - -### Fixed - -- **A `set[T]` of a primitive reads back as a set everywhere.** A downstream pod annotated - `set[int]` received a `list`, even inside the job that computed it; `node.as_source()` read - a `list` under a schema that said `set[int]`; and a job loaded in another process said - `list[int]`. The row `{1, 2, 3}` as `set[int]` also hashed exactly like `[1, 2, 3]` as - `list[int]`. A set now keeps its type on every path: the node, a downstream pod, - `as_source()`, `DerivedSource`, and a job loaded in a fresh process, sync and async, on - Delta and in memory. Its Python type is that of the element's Arrow type, as for a list: - `set[np.int8]` and `set[int]` both read as `set[int]`, each column keeps its own width, and - both can be declared in one process. `list[T]` of a primitive is unchanged (a plain - `large_list`). (ITL-611) -- **A job loaded in a fresh process restores `list[T]` and `set[T]` columns of a logical type** - (`set[Path]`, `set[UUID]`, `list[Path]`): their saved type could not be restored unless the - type had been used in that process first, and the stored view read the raw storage, with a - row hash that differed from the one in the process that wrote it. A saved schema now rebuilds - such a container from its identifier. (ITL-611) - -- **A stored column whose storage is not its logical type's storage raises - `LogicalTypeIdentifierError`**, naming the column and both storage types, whether or not the - type was registered in the process before (it raised a `TypeError` from Arrow when it was). - (ITL-611) - -### Known limitations - -- `frozenset[T]` is not supported: declare `set[T]`. Support is to be evaluated in ITL-1140. - (ITL-611) diff --git a/changelog.d/ITL-807.md b/changelog.d/ITL-807.md deleted file mode 100644 index d1eec85b3..000000000 --- a/changelog.d/ITL-807.md +++ /dev/null @@ -1,15 +0,0 @@ -### Changed - -- Calling a function pod, `pod(a, b)`, now drops the data columns its function does not take - (ITL-807, ITL-193). The inputs are joined as before; when the joined stream carries data - columns that are not parameters of the function, one automatic `SelectDataColumns` is - placed after the automatic join and before the pod. It is an ordinary operator step: the - graph, the hashes and the stored tables are those of a pipeline with that select written by - hand (`pod(wide.select_data_columns([...]))`, or `pod(a.join(b).select_data_columns([...]))`), so a change in an unused column no longer causes a recomputation. When nothing would - be dropped no step is added and the pipeline is the one built before. One message at `INFO` - level names the dropped columns. `pod(a, b, strict=True)` rejects an extra column, and - `pod.process(a, b)` is the exact primitive and always rejects it. Code that relied on - `pod(wide)` raising for an extra column must use `strict=True` or `process`. A missing - column for a parameter without a default is an error in every form; tag columns are never - dropped; two inputs sharing a data column still fail at the join. Operators and - `SideEffectPod` are unchanged: their `__call__` stays an alias of `process`. diff --git a/changelog.d/ITL-829.md b/changelog.d/ITL-829.md deleted file mode 100644 index cb06f5c17..000000000 --- a/changelog.d/ITL-829.md +++ /dev/null @@ -1,45 +0,0 @@ -### Breaking changes and migration - -#### `SemiJoin` keys on the shared tag columns only (ITL-1088) - -`SemiJoin` used every name its two inputs share, tag or data, as a match key. A left row was -kept only when its tags **and** its values in every shared data column equalled those of some -right row, and a name that was a tag on one side and a data column on the other was a key too. -So the operator read data values, which only filters may do. The match keys are now exactly the -columns that are tag columns of both inputs. Any other shared name (data on both sides, or a -tag on one side and data on the other) is ignored, and the right input's data columns are -never read. - -```python -left = DictSource(data=[{"id": 1, "y": 1}, {"id": 2, "y": 2}], tag_columns=["id"]) -right = DictSource(data=[{"id": 1, "y": 1}, {"id": 2, "y": 9}], tag_columns=["id"]) -SemiJoin()(left, right) -# v0.2.0: only id 1 (id and y must both match) -# now: id 1 and id 2 (only the shared tag id decides), and a SemiJoinIgnoredColumnWarning -# naming 'y' -``` - -- **What changes.** If the two `SemiJoin` inputs share a data column name, the result now keeps - every left row whose shared tags match, whatever the data values. Before, rows whose shared - data values differed were dropped, so the new result always contains the old one. Filtering a - stream by a subset of itself (every data name shared, equal values) gives the same rows as - before. A shared data column whose type or nullability differs no longer raises, because it - is not a key. -- **The warning.** Each ignored shared name is listed in a `SemiJoinIgnoredColumnWarning` - (new, in `orcapod.errors`), once per operator and set of names. Silence it by renaming or - dropping the column on the right input before the semi join, or with - `warnings.filterwarnings("ignore", category=SemiJoinIgnoredColumnWarning)`. This behaviour is - provisional in v0.3 and may become an error. -- **Unchanged.** `SemiJoin`'s identity and pipeline hash do not change. The output schema is - still the left input's. With no shared column at all, every left row is still returned. -- **Stored operator caches.** A cached `SemiJoin` node keeps its table path, so what it stored - before the upgrade is still there. With caching off it computes the new result. With - `CacheMode.LOG` it computes the new result on its next run, passes it downstream, and adds - the rows it had not stored yet. With `CacheMode.REPLAY`, and for a pipeline loaded from a - file, it serves the stored rows, which are the old, smaller result. To get the new result - there, run the pipeline once with the operator cache in `LOG` mode, or clear the operator's - cache. - -### Fixed - -- `Join`, `MergeJoin` and `SemiJoin` accept a shared tag key that is non-nullable in one input and nullable in another (ITL-829). They raised `InputValidationError` ("Type conflict for key 'id': vs int | None", or "Types int | None and are not compatible") when the graph was built. A null key matches nothing, so `Join` and `MergeJoin` now give the key `T` when any input has it `T`, and `T | None` only when every input has it nullable; `SemiJoin`'s output keeps the left input's schema as it is. Output prediction and computed Arrow fields agree on the sync and async paths. Any other type difference still raises, a data column that differs in nullability behaves as before in `Join` and `MergeJoin`, and every pipeline that worked before gives the same rows, schema and hashes. diff --git a/changelog.d/ITL-864.md b/changelog.d/ITL-864.md deleted file mode 100644 index cfc4bc3dc..000000000 --- a/changelog.d/ITL-864.md +++ /dev/null @@ -1,16 +0,0 @@ -### Fixed - -#### Delta reads return every row the database's handles wrote (ITL-864) - -Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` could -return only some of the rows (1 of 2 or 3) on both orchestrators. Handles made with `at()` share -their pending writes, and any handle's flush writes all of them, but each handle kept its own -loaded copy of the table, so a sibling kept reading the version it had loaded before the flush. -The loaded tables and the stored-id sets are now shared by every handle of one root. Every read -method (`get_all_records`, `get_record_by_id`, `get_records_by_ids`, -`get_records_with_column_value`) returns what a flush would leave: the stored rows plus the -pending rows whose record id is not stored, or `None` when there are none. Before, -`get_records_by_ids` ignored pending rows, `get_records_with_column_value` ignored stored rows -while a pending batch existed, and a pending row whose id was already stored could be returned -twice or instead of the stored row. Writes made by another process or another root on the same -path are still seen only after the table is reloaded. (ITL-864 #293) diff --git a/changelog.d/ITL-865.md b/changelog.d/ITL-865.md deleted file mode 100644 index e78f08fa6..000000000 --- a/changelog.d/ITL-865.md +++ /dev/null @@ -1,16 +0,0 @@ -### Fixed - -- An async barrier operator (`PolarsFilter`, `GroupBy`, `MergeJoin`, and the default - `async_execute` of `UnaryOperator`, `BinaryOperator` and `StaticOutputOperatorPod`) whose - input delivers no rows emits no rows and finishes, as the sync path returns an empty - stream. It raised `ValueError: Cannot materialize an empty list of rows into a stream`, so - a pipeline with a filter that matches no row upstream of such an operator completed under - `SyncPipelineOrchestrator` and failed under `AsyncPipelineOrchestrator`. (ITL-865) - -### Known limitations - -- An async channel carries rows, not a schema. Where the sync path returns an empty stream - with typed columns, the async path emits nothing, and an operator node that received no - rows reads back columns typed from its Python output schema (`int8` becomes `int64`). - Giving async execution the stream's schema is ITL-858; the read-back types are ITL-912. - (ITL-865) diff --git a/changelog.d/ITL-871.md b/changelog.d/ITL-871.md deleted file mode 100644 index bf8bd9697..000000000 --- a/changelog.d/ITL-871.md +++ /dev/null @@ -1,77 +0,0 @@ -### Added - -#### PyTorch tensors as data values (ITL-871) - -- **`torch.Tensor` is a value type**: a source column, a function pod parameter or return value, - stored, cached and re-run, sync and async, in memory and in Delta stores, and inside - `list[...]` (`Batch`, `GroupBy`, `MergeJoin`). - PyTorch is optional: support works whenever `torch` can be imported, however it was - installed; the new extra `orcapod[torch]` (`torch>=2.6`) installs it from PyPI. PyTorch 2.6 - is the first version whose restricted loader reads `uint16`, `uint32` and `uint64` - tensors. Tested with PyTorch 2.15. The PyTorch build must support the installed NumPy, because a tensor's - elements are hashed through NumPy. - `import orcapod` never imports PyTorch: the type is set up on first use, and reading a - stored tensor column without PyTorch raises `orcapod.errors.MissingOptionalDependencyError`. -- **Storage is PyTorch's own serialization.** Extension type `orcapod.torch.tensor` over - `large_binary`: one cell holds what `torch.save` writes for one tensor. A cell is read with - `torch.load(map_location="cpu", weights_only=True)`, PyTorch's restricted loader, always - with both arguments explicit and with no unrestricted fallback. Stored and read back: - dtype, shape, elements, `requires_grad`, strides (transposed and `channels_last` layouts), - the lazy conjugate and negative bits, and the class (`torch.Tensor` or - `torch.nn.Parameter`). Not stored: the device (a tensor reads back on the CPU) and the - autograd history (`grad_fn`, `.grad`; a tensor reads back as a leaf). -- **Only the passed elements are stored.** A tensor that does not cover its underlying buffer - exactly (a slice, a step slice, an expanded view) is stored as - `value.clone(memory_format=torch.preserve_format)`, so the rest of the buffer is never - written; such a view has the clone's strides and its lazy bits are resolved. The caller's - tensor is never modified. -- **The hash is strict and defined by Orcapod**, never on the stored bytes. - `TorchTensorHandler` returns `ContentHash("torch_tensor_v1", D)`, where - `D = SHA-256(lp("orcapod.torch.tensor") || E || requires_grad || conj_bit || neg_bit || ndim - || strides || lp(class name))` and `E` is the array value digest of the elements - (`docs/reference/hashing.md` §8). Dtype, shape and every element bit always count. - `requires_grad`, the memory layout, the lazy conjugate and negative bits and the class count - by default. Device and autograd history never count, because storage does not keep them. A - tensor and an ndarray with equal elements never hash the same, also inside an untyped - Python structure. -- **What the hash ignores is an option of the type.** The five optional properties - (`requires_grad`, `memory_layout`, `conj_bit`, `neg_bit`, `subclass`) can be named in the - type's ignore list: `hash_ignore` in `orcapod/contexts/logical_types.json`, empty by - default. The list is in the extension metadata (`{"ignore":[],"version":1}`), so it is part - of the logical type identifier, and a data context refuses to read a tensor column stored - under another list. An unknown option name raises when a tensor is first used. There is no - user API for the list in v0.3. -- **Dtypes.** The NumPy-shared dtypes, plus `bfloat16`, `complex32` (real then imaginary - `float16`) and `float8_e4m3fn`, `float8_e5m2`, `float8_e4m3fnuz`, `float8_e5m2fnuz`, - `float8_e8m0fnu`, hashed as raw bits under the PyTorch dtype name. -- **`orcapod.errors.ValueConversionWarning`** (a `UserWarning`) reports a tensor on a non-CPU - device or with autograd history (a `grad_fn`), which is stored as a CPU leaf tensor. It - names the function and output (or the source and column), what was converted and how to - avoid it (`value.detach().cpu()`). A function warns once per output and conversion, however - many rows it processes; `DictSource` and `ListSource` warn once per column with the count. - A leaf tensor with `requires_grad=True` and an `nn.Parameter` do not warn. After - `warnings.simplefilter("error", ValueConversionWarning)` every conversion raises (Python's - `-W` option cannot name the class: it is read before installed packages are importable). -- **Refused at write time**, naming the column and reason: sparse, mkldnn, quantized, nested - and meta tensors, named tensors, other dtypes (sub-byte and packed types, `bcomplex32`), - and subclasses other than `torch.nn.Parameter` (the restricted loader cannot read them - back). Such a subclass is not hashed as a tensor either: a subclass with its own logical - type needs its own value handler. -- **Declared outputs.** A pod declared `-> torch.Tensor` must return a tensor; an ndarray or a - number raises `OutputTypeError`. A `torch.nn.Parameter` is accepted and is stored, read - back and hashed as an `nn.Parameter`. A pod declared with a subclass must return that - subclass. -- **Registration on first use.** `LogicalTypeRegistry(deferred_logical_types=...)` and - `PythonTypeHandlerRegistry.register_deferred(...)` name a value type by its qualified class - name and extension name and load it on first use, with the configuration (the ignore list) - given in the entry. The tensor entries are in `orcapod/contexts/logical_types.json` - (`deferred_logical_types`, `deferred_value_handlers`); the `std:v1:default` data context - file is unchanged. No existing hash, schema or table path changes. - -### Fixed - -- **A declared output class with a logical type requires an instance of it (ITL-895).** The - output check required `isinstance` only for `np.ndarray`; for any other class with a - registered logical type a wrong value was caught only if its conversion failed, so a plain - tensor returned for `-> torch.nn.Parameter` was accepted. Such a value now raises - `OutputTypeError` naming the function, the output, the declared type and the returned type. diff --git a/changelog.d/ITL-879.md b/changelog.d/ITL-879.md deleted file mode 100644 index 0d18d2d04..000000000 --- a/changelog.d/ITL-879.md +++ /dev/null @@ -1,10 +0,0 @@ -### Breaking changes and migration - -- `AsyncPipelineOrchestrator(error_policy="fail_fast")` now stops the run at the first row whose function raises and raises that row's own exception, as `SyncPipelineOrchestrator` does (ITL-879). Before, the async orchestrator ignored the policy: the run skipped the row, computed the others and returned. An async run with `fail_fast` that used to finish now raises; pass `error_policy="continue"` (the default) to keep the old behaviour. Rows run concurrently, so the row raised is the first to fail, which is the first in input order only with `max_concurrency=1`. Every stored row is a correct result of its input row and the failing row is never stored; which other rows are stored before the run stops is not specified (see `docs/concepts/function-pods.md`, "Stopping at the first failed row"). -- An async run raises the failing node's own exception, never the task group's `ExceptionGroup` (ITL-879). This covers a side-effect pod with `on_error="raise"`, an operator that rejects its inputs, and any other node failure, as store errors already were. When several nodes fail before the run stops, one exception is raised and the others are added to it as notes (`error.__notes__`): a store error (`StoreError`) first, then a stored row that cannot be decoded as its class (`TypeIdentityError`), as before; among the other failures, the one of the node that comes first in the pipeline. Replace `except* SomeError` or `except ExceptionGroup` around `job.run()` / `job.run_async()` / `AsyncPipelineOrchestrator.run()` with `except SomeError`. -- `FunctionNodeProtocol.async_execute`, `FunctionJobNode.async_execute` and the function pods' `async_execute` take a keyword-only `error_policy` (default `"continue"`), as `execute` already did. A class implementing `FunctionNodeProtocol` must accept it. - -### Fixed - -- Under the async orchestrator a function node whose run stops on a row failure now calls the observer's `on_node_end` before raising, as the sync orchestrator does (ITL-879). -- A function node that the async orchestrator cancels because another node failed now commits the results it wrote before the cancellation propagates, as a failing node does, and then holds no rows: it reads as a node with no run, as a node whose run raised (ITL-879, ITL-1093). Before, its ephemeral results were never committed while the job committed its pipeline records, and on a Delta store its results were not committed ahead of its pipeline records. diff --git a/changelog.d/ITL-892.md b/changelog.d/ITL-892.md deleted file mode 100644 index f6510cb02..000000000 --- a/changelog.d/ITL-892.md +++ /dev/null @@ -1,16 +0,0 @@ -### Fixed - -#### A script that read a local Delta store no longer hangs at exit (ITL-892) - -A process that had read a Delta table through `DeltaTableDatabase` or `DeltaTableSource` could -finish its work and then never exit: it hung in Arrow's thread-pool destructor, or, on Linux with -Python 3.11 or 3.12, aborted with `terminate called without an active exception`. It happened in -up to three runs in four on macOS and about one in ten on Linux, on every supported pyarrow and -deltalake version. The cause is upstream (apache/arrow#50482, apache/arrow#54536). deltalake's -default reader goes through a filesystem written in Python, and Arrow can release one of its -buffers on a worker thread while the interpreter is shutting down. - -Orcapod now reads local Delta tables through a native pyarrow filesystem, which returns the same -schema and values. Tables on cloud storage (`s3://`, `gs://`, `az://`) still use deltalake's -reader and stay exposed until pyarrow ships the fix; see -[Troubleshooting](docs/troubleshooting.md). diff --git a/changelog.d/ITL-895.md b/changelog.d/ITL-895.md deleted file mode 100644 index 9063dafac..000000000 --- a/changelog.d/ITL-895.md +++ /dev/null @@ -1,36 +0,0 @@ -### Added - -- `orcapod.errors.OutputTypeError` identifies function-output mismatches by function, - output column, declared type and returned type, with an explicit-conversion hint. - -### Fixed - -- Function outputs are validated before iteration, hashing or storage on standalone, - cached, sync and async paths (ITL-895, ITL-901). Float-for-int truncation and unordered - set-for-list conversion are rejected. Nested values follow the same checks; exact and - structural user classes reuse their logical type compatibility, and ndarray mismatches - use the same error. Diagnostics do not expose returned mapping keys or converter values. -- Same-kind numeric values are accepted silently only when represented exactly, including - NumPy scalars for Python numbers and in-range Python integers for NumPy integer types. - Numeric narrowing, non-subclass kind changes and iterable-to-list require explicit - conversion. Subclasses are accepted unchanged in memory; declared base storage omits - subclass-only fields and their contribution to row hashes. This includes bool for int - and datetime for date, whose stored form is the declared base type. Accepted numeric - exceptions are normalized to the declared type immediately, so downstream pods see the - value that is stored. A subclass value is the exception: downstream pods in memory - receive the object as returned (a `datetime` declared as `date`, or a user-class - subclass with its extra fields), while storage holds only the declared base form. - Conforming-data hashes are unchanged. -- A declared `dict` accepts only a `dict` instance (other mappings such as - `MappingProxyType` need `dict(value)`), and a declared `Literal` accepts only its - members, matched by type and value with no numeric exception. - -### Known limitations - -- Previously stored rows remain unchanged and are not automatically recomputed or repaired. - Python-value inputs (sources and datagrams) remain outside output validation (ITL-895). -- The declared type's logical type decides what is stored. A function that reads fields its - declared parameter type does not have (the extra fields of a subclass returned for a base - type, or the time of a `datetime` declared as `date`) gets results that can depend on the - execution path and can be reused across values that differ only in those fields - (ITL-978). diff --git a/changelog.d/ITL-896.md b/changelog.d/ITL-896.md deleted file mode 100644 index c96348a85..000000000 --- a/changelog.d/ITL-896.md +++ /dev/null @@ -1,12 +0,0 @@ -### Fixed - -#### An operator node returns its rows after an async run (ITL-896) - -After a job ran on `AsyncPipelineOrchestrator`, an operator node (`job.nodes[label]` for -`Join`, `MergeJoin`, `SemiJoin`, `Batch`, `GroupBy`, `MapTags`, `MapData`, `PolarsFilter` and -the column-selection operators) returned no rows from `as_table()` and `iter_data()`, without -an error. It now returns the same rows as after a sync run, whatever `materialize_results` -is, and a later run replaces those rows even when it produces none. With the default -`cache_mode=OFF` (the only mode a job sets) an operator node keeps its output in memory and -stores nothing, so `get_all_records()` returns `None` for it on both orchestrators. -(ITL-896) diff --git a/changelog.d/ITL-897.md b/changelog.d/ITL-897.md deleted file mode 100644 index ba186c6e0..000000000 --- a/changelog.d/ITL-897.md +++ /dev/null @@ -1,110 +0,0 @@ -### Breaking changes and migration - -#### pandas values: a strict, content-based hash; column order is identity (ITL-897, ITL-887) - -A `pandas.DataFrame` or `pandas.Series` value now hashes under its own label, -`pandas_dataframe_v1` / `pandas_series_v1`, instead of as an Arrow table (`arrow_v1`). -Storage and hashing are separate: a value is stored with pyarrow's own conversion (unchanged), -and the hash is defined by Orcapod and is about the content of the value. The hash is strict: -everything about a value that can be changed is part of its identity, except a property -storage does not restore (developer, November 11 and 12, 2026). - -Before, these pairs of values shared a hash, so a cached pod could be handed the result -computed for the other one (for example the wrong `df.columns[0]`): - -| Property | Before | Now | -|---|---|---| -| Column order | not hashed | hashed | -| MultiIndex level order (frame and Series) | not hashed | hashed | -| Struct field order in a `pd.ArrowDtype` struct column | not hashed | hashed | -| `columns.name` / `columns.names` | not hashed | hashed | -| Dtype of the column axis, per level (an `int64` versus an `object` Index of the same labels) | not hashed | hashed | -| Column label type (`0` versus `"0"`), MultiIndex columns versus flat labels | not hashed | hashed | -| Series name type (`0` versus `"0"`); a frame versus its one-column Series | not hashed | hashed | -| The exact Arrow type of a non-nested `pd.ArrowDtype` column (`binary` versus `large_binary`, `null` versus `large_string`) | not hashed | hashed, as storage restores it (a nested type, and a decimal, which storage restores as `object`, are not distinguished; ITL-1054 decides nested types) | -| pandas dtype where the Arrow values coincide (`Int64` versus `int64`, `boolean` versus `bool`, `string` versus `object`) | not hashed | hashed | -| Categorical versus string; the category list (unused categories included), category order, `ordered` (ITL-887) | not hashed | hashed | -| `attrs` | not hashed | hashed, as the JSON object storage keeps | -| Values, row order, index values, index names and dtype, column dtype, Series name, `-0.0` versus `0.0` | hashed | hashed | -| `flags`, `RangeIndex` versus an equal `int64` index, an index's `freq`, unused MultiIndex levels, `string[python]` versus `string[pyarrow]`, key order of dicts in an `object` column | not hashed | never hashed: storage does not restore them (a `dict` is unordered) | -| Type of an index level name (`1` versus `"1"`) | hashed | never hashed: storage writes an index name as text, so both read back `"1"` | -| Types of column labels of mixed type (`0` beside `"b"`) | not hashed | never hashed: storage writes labels of mixed type as text | -| A masked, string or Arrow dtype of an index level (`Int64` versus `int64`) | not hashed | never hashed: storage restores an index level as NumPy | -| The dtype of a categorical's categories where storage changes it (`object` categories `[1, 2]` versus `int64`) | not hashed | never hashed: storage restores the categories from the Arrow dictionary | - -A value read back from a store now always hashes like the value that was written. Before, an -`object` column of non-strings (`[1, 2]` reads back `int64`) hashed differently after the -round trip; such a column is now hashed in the form storage restores. The same holds for an -`object` column of lists or dicts (a list `[1, None]` reads back `[1.0, nan]`) and for index -level names (the name `1` reads back `"1"`). - -Storing a pandas value that storage would corrupt now raises `ValueError`: column labels -and index names that collide once written as text (the index name `1` with a column `"1"`; -integer names of MultiIndex levels), and distinct labels that read back equal (the labels -`True` and `False`). Before, such a value was written and read back without its index (the -levels came back as columns) or with duplicate labels. - -`attrs` must be JSON. `attrs` with an entry JSON cannot write (a `set`, a tuple as a key) -are refused by the hash and by storage with `ValueError` naming the key, on every supported -pyarrow version. Before, pyarrow dropped the whole `attrs` with a warning or raised -`TypeError`, depending on its version. A key that is not text is still accepted and is -written as text (`{1: "x"}` reads back `{"1": "x"}`). - -The hash is `sha256(descriptor + b"\x00" + D)`: `D` is the Arrow hasher's ordered digest -(starfix 0.5 with `ordered_columns` and `ordered_struct_fields`) of the value's Arrow table -with positional field names, and the descriptor is canonical JSON in an Orcapod-owned grammar -of `attrs`, typed labels, `columns.names`, the column-axis dtype, index levels and pandas -dtypes. The definition and -the grammar are in `docs/reference/hashing.md`, section 10. A label or dtype outside the -grammar raises `ValueError` naming it. - -#### pandas logical types are named `orcapod.pandas.dataframe` / `orcapod.pandas.series` (ITL-897) - -The Arrow extension names of the two pandas logical types gain the `orcapod.` prefix (they -were `pandas.dataframe` / `pandas.series`), and the metadata is `{"ignore":[],"version":1}`: -the version, and the list of properties the data context configures the value hash to -ignore. Nothing is ignored by default. `hash_ignore` in the logical type registration file -can name `column_order`, `struct_field_order` and `attrs` for a frame, and `series_name`, -`struct_field_order` and `attrs` for a Series; two lists are two logical types, and a column -stored under another list is refused on read. The old names -are not registered any more: a column stored under them is refused on read, never read as -the current types or as raw bytes. - -**What moves.** The hash of every pandas value, and with it the data hash and the cache keys -of every row that holds a pandas value. Because the name is part of the logical type -identifier, the type hash of `pd.DataFrame` / `pd.Series`, the schema hash, the pipeline hash -and so the table paths and system-tag column names of every node with a pandas column move -too, and of every node downstream of one. The stored bytes of a pandas value are unchanged. -v0.3 already recomputes everything a v0.2 store holds (ITL-910), and v0.2 tables are still -found by name, reported and left unchanged, so this adds no migration step for a released -store. A store written by an unreleased v0.3 development build computes nodes with a pandas -column once more, in new tables; the older tables stay in place, unchanged, and are not -reported. - -#### Public interface: `ArrowHasherProtocol.hash_table_ordered` (ITL-897) - -`ArrowHasherProtocol` gains a required method, `hash_table_ordered(table)`, implemented by -`StarfixArrowHasher`: the same procedure as `hash_table`, with column order and struct field -order at every depth part of the digest (`struct_field_order=False` leaves struct field -order out). `hash_table` is unchanged and stays order -independent (streams, datagrams, rows). A custom Arrow hasher must add the method. -`PANDAS_DATAFRAME_HASH_LABEL` and `PANDAS_SERIES_HASH_LABEL` in -`orcapod.hashing.pandas_value_hash` define the two labels, and -`PANDAS_IDENTITY_PROPERTIES` there lists every property and whether it is part of the hash. - -#### pyarrow 22.0.0 or newer is required (ITL-897) - -The minimum pyarrow version is raised from 20.0.0 to 22.0.0. pyarrow 20 and 21 do not -restore `DataFrame.attrs` when a frame is read back, so a frame with `attrs` would hash -differently after it was stored; from 22.0.0 `attrs` is restored. The locked version -(23.0.1) is unchanged. - -### Known limitations - -- A `pa.Table` or `pa.RecordBatch` given to the semantic hasher as a value still hashes - without regard to column order and struct field order (`arrow_v1`). Such a table cannot be - a pod input or output (ITL-897, second part). -- pandas is capped below 3 (`pandas>=2.2.3,<3`) for v0.3.0. The pandas value hash and its - round trip through storage are verified on pandas 2 only; pandas 3 changes the default - string dtype and other behaviour the hash depends on, and has not been checked. ITL-1054 - lifts the cap. diff --git a/changelog.d/ITL-904.md b/changelog.d/ITL-904.md deleted file mode 100644 index f8ee96bde..000000000 --- a/changelog.d/ITL-904.md +++ /dev/null @@ -1,48 +0,0 @@ -### Fixed - -#### A loaded job can read its results (ITL-904) - -A job loaded with `PipelineJob.load` can read its results. A function node's `as_table()` and -`iter_data()` return the rows the saved run stored, with their stored types (they used to raise -"no function pod and no database attached" while `get_all_records()` worked). Without a store, -reading says to pass `PipelineJob.load(path, database=...)` or call `job.bind(database=...)`. `save()` -now records whether the store held records of each function node (`run.node_has_stored_records`, -an optional field; the format version is unchanged). A node saved with records raises, naming -the store, when the attached store holds none of them, instead of returning no rows. When an -entry has several records (it was recomputed), a function node now reads the one with the -greatest recomputation index, on a live node too; it used to keep whichever record the store -returned last. Loading a job in which an operator consumes a function node no longer crashes -with `InputValidationError`, and a loaded node's `output_schema()` reports Python types; a saved -type it cannot restore raises an error naming the column. Registered logical types such as -`Path` restore normally, so downstream operators remain available. Saved -`timestamp`, `duration`, `time` and `decimal` types are now read back. Reading a loaded operator -node without a readable cache raises instead of returning an empty table; operators can store -outputs when caching is enabled. In-memory sources (`DictSource`, `ListSource`, -`ArrowTableSource`, `DataFrameSource`) are -logged at INFO with the remedy instead of a "skipping" warning. A loaded job stays read-only and -cannot be run again. See *Saving and loading a job* in the function-pods guide. (ITL-904) - -Stored-record expectations count only rows of the node's saved data context. Operator -descriptors without a reconstructed operator still lack the saved runtime identity needed -to find cached rows; the runtime-descriptor change is deferred to ITL-977. Direct execution -of loaded operator nodes is rejected. A newer unavailable ephemeral result yields an -`EmptyData` placeholder instead of exposing an older persistent result. - -### Breaking changes and migration - -- Saved pipeline/job schema text now includes the full logical type identifier as - `extension`, including nested logical types. Loading validates it - against the registered type; incompatible or malformed identifiers raise - `LogicalTypeIdentifierError` naming the node and column. Name-only extension text from - older saved files is rejected with an earlier-version error. Read those files with the - version that wrote them, and save a new job from the original pipeline under this version. - Stored table layouts, value decoding and hashes are unchanged. Unregistered logical types - are reported by node and column; affected operators load as read-only stubs. (ITL-904) - -### Changed - -- When persistent and ephemeral results exist for the same input, reads select the - greatest recomputation index across both stores; persistent rows retain precedence - for equal indices. Loaded REPLAY operators report an absent cache table explicitly, - loaded functions reject `run()`, and loaded jobs report `is_runnable()` as false. - Source schema restoration uses the saved data context's type converter. (ITL-904) diff --git a/changelog.d/ITL-905.md b/changelog.d/ITL-905.md deleted file mode 100644 index 1546fb79a..000000000 --- a/changelog.d/ITL-905.md +++ /dev/null @@ -1,41 +0,0 @@ -### Breaking changes and migration - -#### A `dict` or `set` value hashes the same whatever order it was built in (ITL-905) - -A `dict[K, V]` value is stored as `large_list>` and a plain `set[T]` -(element type without a logical type) as `large_list`. Both were stored in Python -iteration order, so `{"a": 1, "b": 2}` and `{"b": 2, "a": 1}` had different row hashes, -and the same `set[str]` had a different row hash in every process (`PYTHONHASHSEED`). -Orcapod now stores dict entries sorted by key and set elements sorted, wherever a Python -value becomes Arrow; see the hashing reference, section 1e, for the order. A NaN key or -element, values that cannot be compared, and two keys that convert to the same stored -value raise `CanonicalOrderError` (a `ValueError`) naming the column. A datagram built from -Python values (and values set with `update` or `with_columns`) holds its dicts in the same -order, so a newly computed pod output and one served from the cache iterate alike. Dicts -nested in dataclass or pydantic instances are the exception: a fresh instance keeps insertion -order, while a read-back instance uses key order. An Arrow table is still hashed exactly as -stored: a key/value list that arrives in Arrow form is never re-sorted. - -Set values passed to `update` after Arrow materialization, or to `with_columns` with an -explicit Arrow list field, are also stored in canonical order. The existing Arrow field -(width, nullability and metadata) is preserved; actual list values keep their supplied order. - -Which hashes move because of this change, on top of the v0.3 hash step (ITL-910, ITL-945), -which moves every stored hash: only those of values holding a dict whose insertion order was -not key order, or a `set[T]` of two or more elements whose iteration order was not sorted -(for `str` and `bytes` that order changed with every process, so it had no stable value). -For such a row, its data hash moves, and with it the function pod cache key, the pipeline -record and entry ids, and the default `source_id` (with its per-row record ids) of a -source built without an explicit `source_id`. Schema hashes, `pipeline_hash` values and -explicit `source_id`s do not move for this reason, and a dict already built in key order is -stored byte-for-byte as before if every nested unordered container is also canonical. -In the golden hash suite exactly two entries moved: -`pipeline:index_pick.index.row[{"subject":"s1"}].data_hash` and `...s2...`, whose rows hold -`meta={"k": ..., "j": ...}`. - -What a `0.2.0` store sees (checked against a Delta store written by `0.2.0`): nothing -changes because of this fix. v0.3 reads no table `0.2.0` wrote (every table is at a new path, -see "Data context `std:v1:default`", ITL-944), so every input is computed again whether its -dict was in key order or not, and no row stored in `0.2.0`'s insertion order is ever served. -The `0.2.0` tables stay where they are, untouched. Every dict and plain set a v0.3 store -holds from a pod output or a source built from Python values is in canonical order. diff --git a/changelog.d/ITL-906.md b/changelog.d/ITL-906.md deleted file mode 100644 index ac9b37d6e..000000000 --- a/changelog.d/ITL-906.md +++ /dev/null @@ -1,33 +0,0 @@ -### Breaking changes and migration - -#### The async orchestrator names system-tag columns as the sync orchestrator does (ITL-906, ITL-885) - -`MergeJoin`, `Batch` and `GroupBy` put their inputs' pipeline hashes into the names of the -system-tag columns they write. When one of these operators read an input that is not a -source (an operator's or a function pod's output), `AsyncPipelineOrchestrator` used a -different hash than `SyncPipelineOrchestrator`, so the same graph got different system-tag -column names, and different record ids, depending on which orchestrator ran it. The async -orchestrator now uses the input's pipeline hash, and its names and record ids are the ones -the sync orchestrator has always produced. Nothing a sync run produces changes, and graphs -where these operators read a source directly are unaffected. - -Every operator's `async_execute` now requires the keyword argument `input_pipeline_hashes`, -one pipeline hash per input, as the operator node always passes it; a missing argument raises -`TypeError`. The operators that use the hashes (`Join`, `MergeJoin`, `Batch`, `GroupBy`, -`PolarsFilter`, and any operator on the default barrier path) raise `ValueError` when the count -does not match the inputs; the per-row operators and `SemiJoin` take the argument without -using it. Before, these -operators fell back without it to the schema-only hash, and `Join`'s async path to names built -from the input positions (`::0`, `::1`, ...), neither of which the sync path uses. Code that -calls an operator's `async_execute` directly must pass the inputs' pipeline hashes. - -This also fixes ITL-885: a store written by one orchestrator can now be extended by the -other. Before, the function node downstream of such an operator wrote two sets of -system-tag columns into one pipeline table, and the second run failed while storing its -results. - -Migration: stores written by v0.2 are not affected, because v0.3 never reads a table -written by v0.2. Only a store written by an async run of an unreleased v0.3 build, for a -graph with `MergeJoin`, `Batch` or `GroupBy` after an operator or a function pod, holds the -old names: a later run on that store fails when it writes the pipeline table of the function -node downstream. Delete that table, or use a fresh store, and run again. diff --git a/changelog.d/ITL-910.md b/changelog.d/ITL-910.md deleted file mode 100644 index bf88f31d7..000000000 --- a/changelog.d/ITL-910.md +++ /dev/null @@ -1,38 +0,0 @@ -### Breaking changes and migration - -#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910) - -**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1` -to `arrow_v1`.** starfix 0.4 (hash format 1) lost information for some inputs, so different -data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and -`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean -columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash -format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept. - -- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default - `source_id`s and the `record_id`s and source content hashes built from them, function - input-data hashes, pipeline base and entry ids, operator and cached-source record hashes, - invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The - hash label in front of each (`arrow_v1:000002...`) shows the new format. Pipeline hashes, - schema hashes, system-tag column names and table paths under the default - `table_scope="pipeline_hash"` do not move for this reason; they move in v0.3 because of - the hash label rule (ITL-945, below). Paths that embed a content hash - (`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or - `Batch` orders members by `record_id`, the order of members can change too. -- **Results stored by 0.2.0 are not reused.** Every table v0.3 writes is at a new path and - every row names its data context; see "Data context `std:v1:default`" below (ITL-944) for - what a v0.2 store looks like to v0.3. -- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator - carrying one works, and its type is kept. A `fixed_size_list[n]` hashes differently from a - `large_list` with the same values. pyarrow's canonical extension types (`arrow.uuid`, - `arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid` - column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and - `StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846). -- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed - silently: a non-nullable field holding a null; a zero-row table with a `map` column; two - schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a - column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`, - interval, `string_view` and `binary_view` columns, union and run-end-encoded types are - still refused, now with `ValueError` instead of `NotImplementedError`. A source built from - a table with such columns raises `ValueError` naming every refused column and its type. -- Column order and struct field order are still not part of an Arrow hash. diff --git a/changelog.d/ITL-912.md b/changelog.d/ITL-912.md deleted file mode 100644 index c5a7011b3..000000000 --- a/changelog.d/ITL-912.md +++ /dev/null @@ -1,62 +0,0 @@ -### Changed - -- `head()` contract (ITL-912): `head()` and the notebook preview never compute, poll or deliver a row (they read only what is held or stored, or make a bounded table read), never call user code (a function, a predicate, a side effect) and never write. - With no rows to show, they now give the columns and Arrow types the rows would have: they - apply each operator to empty (zero-row) tables, which processes no data (a chain of N - operators applies up to N operators to empty tables, about 60 ms for 20), and a filter is not - applied at all, since its columns are its input's. A user-defined operator's `static_process` is applied to zero-row tables and must not have effects. So a pipeline whose inputs disagree on a - column's Arrow type (`int8` against `int64`) now raises from `head()` the - `InputValidationError` its run raises; the notebook preview shows it as "could not read". A - filter whose predicate does not fit its columns shows a table in `head()` and raises when it - runs. - -### Fixed - -#### A table with no rows has the types of the rows (ITL-912) - -An operator node read before its run, after `clear_cache()`, or after a run that produced no -row, returned a zero-row table built from its Python output schema: `int8` became `int64`, -a non-nullable list item became nullable, source-info columns changed nullability and the -system-tag columns were missing. A function pod's stream and a function node did the same for -their tag columns, and left out the `_context_key` and source-info columns; a side-effect -pod's stream or node that delivered no row did the same, and so did `head()` and the notebook -preview of a stream or node with no rows to show. Both orchestrators built an -operator node's input the same way when an upstream node delivered no row, so a `Join` of -such an input with a non-empty one whose key is, say, `int8` raised `InputValidationError`. - -Each of these tables now has exactly the columns, Arrow types and nullability of the rows, -on both orchestrators, and that `Join` completes with typed zero rows on both paths. Nothing -is computed, read or called to get them: an operator applies -itself to its inputs' zero-row tables, a function pod combines its input's tags with its -declared output types, a side-effect pod gives its input's, and a source gives the table or -schema it already holds. Nothing is -stored for an empty table, so no hash, stored table or path changes. - -### Known limitations - -- A zero-row table is still built from the Python output schema (`int8` reads as `int64`, - system-tag columns are absent) when an upstream cannot say its Arrow types without rows: a - `PollingSource` before its first poll and an unbound input slot or `SourceProxy` (both are - declared with a Python schema only), a `DerivedSource`, and a loaded job node with no live - upstream and no stored table. `Index` and `Pick` over zero rows still lack the system-tag - columns (ITL-985). These cases are tracked in ITL-1089. (ITL-912) -- A source-info column (`_source_*`) of an operator node's rows is nullable only when some row's - provenance token is `None`, a rule that depends on the rows. A zero-row table has no token, - so it declares the column non-nullable, as rows with a known token do; when the rows have an - unknown (`None`) token, the field's nullability differs between the zero-row table and the - rows. A row-independent rule would change the operator node's stored record hashes. Tracked in - ITL-1089. (ITL-912) - -#### Zero-row tables over non-large inputs (ITL-912) - -A source holding `string`, `binary` or `list` columns gave a function node's or operator -node's table with no rows the non-large types, while the table with rows has the large ones -(`large_string`, ...). The zero-row table now takes its input's types in the large-normalized -form, as the populated table does, and an operator node's zero-row table lists its columns in -the order its rows do (a join's system tags before its data). A source's own table is not -changed. - -A side-effect pod's stream takes its input's zero-row types in the same large form; a -`PolarsFilter()` with no predicate shows its input's types, as it passes the input through; and -a `CachedSource` over a source with no rows, with nothing stored, reads with the source's own -Arrow types and system-tag columns, as its `head()` shows them. diff --git a/changelog.d/ITL-919.md b/changelog.d/ITL-919.md deleted file mode 100644 index f7312d516..000000000 --- a/changelog.d/ITL-919.md +++ /dev/null @@ -1,52 +0,0 @@ -### Added - -- **`PipelineJob.run(commit_every_rows=100)`** — how many rows a function node processes - between commits of its results and pipeline records (an `int >= 1`). Nodes also commit when - they finish, before a failed run raises, and at the end of the job; `1` commits after every - row. A hard crash on a durable store loses at most the rows the running node processed - since its last commit, and the rerun computes only those. (ITL-919) - -### Fixed - -#### A `PipelineJob` on an in-memory store runs in time linear in its rows (ITL-919) - -A `PipelineJob` on an `InMemoryArrowDatabase` runs in time linear in its rows. Every row a -function node processed used to filter, copy or rebuild the whole stored table (the -result lookup, the recomputation-index lookup, the duplicate check and the flush, each -over a table that gained one chunk per row), so 1,000 rows took about a minute. Lookups -now use hash indexes over the stored rows and a flush appends without copying them; -stored rows, ids and hashes are unchanged. - -#### A `PipelineJob` on a Delta store runs in time linear in its rows (ITL-919) - -A `PipelineJob` on a `DeltaTableDatabase` committed every row of every function node as -its own Delta version (a MERGE that reads the whole table) and re-read the stored table for -each per-row lookup, so 1,000 rows took about eight minutes. Function nodes inside -`job.run()` now commit every `commit_every_rows` rows (default 100, see *Added*); a batch -whose record ids are all new is appended instead of merged; and lookups are answered from -an indexed in-memory copy of the committed rows, read once and extended with each commit. -Stored rows, record ids, hashes, paths and schemas are unchanged. Behaviour changes: a store -failure that only a commit reveals (an unreadable table, a failed commit) surfaces at the -node's next commit point and names the node but no row, and the rows computed since the -last commit are computed again on the next run; `commit_every_rows=1` restores per-row -surfacing. A `DeltaTableDatabase` object keeps answering from the table version it loaded -until its own next commit (as before; it no longer re-reads the table per lookup). - -#### An asynchronous run whose node fails raises instead of hanging (ITL-932) - -On the asynchronous orchestrator, a node failure while more rows were in flight than the -channel buffers hold (about 16 to 64 rows) hung the run forever: the failure cancelled the -readers, and closing a full channel waited for room that never came. Closing a channel no -longer blocks; readers still receive every item sent before the close. - -#### An asynchronous rerun of an ephemeral-result node recomputes its rows (ITL-923) - -Rerunning a node with `NodeConfig(is_result_ephemeral=True)` on the asynchronous -orchestrator, after its ephemeral results were gone, neither recomputed nor recorded the -rows, and reading the node afterwards raised `EmptyDataAccessError`. It now recomputes them, -as the synchronous orchestrator does. - -#### A failed side-effect completion lookup names the row - -A `StoreReadError` from a side-effect pod's check of earlier deliveries now names the pod -and the row, as a failed write of the delivery record does. diff --git a/changelog.d/ITL-926.md b/changelog.d/ITL-926.md deleted file mode 100644 index c6f11b913..000000000 --- a/changelog.d/ITL-926.md +++ /dev/null @@ -1,55 +0,0 @@ -### Breaking changes and migration - -#### A Spiral source's default `source_id` is `//
` (ITL-926) - -`SpiralDBTableSource` used the bare table name as its default `source_id`. Tables with the -same name in different projects (the EDI sample projects all share their table names) then -had the same `content_hash`, record IDs and source-info tokens. In one store, a job on one -project read back the other project's pipeline records along with its own (cached results -were never mixed up: they are keyed on the row's data). The default is now the project id, -the dataset and the table name, for example `edi-sample-v1-855632/default/session`. - -For a Spiral source built without `source_id`, the `content_hash`, every record ID and every -source-info token change, so pipeline records stored by 0.2 for such a source are not found -again. The rows are recorded once more and no result is recomputed, because function results -are keyed on the data. For example, table `subject` (key `subject_id`, one row) in project -`edi-sample-v1-855632`: - -- `source_id`: `subject` becomes `edi-sample-v1-855632/default/subject`. -- `content_hash`: `semantic_v0.1:0380725acb90...e7f4f6` becomes - `semantic_v0.1:3a659ed610bb...29a93`. -- Row 0's record ID: `18d1efa53fbc52b6a40f406eca35ca81` becomes - `096c4e22981c52daa2b088bffe729ee7`. - -`pipeline_hash` does not change (it depends only on the schema), nor does anything for a -source given an explicit `source_id`. To keep 0.2 identities, pass the old value: -`SpiralDBTableSource(project, "subject", source_id="subject")`. Jobs saved by 0.2 store -their `source_id` explicitly and keep it when loaded. - -### Fixed - -#### SpiralDB table source (ITL-926) - -- The `spiraldb` extra requires `pyspiral>=0.16.33` (was `>=0.14.0`). pyspiral 0.14.9 does not - use the login saved by a current `spiral login`; 0.16.33 does, and keeps every call - `SpiralDBConnector` makes. -- `SpiralDBConnector.get_column_info` and `iter_batches` normalize every column to its large - Arrow variants at every nesting level (`list` becomes `large_list`), - so both report the same types (ITL-40). Before, only top-level `string` and `binary` - columns were converted. Widths, nullability and nested field names are unchanged, and so - are all hashes. -- Building any source converts only its `record_id_column` to Python, not the whole table: - about half the peak memory (200,000 rows: +210 MB down to +115 MB). Tokens, record IDs and - hashes are byte-identical. -- SpiralDB keeps no non-null flag except on list items, so a Spiral column arrives as `T` - only when the table carries Orcapod's nullability record (ITL-936, under Added); otherwise - every column is `T | None`. The pyspiral fake now behaves like the server in this respect. -- A source whose table has a column the Arrow hasher refuses (for example `map`) raises - `ValueError` naming every such column and its Arrow type, chained to the hasher's error. - Before, it raised the hasher's bare `Unsupported leaf type: ...`, which named no column. - `fixed_size_list` columns are hashed since starfix 0.5 (ITL-910). -- `SpiralDBTableSource` errors name the project and dataset searched. Before, a wrong - project, dataset or table all gave `Table 'x' not found in database.` -- The docstrings give the real location of the saved `spiral login` - (`~/Library/Application Support/pyspiral/` on macOS). `docs/concepts/sources.md` has a - SpiralDB section: setup, discovery, nullable columns, and current limits. diff --git a/changelog.d/ITL-928.md b/changelog.d/ITL-928.md deleted file mode 100644 index d14762b01..000000000 --- a/changelog.d/ITL-928.md +++ /dev/null @@ -1,15 +0,0 @@ -### Breaking changes and migration - -#### The Delta store keeps every column's Arrow type; rebuild a Delta store written by an earlier v0.3 build (ITL-928) - -Delta Lake has no unsigned integer, no `timestamp` in seconds, milliseconds or nanoseconds, no zone other than UTC, and no `date64`, `time` or `duration` type. `DeltaTableDatabase` used to let the Delta writer convert such a column, so a read gave back another type: `uint32` as `int32` (and a value above the signed maximum failed the write), `timestamp[s]`, `[ms]` and `[ns]` as `timestamp[us]` (losing nanosecond digits), `timestamp[us, tz=America/New_York]` as UTC; `date64`, `time` and `duration` columns could not be stored at all. A tag read back with another type changes the record ids computed from it, so on a Delta store a re-run over such a tag wrote the downstream nodes' pipeline rows a second time (a fresh job then read 6 rows for 3). - -Now such a column's original Arrow type is recorded in the Delta field's metadata, an unsigned integer is written as the signed integer of the same width bit for bit, and `timestamp[ns]`, `date64`, `time` and `duration` are written as their integer value; every read casts the column back to its recorded type. Nested columns (`list`, a struct with a `timestamp[ms]` field) are covered. Columns of every other type, and logical-type columns, are stored as before. - -A Delta store written by an earlier unreleased v0.3 build has no recorded types: its tables keep reading back Delta's types, a re-run over such a tag still writes duplicate pipeline rows, rows it already duplicated stay, a function node whose rows mix rows read from it with newly computed rows raises `InputValidationError` naming the column and both types, and appending a `timestamp[ns]` column to one of its tables is refused (the stored `timestamp[us]` differs from the new integer form). Rows with an unsigned column are still appended to such a table's signed column, as before, and read back signed; a value the signed type cannot hold is refused with `InputValidationError` and nothing is written. Nothing is repaired automatically: rebuild such a store by running the pipeline into a new store. Stores holding none of these column types are not affected. No content hash, pipeline hash, record id or table path changes. - -### Fixed - -- A function node's `as_table()` keeps the Arrow type of every tag column. Before, it rebuilt the table from Python values, so after `job.run()` an `int8` or `uint32` tag came back as `int64`, `float32` as `double`, a naive `timestamp[us]` as `timestamp[us, tz=UTC]` and `date64` as `date32`, and a `time`, `duration`, `decimal`, `uuid` or `Path` tag made it raise. Joining a function node's output with another source on such a tag no longer fails with "Join inputs disagree on the Arrow type". `as_table(all_info=True)` of a function node now also has the `_source_` and `_context_key` columns, as the same pod's stream outside a job does. When two of a node's rows disagree on a column's Arrow type, `as_table()` raises `InputValidationError` naming the node, the column and both types, instead of returning a type neither has. -- The table of a function pod's stream (the pod applied outside a job) no longer raises for a `uuid` or `Path` tag ("Expected bytes, got a 'UUID' object"). -- On a Delta store, a re-run over a tag of an unsigned, non-microsecond or zoned timestamp type writes no pipeline row again, and a fresh job, a loaded job, an operator node's replay and a `CachedSource` read the tag's own type (see above). diff --git a/changelog.d/ITL-935.md b/changelog.d/ITL-935.md deleted file mode 100644 index 2b98956f0..000000000 --- a/changelog.d/ITL-935.md +++ /dev/null @@ -1,44 +0,0 @@ -### Breaking changes and migration - -#### Logical type identifiers: every built-in type writes one, and all of it is hashed (ITL-935) - -A logical type is now identified by its Arrow extension name **and** its Arrow extension -metadata, and the whole identifier is part of the column's data type -(`docs/concepts/logical-type-identifiers.md`). The identifier belongs to whoever implements -the type and changes only when an existing value would be stored or hashed differently. - -- **Built-in types write an identifier.** `orcapod.path`, `orcapod.upath`, `orcapod.uuid`, - `orcapod.file`, `orcapod.directory`, `pandas.dataframe` and `pandas.series` now write the - extension metadata `{"version":1}` (they wrote none); `orcapod.ndarray` already did. - `list[T]` and `set[T]` of a logical type (the native containers) have no version of their - own, by design: their metadata carries the element's name and new metadata, so their - identifier changes exactly when the element's does. -- **The whole identifier is hashed.** A type hash, a schema hash and a function signature - used to contain the logical type's name only. They now contain - `;` (`LogicalTypeIdentifier.text`). The hashed cell - of a value-hashed type (UUID, file, directory, array, pandas) starts with the same text, - where it used to start with the name alone, written with `:` for `.`. -- **Fixed: `[]` and `[None]` hashed the same in a list or set of a value-hashed type.** The - hashed cell of a `list[T]` / `set[T]` column (T a UUID, file, directory, array or pandas - type) joined its element hashes with a separator and wrote nothing for a null element, so - an empty list, `[None]` and `[None, None]` shared one hash. The cell now holds the element - count and, per element, a null marker or a length-prefixed element hash. -- **What moves, once, in v0.3.** For every node whose schema or function signature has a - column of a logical type (including dataclass and pydantic columns): its schema, function - and pipeline hashes, hence the paths of its stored tables, and the hashes of the rows that - hold such values. Together with the other v0.3 hash changes this means no result stored by - v0.2 is reused; nothing stored is changed or deleted. **Nodes with only plain types are - not affected by this change.** -- **Reading.** A stored column is read only if its stored metadata is one the registered - type declares readable (`readable_extension_metadata()`, new on `LogicalTypeProtocol`, - with a default in `BaseLogicalType`). Otherwise `LogicalTypeIdentifierError` (a - `ValueError`) names the column, the stored identifier and what this installation reads. - Before, a column read as plain storage was wrapped as the registered type whatever its - stored metadata said. -- **Data written before v0.3 stays readable.** The seven types above read columns stored - with empty metadata as version 1 (the layout is the same), for example a Delta table - written by v0.2 and opened as a source. -- **For authors of logical types.** `make_arrow_extension_type` takes `also_reads`: older - metadata values the type still decodes. An extension name may not contain `;` - (`register_logical_type` raises `ValueError`). A custom logical type that does not derive - from `BaseLogicalType` must implement `readable_extension_metadata()`. diff --git a/changelog.d/ITL-936.md b/changelog.d/ITL-936.md deleted file mode 100644 index 06a2feb21..000000000 --- a/changelog.d/ITL-936.md +++ /dev/null @@ -1,27 +0,0 @@ -### Added - -- **SpiralDB nullability record** (ITL-936). SpiralDB reports every top-level column (key - columns included) and every struct field as nullable, whatever was written; only list items - keep a non-null flag. `SpiralDBConnector` now keeps a *nullability record* of each table's - non-nullable columns and nested fields: an entry in the table's key-value metadata, not a - data row (key `__orcapod_nullability__`, format `orcapod.nullability` version 1, chunked - under SpiralDB's 8 KiB entry limit). It writes the nullability record on a table's first - write, and `get_column_info` and `iter_batches` restore it, so a `SpiralDBTableSource` over - such a table gives `T`, not `T | None`, and a function pod can take `x: float`. Every batch - of such a table is checked: a null in a listed column raises - `orcapod.errors.NullInNonNullableColumnError` naming the table and column. A write holding a - null in a non-nullable column is refused before anything is written (SpiralDB stores a null - written to a non-nullable struct field as `0`), and a write whose non-nullable columns differ - from the nullability record raises `NullabilityRecordMismatchError`. An unreadable - nullability record raises `NullabilityRecordError`. - `orcapod.databases.write_nullability_record` and `read_nullability_record` let other writers - give a table its nullability record. Tables without one read exactly as before (same schema - and hashes); a table that gains one has a different schema, `pipeline_hash` and - `content_hash`, so write it before building pipelines on the table. A - `ConnectorArrowDatabase` on SpiralDB gives new result tables a nullability record, so results - read back with their written nullability; existing result tables are not changed. -- **SpiralDB null structs are refused** (ITL-936). SpiralDB cannot store a null struct column: - pyspiral refuses one, and with `push_down_nulls=True` it reads back as a struct of null fields - (live, pyspiral 0.16.33, top-level struct columns). `SpiralDBConnector.upsert_records` now refuses a null struct at any depth - with `orcapod.errors.UnsupportedValueError` naming the table and column, before anything is - written; it never passes `push_down_nulls`. diff --git a/changelog.d/ITL-939.md b/changelog.d/ITL-939.md deleted file mode 100644 index e42804a65..000000000 --- a/changelog.d/ITL-939.md +++ /dev/null @@ -1,12 +0,0 @@ -### Added - -- **`DerivedSource(origin, tags_as_data={tag: data_name})`** — a source derived from - another source or from a job node that keeps the origin's tags and also offers chosen tag - values as data columns, so a key (for example a session start time) can be an input to a - function pod. Also available as `node.as_source(tags_as_data=...)`. The copy has the tag's - exact Arrow type and nullability. The result is a new root source with its own provenance; - its default `source_id` is `tags_as_data: