diff --git a/changelog.d/ITL-578.md b/changelog.d/ITL-578.md new file mode 100644 index 000000000..8ce8bdfef --- /dev/null +++ b/changelog.d/ITL-578.md @@ -0,0 +1,27 @@ +### Added + +#### `polars.DataFrame` and `polars.Series` values (ITL-578 #286) + +A function pod can take or return a `pl.DataFrame` or `pl.Series`; before, declaring one +raised `TypeError: No LogicalType or LogicalTypeFactory registered`. The logical types +`orcapod.polars.dataframe` and `orcapod.polars.series` store the value with Polars' own +`write_ipc` in `large_binary`, and a value reads back with exactly the schema it was stored +with (column order, struct field order, every dtype parameter). Chunking and sortedness flags +are not stored. `Object`, `Int128`, `UInt128` and Polars extension dtypes raise `ValueError` +naming the column when the value is stored. + +The content hash (the cache key) is labelled `polars_dataframe_v1` / `polars_series_v1` and +is strict: it covers everything a pod can observe as the value's logical content, that is +row count, column names and order, dtypes (struct field order, Enum categories, a +`Categorical`'s `Categories`), a Series' name, and values, where a null struct differs from +a struct of nulls. Chunking, sortedness flags and physical `Categorical` codes are not part +of it. A data context can leave out `column_order` and `struct_field_order` for a frame, and +`series_name` and `struct_field_order` for a Series, through the `hash_ignore` lists of +`contexts/logical_types.json`. See *Polars DataFrame and Series values* in the hashing +reference. Storing a frame in a SQLite store is not supported yet (ITL-877). + +### Fixed + +- A data context file (`contexts/logical_types.json` or a manifest) in which a JSON object + repeats a key is refused with `ContextValidationError` naming the file and the key. Before, + the last of the repeated keys silently replaced the others (ITL-578). diff --git a/design_issues/ITL-578-PL1.md b/design_issues/ITL-578-PL1.md new file mode 100644 index 000000000..e6288ff11 --- /dev/null +++ b/design_issues/ITL-578-PL1.md @@ -0,0 +1,36 @@ +## `src/orcapod/logical_types/polars_type.py` — Polars values + +### PL1 — `pl.DataFrame` had no logical type, so a pod could not take or return one +**Status:** resolved +**Severity:** high +**Issue:** ITL-578 + +`@function_pod(output_keys="df") def f(n: int) -> pl.DataFrame` raised at decoration: +`TypeError: No LogicalType or LogicalTypeFactory registered for `. + +**Fix:** `LogicalPolarsDataFrame` (`orcapod.polars.dataframe`) and `LogicalPolarsSeries` +(`orcapod.polars.series`, a one-column frame) store the value in `large_binary` with Polars' own +`write_ipc` (Polars' defaults, pinned) and read it with `read_ipc` from memory (developer's +decision, November 12, 2026: storage is the library's own serialization, the hash is defined +separately from the value's content). Every supported dtype reads back exactly; chunking and +sortedness flags are not stored. `Object`, `Int128`, `UInt128` and Polars extension dtypes +are rejected at write time, naming the column. The stored bytes are never hashed, +so `PolarsDataFrameHandler` / `PolarsSeriesHandler` hash +`sha256(Orcapod dtype descriptor + starfix digest of the values)` instead, under the labels +`polars_dataframe_v1` / `polars_series_v1` (developer's decision, November 10, 2026). +Everything a pod can observe through the public Polars API as the value's logical content is +identity (developer's decision, November 11, 2026, reversing the November 10 ruling that +order was not): the descriptor lists columns and struct fields in order at every depth, a +`Categorical` carries its `Categories` (name, namespace, physical width), and the values are +digested by starfix 0.5 in its ordered mode (`ordered_columns`, `ordered_struct_fields`, +ITL-898), which also tells a null struct from a struct of nulls. Physical properties stay +out: chunking, sortedness flags (not stored), physical `Categorical` codes, Arrow layout. +Both types carry the identifier metadata `{"ignore":[],"version":1}` (the ignore-list +mechanism of ITL-981; the hash is strict by default and can leave out `column_order`, +`struct_field_order` and `series_name`) and are registered, with their value handlers, in +`contexts/logical_types.json`. Golden values (value and data hashes, type and +`Schema` hashes, source identities, columns without values) are pinned in the shared golden +suite, `tests/test_hashing/golden` (the `polars_*` cases). Existing hashes are unchanged: +every pre-existing golden entry passes. +Open: SQLite stores reject every logical-typed column (ITL-877); the pandas handler ignores +column order, Categorical vs String and struct field order (ITL-897). diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 7e613ff79..778fe591a 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -632,6 +632,79 @@ Each result row also records the full version string of the function that produc the column `__pf_var_function_version` (for example `v3.7`); only the major part is in the table path. +## Value types + +A function pod's parameters and return values can be plain Python scalars (`int`, `float`, +`str`, `bool`, `bytes`), lists, dataclasses, pydantic models, and the built-in logical types +below. A logical type is stored as an Arrow extension type: the extension name is recorded +in the stored schema, so a value read back from a database comes back as the same Python +type. + +| Python type | Extension name | Arrow storage | +|---|---|---| +| `pathlib.Path` | `orcapod.path` | `large_string` | +| `upath.UPath` | `orcapod.upath` | `large_string` | +| `uuid.UUID` | `orcapod.uuid` | `large_binary` | +| `orcapod.File` | `orcapod.file` | `large_string` | +| `orcapod.Directory` | `orcapod.directory` | `large_string` | +| `numpy.ndarray` | `orcapod.numpy.ndarray` | `large_binary` (`.npy` bytes, see [Arrays](arrays.md)) | +| `pandas.DataFrame` | `pandas.dataframe` | `large_binary` (Arrow IPC stream) | +| `pandas.Series` | `pandas.series` | `large_binary` (Arrow IPC stream) | +| `polars.DataFrame` | `orcapod.polars.dataframe` | `large_binary` (Polars IPC file, `write_ipc`) | +| `polars.Series` | `orcapod.polars.series` | `large_binary` (Polars IPC file, `write_ipc`) | + +A `polars.DataFrame` comes back with exactly the schema it was stored with: column order and +names, integer and float widths, `Categorical`, `Enum` with its category order, `Decimal` +precision and scale, `Datetime` unit and time zone, `Duration` unit, `List`, `Array` shape, +`Struct` fields in their order and `Null`. A `polars.Series` keeps its name, dtype and values the same way. +Dtypes that cannot be stored and read back exactly (`Object`, `Int128`, `UInt128`, and Polars +extension dtypes) raise `ValueError` naming the column when the value is stored, so convert +such a column explicitly before returning the frame. + +A Polars value's content hash, which is its cache key, covers the value's logical content: its +row count, column names and their order, dtypes (struct fields and their order included) and +values. So a pod that depends on column order, such as one returning `df.columns[0]`, gets a +separate cache entry for each order. Physical properties are not part of the hash: chunking +(`n_chunks()`) and sortedness flags (`Series.flags`, which are not stored either). A pod must +not depend on them, because equal frames that differ only there share a cache entry. See +[Polars DataFrame and Series values](../reference/hashing.md#1e-polars-dataframe-and-series-values) +for the exact scheme. + +```python +import polars as pl + +from orcapod import function_pod +from orcapod.sources import DictSource + +@function_pod(output_keys="readings") +def load_readings(n_samples: int) -> pl.DataFrame: + return pl.DataFrame( + { + "sample": pl.Series(range(n_samples), dtype=pl.UInt16), + "voltage": pl.Series([0.5] * n_samples, dtype=pl.Float32), + } + ) + +@function_pod(output_keys="n_rows") +def count_rows(readings: pl.DataFrame) -> int: + return readings.height + +recordings = DictSource( + data=[{"session": "s1", "n_samples": 2}, {"session": "s2", "n_samples": 3}], + tag_columns=["session"], +) +readings = load_readings.pod(recordings) +for tag, data in readings.iter_data(): + print(tag["session"], dict(data["readings"].schema)) +# s1 {'sample': UInt16, 'voltage': Float32} +# s2 {'sample': UInt16, 'voltage': Float32} + +for tag, data in count_rows.pod(readings).iter_data(): + print(tag["session"], data["n_rows"]) +# s1 2 +# s2 3 +``` + ## Multiple input streams If you pass multiple streams to a function pod, they are automatically joined (using diff --git a/docs/concepts/logical-type-identifiers.md b/docs/concepts/logical-type-identifiers.md index 0cf49171d..00a494bf5 100644 --- a/docs/concepts/logical-type-identifiers.md +++ b/docs/concepts/logical-type-identifiers.md @@ -80,6 +80,8 @@ Orcapod's own types follow the same contract as yours. Their metadata is the JSO | `numpy.ndarray` | `orcapod.numpy.ndarray` | `{"ignore":[],"version":1}` | | `pandas.DataFrame` | `pandas.dataframe` | `{"version":1}` | | `pandas.Series` | `pandas.series` | `{"version":1}` | +| `polars.DataFrame` | `orcapod.polars.dataframe` | `{"ignore":[],"version":1}` | +| `polars.Series` | `orcapod.polars.series` | `{"ignore":[],"version":1}` | `{"version":N}` is the convention of the built-in types only. Your types may use any bytes. diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 8fbb7a3d3..a1e7470be 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -58,6 +58,8 @@ The labels of v0.3: | `directory_v1` | the directory hash: a SHA-256 Merkle tree over the entries | `DIRECTORY_HASH_LABEL`, `hashing/directory_hashers.py` | | `ndarray_v1` | the array value hash (§8) | `NumpyArrayHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` | | `polars_expr_v1` | the Polars expression hash (§1c) | `POLARS_EXPR_HASH_LABEL`, `hashing/semantic_hashing/polars_expr.py` | +| `polars_dataframe_v1` | the Polars DataFrame value hash (§1e) | `PolarsDataFrameHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` | +| `polars_series_v1` | the Polars Series value hash (§1e) | `PolarsSeriesHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` | | `sha256` | the file content hash: SHA-256 of the file's bytes | the file hasher's `algorithm` | Where a hash appears, and in which form: @@ -478,6 +480,97 @@ function pod's identity does **not** use it: the pod's `uri`, its `pipeline_hash schema (see [Data Function URI Hash](#5-data-function-uri-hash)), so changing a default moves no path. +### 1e. Polars DataFrame and Series values + +A `polars.DataFrame` or `polars.Series` is a value-hashed logical type +(`PolarsDataFrameHandler`, `PolarsSeriesHandler`; the algorithm is defined in +`orcapod.logical_types.polars_type`). Inside a table its cell token is, like every +value-hashed type's (see [the hash label rule](#the-hash-label-rule)), +`b"orcapod.polars.dataframe;::" + digest`: the logical type's +identifier, then the digest, without the label. The extension metadata is +`{"ignore":[],"version":1}`: the version and the list of properties the hash ignores, empty +by default. + +Storage and hashing are separate. The value is stored with Polars' own `write_ipc`; the +stored bytes are never hashed (they depend on the Polars version that wrote them). The hash +is computed from the value's content: + +```text +sha256(descriptor + b"\x00" + starfix.ArrowDigester.hash_table( + values, ordered_columns=True, ordered_struct_fields=True)) +``` + +under the label `polars_dataframe_v1` for a frame and `polars_series_v1` for a Series (a +Series is hashed as its one-column frame, with its own descriptor). + +`values` is the frame exported to Arrow with `compat_level=oldest`, with top-level `Null` columns left +out, `Array` cast to `large_list`, nested `Null` cast to an all-null `bool` and dictionaries +decoded, for hashing only. starfix runs in its ordered mode, so column order and struct field +order at every depth are part of the digest, and it tells a null struct from a struct whose +fields are all null. The descriptor carries what the Arrow values cannot (Enum categories, a +`Categorical`'s `Categories`, `Array` size, `Null` columns, the row count, a Series' name), +and states names, order and dtypes again in Orcapod's own terms. `descriptor` is canonical JSON (keys +sorted, compact separators, UTF-8) in a grammar Orcapod owns, never Polars' `repr`: + +```text +frame: {"kind": "orcapod.polars.dataframe", "height": , + "columns": [[, ], ...]} in column order +series: {"kind": "orcapod.polars.series", "name": , "height": , + "dtype": } + + := "int8" | "int16" | "int32" | "int64" | "uint8" | "uint16" | "uint32" | "uint64" + | "float16" | "float32" | "float64" | "boolean" | "string" | "binary" + | "date" | "time" | "null" + | {"categorical": [, , ]} its Categories + | {"enum": [, ...]} every declared category, in order + | {"decimal": [, ]} + | {"datetime": [,