Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
53a2d5d
refactor(logical_types): share the Arrow IPC helpers between frame types
Nov 11, 2026
9fd86b7
feat(logical_types): add polars.DataFrame logical type with Arrow IPC…
Nov 11, 2026
32dd5f7
feat(contexts): register polars.DataFrame in the default data context…
Nov 11, 2026
75c8b55
test(function_pod): polars.DataFrame values through pods, operators a…
Nov 11, 2026
504b4c1
docs(function_pods): list value types, including polars.DataFrame (IT…
Nov 11, 2026
cca88ca
fix(logical_types): type the polars dtype walker and polars value tes…
Nov 11, 2026
7d0e21e
feat(hashing): hash polars values by an Orcapod dtype descriptor plus…
Nov 11, 2026
1516e37
docs: document polars value storage and hashing; log ITL-578 in DESIG…
Nov 11, 2026
a0c5d9e
Merge remote-tracking branch 'origin/main' into eywalker/itl-578-add-…
Nov 11, 2026
f1a711e
test(hashing): pin that an Enum's full category list and order are pa…
Nov 11, 2026
d00b2ce
docs(function_pods): print the schema as a dict so the sample is stab…
Nov 11, 2026
c669363
docs(logical_types): docstrings for polars helpers; explicit check=Fa…
Nov 11, 2026
ec1f819
Merge remote-tracking branch 'origin/main' into eywalker/itl-578-add-…
Nov 11, 2026
7dad080
test(hashing): pin Polars hashes in the shared golden suite (ITL-578)
Nov 11, 2026
cdfa75d
docs(hashing): docstrings for the Polars golden cases (ITL-578)
Nov 11, 2026
017843d
feat(hashing): label Polars value hashes polars_dataframe_v1 / polars…
Nov 11, 2026
9e9a723
docs(hashing): versioned per-type value hash labels (ITL-578)
Nov 11, 2026
fbf2ed6
Merge remote-tracking branch 'origin/main' into eywalker/itl-578-add-…
Nov 11, 2026
2abaf4c
test(polars): SemiJoin keeps polars.DataFrame columns exactly (ITL-578)
Nov 11, 2026
9ed28ca
docs(polars): pods must not depend on a frame's column order (ITL-578)
Nov 11, 2026
4bb0875
feat(polars): column order, struct field order and struct validity ar…
Nov 11, 2026
06f1682
refactor(polars): type the struct-validity projection for basedpyrigh…
Nov 11, 2026
e7a095f
docs(plans): ITL-578 progress notes
Nov 11, 2026
020ad82
refactor(polars): isolate the starfix call and order handling (ITL-57…
Nov 11, 2026
99674d8
refactor(polars): drop the error-only column parameter of polars_dtyp…
Nov 11, 2026
b9a8f25
fix(polars): hash zero-field structs; vary orders independently in te…
Nov 11, 2026
e92dd31
docs(plans): ITL-578 refresh onto the v0.3 hash step
Nov 12, 2026
3833616
merge: origin/main (c8e9eb21, v0.3 hash step) into ITL-578
Nov 12, 2026
712a72f
feat(polars): v0.3 logical type rules for polars.DataFrame and polars…
Nov 12, 2026
0150a5c
test(polars): typed decode helper; registration listing includes the …
Nov 12, 2026
270e94c
docs(plans): ITL-578 refresh progress notes
Nov 12, 2026
3adf073
feat(polars): store with Polars' own write_ipc; hash functions take i…
Nov 12, 2026
d88a1ce
feat(polars)!: extension names orcapod.polars.dataframe and orcapod.p…
Nov 12, 2026
b74c7d6
merge: origin/main (25650c42: #299 output types, #303 loaded jobs, #3…
Nov 12, 2026
40957a8
feat(polars): the hash descriptor's kind is the extension name (ITL-578)
Nov 12, 2026
abea271
Merge remote-tracking branch 'origin/main' into eywalker/itl-578-add-…
Nov 13, 2026
a26b7cb
feat(polars): ignore-list identifiers and handlers for the Polars val…
Nov 12, 2026
b1fca33
test(polars): follow the ndarray rename and storage of ITL-981 in a f…
Nov 13, 2026
5cad706
fix(contexts): refuse a data context file whose JSON repeats a key (I…
Nov 13, 2026
58a814d
docs(changelog, plans): ITL-578 fragment for the final design; refres…
Nov 13, 2026
00ced35
fix(polars, contexts): review round 1 on 58a814d1 (ITL-578)
Nov 13, 2026
4d77a4b
test(polars): one-line docstrings on every Polars test, helper and po…
Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions changelog.d/ITL-578.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
### Added

#### `polars.DataFrame` and `polars.Series` values (ITL-578 #286)

A function pod can take or return a `pl.DataFrame` or `pl.Series`; before, declaring one
raised `TypeError: No LogicalType or LogicalTypeFactory registered`. The logical types
`orcapod.polars.dataframe` and `orcapod.polars.series` store the value with Polars' own
`write_ipc` in `large_binary`, and a value reads back with exactly the schema it was stored
with (column order, struct field order, every dtype parameter). Chunking and sortedness flags
are not stored. `Object`, `Int128`, `UInt128` and Polars extension dtypes raise `ValueError`
naming the column when the value is stored.

The content hash (the cache key) is labelled `polars_dataframe_v1` / `polars_series_v1` and
is strict: it covers everything a pod can observe as the value's logical content, that is
row count, column names and order, dtypes (struct field order, Enum categories, a
`Categorical`'s `Categories`), a Series' name, and values, where a null struct differs from
a struct of nulls. Chunking, sortedness flags and physical `Categorical` codes are not part
of it. A data context can leave out `column_order` and `struct_field_order` for a frame, and
`series_name` and `struct_field_order` for a Series, through the `hash_ignore` lists of
`contexts/logical_types.json`. See *Polars DataFrame and Series values* in the hashing
reference. Storing a frame in a SQLite store is not supported yet (ITL-877).

### Fixed

- A data context file (`contexts/logical_types.json` or a manifest) in which a JSON object
repeats a key is refused with `ContextValidationError` naming the file and the key. Before,
the last of the repeated keys silently replaced the others (ITL-578).
36 changes: 36 additions & 0 deletions design_issues/ITL-578-PL1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## `src/orcapod/logical_types/polars_type.py` — Polars values

### PL1 — `pl.DataFrame` had no logical type, so a pod could not take or return one
**Status:** resolved
**Severity:** high
**Issue:** ITL-578

`@function_pod(output_keys="df") def f(n: int) -> pl.DataFrame` raised at decoration:
`TypeError: No LogicalType or LogicalTypeFactory registered for <class 'polars.dataframe.frame.DataFrame'>`.

**Fix:** `LogicalPolarsDataFrame` (`orcapod.polars.dataframe`) and `LogicalPolarsSeries`
(`orcapod.polars.series`, a one-column frame) store the value in `large_binary` with Polars' own
`write_ipc` (Polars' defaults, pinned) and read it with `read_ipc` from memory (developer's
decision, November 12, 2026: storage is the library's own serialization, the hash is defined
separately from the value's content). Every supported dtype reads back exactly; chunking and
sortedness flags are not stored. `Object`, `Int128`, `UInt128` and Polars extension dtypes
are rejected at write time, naming the column. The stored bytes are never hashed,
so `PolarsDataFrameHandler` / `PolarsSeriesHandler` hash
`sha256(Orcapod dtype descriptor + starfix digest of the values)` instead, under the labels
`polars_dataframe_v1` / `polars_series_v1` (developer's decision, November 10, 2026).
Everything a pod can observe through the public Polars API as the value's logical content is
identity (developer's decision, November 11, 2026, reversing the November 10 ruling that
order was not): the descriptor lists columns and struct fields in order at every depth, a
`Categorical` carries its `Categories` (name, namespace, physical width), and the values are
digested by starfix 0.5 in its ordered mode (`ordered_columns`, `ordered_struct_fields`,
ITL-898), which also tells a null struct from a struct of nulls. Physical properties stay
out: chunking, sortedness flags (not stored), physical `Categorical` codes, Arrow layout.
Both types carry the identifier metadata `{"ignore":[],"version":1}` (the ignore-list
mechanism of ITL-981; the hash is strict by default and can leave out `column_order`,
`struct_field_order` and `series_name`) and are registered, with their value handlers, in
`contexts/logical_types.json`. Golden values (value and data hashes, type and
`Schema` hashes, source identities, columns without values) are pinned in the shared golden
suite, `tests/test_hashing/golden` (the `polars_*` cases). Existing hashes are unchanged:
every pre-existing golden entry passes.
Open: SQLite stores reject every logical-typed column (ITL-877); the pandas handler ignores
column order, Categorical vs String and struct field order (ITL-897).
73 changes: 73 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -632,6 +632,79 @@ Each result row also records the full version string of the function that produc
the column `__pf_var_function_version` (for example `v3.7`); only the major part is in the
table path.

## Value types

A function pod's parameters and return values can be plain Python scalars (`int`, `float`,
`str`, `bool`, `bytes`), lists, dataclasses, pydantic models, and the built-in logical types
below. A logical type is stored as an Arrow extension type: the extension name is recorded
in the stored schema, so a value read back from a database comes back as the same Python
type.

| Python type | Extension name | Arrow storage |
|---|---|---|
| `pathlib.Path` | `orcapod.path` | `large_string` |
| `upath.UPath` | `orcapod.upath` | `large_string` |
| `uuid.UUID` | `orcapod.uuid` | `large_binary` |
| `orcapod.File` | `orcapod.file` | `large_string` |
| `orcapod.Directory` | `orcapod.directory` | `large_string` |
| `numpy.ndarray` | `orcapod.numpy.ndarray` | `large_binary` (`.npy` bytes, see [Arrays](arrays.md)) |
| `pandas.DataFrame` | `pandas.dataframe` | `large_binary` (Arrow IPC stream) |
| `pandas.Series` | `pandas.series` | `large_binary` (Arrow IPC stream) |
| `polars.DataFrame` | `orcapod.polars.dataframe` | `large_binary` (Polars IPC file, `write_ipc`) |
| `polars.Series` | `orcapod.polars.series` | `large_binary` (Polars IPC file, `write_ipc`) |

A `polars.DataFrame` comes back with exactly the schema it was stored with: column order and
names, integer and float widths, `Categorical`, `Enum` with its category order, `Decimal`
precision and scale, `Datetime` unit and time zone, `Duration` unit, `List`, `Array` shape,
`Struct` fields in their order and `Null`. A `polars.Series` keeps its name, dtype and values the same way.
Dtypes that cannot be stored and read back exactly (`Object`, `Int128`, `UInt128`, and Polars
extension dtypes) raise `ValueError` naming the column when the value is stored, so convert
such a column explicitly before returning the frame.

A Polars value's content hash, which is its cache key, covers the value's logical content: its
row count, column names and their order, dtypes (struct fields and their order included) and
values. So a pod that depends on column order, such as one returning `df.columns[0]`, gets a
separate cache entry for each order. Physical properties are not part of the hash: chunking
(`n_chunks()`) and sortedness flags (`Series.flags`, which are not stored either). A pod must
not depend on them, because equal frames that differ only there share a cache entry. See
[Polars DataFrame and Series values](../reference/hashing.md#1e-polars-dataframe-and-series-values)
for the exact scheme.

```python
import polars as pl

from orcapod import function_pod
from orcapod.sources import DictSource

@function_pod(output_keys="readings")
def load_readings(n_samples: int) -> pl.DataFrame:
return pl.DataFrame(
{
"sample": pl.Series(range(n_samples), dtype=pl.UInt16),
"voltage": pl.Series([0.5] * n_samples, dtype=pl.Float32),
}
)

@function_pod(output_keys="n_rows")
def count_rows(readings: pl.DataFrame) -> int:
return readings.height

recordings = DictSource(
data=[{"session": "s1", "n_samples": 2}, {"session": "s2", "n_samples": 3}],
tag_columns=["session"],
)
readings = load_readings.pod(recordings)
for tag, data in readings.iter_data():
print(tag["session"], dict(data["readings"].schema))
# s1 {'sample': UInt16, 'voltage': Float32}
# s2 {'sample': UInt16, 'voltage': Float32}

for tag, data in count_rows.pod(readings).iter_data():
print(tag["session"], data["n_rows"])
# s1 2
# s2 3
```

## Multiple input streams

If you pass multiple streams to a function pod, they are automatically joined (using
Expand Down
2 changes: 2 additions & 0 deletions docs/concepts/logical-type-identifiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ Orcapod's own types follow the same contract as yours. Their metadata is the JSO
| `numpy.ndarray` | `orcapod.numpy.ndarray` | `{"ignore":[],"version":1}` |
| `pandas.DataFrame` | `pandas.dataframe` | `{"version":1}` |
| `pandas.Series` | `pandas.series` | `{"version":1}` |
| `polars.DataFrame` | `orcapod.polars.dataframe` | `{"ignore":[],"version":1}` |
| `polars.Series` | `orcapod.polars.series` | `{"ignore":[],"version":1}` |

`{"version":N}` is the convention of the built-in types only. Your types may use any bytes.

Expand Down
93 changes: 93 additions & 0 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ The labels of v0.3:
| `directory_v1` | the directory hash: a SHA-256 Merkle tree over the entries | `DIRECTORY_HASH_LABEL`, `hashing/directory_hashers.py` |
| `ndarray_v1` | the array value hash (§8) | `NumpyArrayHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` |
| `polars_expr_v1` | the Polars expression hash (§1c) | `POLARS_EXPR_HASH_LABEL`, `hashing/semantic_hashing/polars_expr.py` |
| `polars_dataframe_v1` | the Polars DataFrame value hash (§1e) | `PolarsDataFrameHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` |
| `polars_series_v1` | the Polars Series value hash (§1e) | `PolarsSeriesHandler.HASH_LABEL`, `hashing/semantic_hashing/builtin_handlers.py` |
| `sha256` | the file content hash: SHA-256 of the file's bytes | the file hasher's `algorithm` |

Where a hash appears, and in which form:
Expand Down Expand Up @@ -478,6 +480,97 @@ function pod's identity does **not** use it: the pod's `uri`, its `pipeline_hash
schema (see [Data Function URI Hash](#5-data-function-uri-hash)), so changing a default
moves no path.

### 1e. Polars DataFrame and Series values

A `polars.DataFrame` or `polars.Series` is a value-hashed logical type
(`PolarsDataFrameHandler`, `PolarsSeriesHandler`; the algorithm is defined in
`orcapod.logical_types.polars_type`). Inside a table its cell token is, like every
value-hashed type's (see [the hash label rule](#the-hash-label-rule)),
`b"orcapod.polars.dataframe;<hex of the extension metadata>::" + digest`: the logical type's
identifier, then the digest, without the label. The extension metadata is
`{"ignore":[],"version":1}`: the version and the list of properties the hash ignores, empty
by default.

Storage and hashing are separate. The value is stored with Polars' own `write_ipc`; the
stored bytes are never hashed (they depend on the Polars version that wrote them). The hash
is computed from the value's content:

```text
sha256(descriptor + b"\x00" + starfix.ArrowDigester.hash_table(
values, ordered_columns=True, ordered_struct_fields=True))
```

under the label `polars_dataframe_v1` for a frame and `polars_series_v1` for a Series (a
Series is hashed as its one-column frame, with its own descriptor).

`values` is the frame exported to Arrow with `compat_level=oldest`, with top-level `Null` columns left
out, `Array` cast to `large_list`, nested `Null` cast to an all-null `bool` and dictionaries
decoded, for hashing only. starfix runs in its ordered mode, so column order and struct field
order at every depth are part of the digest, and it tells a null struct from a struct whose
fields are all null. The descriptor carries what the Arrow values cannot (Enum categories, a
`Categorical`'s `Categories`, `Array` size, `Null` columns, the row count, a Series' name),
and states names, order and dtypes again in Orcapod's own terms. `descriptor` is canonical JSON (keys
sorted, compact separators, UTF-8) in a grammar Orcapod owns, never Polars' `repr`:

```text
frame: {"kind": "orcapod.polars.dataframe", "height": <rows>,
"columns": [[<name>, <dtype>], ...]} in column order
series: {"kind": "orcapod.polars.series", "name": <name>, "height": <rows>,
"dtype": <dtype>}

<dtype> := "int8" | "int16" | "int32" | "int64" | "uint8" | "uint16" | "uint32" | "uint64"
| "float16" | "float32" | "float64" | "boolean" | "string" | "binary"
| "date" | "time" | "null"
| {"categorical": [<name>, <namespace>, <physical>]} its Categories
| {"enum": [<category>, ...]} every declared category, in order
| {"decimal": [<precision>, <scale>]}
| {"datetime": [<unit>, <time zone or null>]}
| {"duration": <unit>}
| {"list": <dtype>}
| {"array": [<dtype>, <size>]} a multi-dimensional array nests
| {"struct": [[<field>, <dtype>], ...]} in field order
```

The rule: everything a pod can observe through the public Polars API as the frame's logical
content is identity. So two frames hash equal exactly when they have the same row count, the
same column names in the same order, the same dtype for each column (struct fields and their
order at every depth, Enum categories and their order, a `Categorical`'s `Categories` name,
namespace and physical width, Decimal precision and scale, Datetime unit and time zone,
Duration unit and Array shape included) and the same values (a null struct differs from a
struct whose fields are all null; a null differs from NaN, and `0.0` from `-0.0`). A pod that
returns `df.columns[0]` therefore never receives another frame's cached result. An Enum's full enumeration of choices is part of its dtype: all its declared categories in
declared order, including categories no row holds, so `Enum(["a", "b"])`, `Enum(["b", "a"])`
and `Enum(["a", "b", "z"])` hash differently over the same values.

Physical properties are not identity: chunking (`n_chunks()`) and rechunking, slice offsets,
the bytes under null slots, sortedness flags (`Series.flags`, which storage does not keep: a
frame read back has them cleared), physical `Categorical` codes (`to_physical()`) and the
contents of the process-wide category registry, which depend on what the process saw before,
the Arrow layout Polars exports (string views, large or small offsets) and memory size.
Supporting a new dtype adds to the grammar without changing any existing hash; only a change
that would give an existing value a different digest needs new labels (`polars_dataframe_v2`,
`polars_series_v2`). A dtype outside the grammar raises `ValueError` naming the column.

One kind of value cannot be hashed: starfix addresses a column or struct field by the path of
its names joined with `/`, so a value in which two names form the same path (a column `"a/b"`
beside a struct column `a` with a field `b`, or the same between a struct's fields) raises
`ValueError` naming the path. A name containing `/` is fine as long as no other name forms
the same path. The Arrow table hash (§9) has the same limit.

The hash is strict by default. An ignore list, part of the logical type's identifier, can
leave out `column_order` and `struct_field_order` for a frame, and `series_name` and
`struct_field_order` for a Series. A data context declares its list once, under `hash_ignore`
in `contexts/logical_types.json` (`"orcapod.polars.dataframe": []` by default); two lists are
two logical types, so their columns never share a hash, and a stored column whose list
differs from the data context's is refused on read. Ignoring column order lists the
descriptor's columns sorted by name and runs starfix with `ordered_columns=False`; ignoring
struct field order does the same for every struct's fields with
`ordered_struct_fields=False`; ignoring a Series' name writes `""` for `"name"` in the
descriptor and hashes its values as a column named `""`.

A `pandas.DataFrame` is hashed differently: as the `arrow_v1` hash of its Arrow table, which
ignores column order. A Polars frame and the equal pandas frame never share a hash.

---

### 1e. `dict` and `set` values: canonical order
Expand Down
46 changes: 45 additions & 1 deletion src/orcapod/contexts/logical_types.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{
"description": "Logical types registered in every data context, with the value handlers that hash their Python values. This file is not frozen: a logical type is versioned by its own identifier (Arrow extension name and metadata), not by the data context version, so adding one edits this file and no released data context manifest. hash_ignore lists, per extension name, the property names the value hash of a library value type ignores (empty: every property the storage keeps is part of the hash); the list is written into the type's extension metadata, so changing it changes the type's identifier. The logical type entry and the value handler entry of such a type both refer to their list as hash_ignore:<extension name>.",
"hash_ignore": {
"orcapod.numpy.ndarray": []
"orcapod.numpy.ndarray": [],
"orcapod.polars.dataframe": [],
"orcapod.polars.series": []
},
"logical_type_registry": {
"_class": "orcapod.logical_types.registry.LogicalTypeRegistry",
Expand Down Expand Up @@ -42,6 +44,22 @@
{
"_class": "orcapod.logical_types.pandas_type.LogicalPandasSeries",
"_config": {}
},
{
"_class": "orcapod.logical_types.polars_type.LogicalPolarsDataFrame",
"_config": {
"ignore": {
"_ref": "hash_ignore:orcapod.polars.dataframe"
}
}
},
{
"_class": "orcapod.logical_types.polars_type.LogicalPolarsSeries",
"_config": {
"ignore": {
"_ref": "hash_ignore:orcapod.polars.series"
}
}
}
],
"factories": [
Expand Down Expand Up @@ -159,6 +177,32 @@
"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PandasSeriesHandler",
"_config": {}
}
],
[
{
"_type": "polars.DataFrame"
},
{
"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PolarsDataFrameHandler",
"_config": {
"ignore": {
"_ref": "hash_ignore:orcapod.polars.dataframe"
}
}
}
],
[
{
"_type": "polars.Series"
},
{
"_class": "orcapod.hashing.semantic_hashing.builtin_handlers.PolarsSeriesHandler",
"_config": {
"ignore": {
"_ref": "hash_ignore:orcapod.polars.series"
}
}
}
]
]
}
Loading
Loading