Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
0703724
docs(itl-895): root-cause findings and evidence scripts for float-to-…
Nov 11, 2026
2e30622
test(function_pod): failing regressions for output type conformance (…
Nov 11, 2026
c3ca32a
fix(data_function): reject function outputs that do not match the dec…
Nov 11, 2026
af7a323
docs: function outputs must match the declared type (ITL-895)
Nov 11, 2026
1b99edb
refactor(output_conformance): type values as object; name conversion …
Nov 11, 2026
2481175
docs(itl-895): note that the evidence scripts show pre-fix behaviour
Nov 11, 2026
715f232
test(objective): None for a non-optional output is rejected (ITL-895)
Nov 11, 2026
5ee9991
docs(plan): ITL-895 status after draft PR
Nov 11, 2026
3299571
docs(plan): ITL-895 developer decisions plan
Nov 11, 2026
5e7fdbe
docs(plan): ITL-895 coordinator reply on the warning design
Nov 11, 2026
e22519c
test(function_pod): developer decisions D1-D7 and ITL-901 (ITL-895)
Nov 11, 2026
9ac62ad
feat(output_conformance): apply the ITL-895 decisions with OutputConv…
Nov 11, 2026
0adea7a
refactor(data_function): shared ValueConversionWarning with explicit …
Nov 11, 2026
e61e51b
Merge origin/main into ITL-895 (ITL-878 store write failures)
Nov 11, 2026
b13d716
refactor(output_conformance): containers annotate their parts' results
Nov 11, 2026
f52fcc0
docs(plan): ITL-895 status after the warning rework
Nov 11, 2026
be31585
test(function_pod): count only ValueConversionWarnings in dedup tests
Nov 11, 2026
0256cf2
docs(itl-895): dedup script binds its row count on every path
Nov 11, 2026
883a89c
fix(output_conformance): address CodeRabbit review on PR #299 (ITL-895)
Nov 11, 2026
1ec7ee8
chore: merge origin/main into ITL-895 (StoreReadError, ITL-884)
Nov 11, 2026
5537c0a
chore: merge origin/main into ITL-895 (polars_expr_v1, ITL-828)
Nov 11, 2026
3446655
chore: merge main and checkpoint output conformance refresh
Nov 12, 2026
a7c895f
fix(outputs): enforce declared types with subclass storage semantics
Nov 12, 2026
c7f53bc
chore: merge current main and verify output conformance integration
Nov 12, 2026
0bc9cc1
fix(outputs): reject non-dict mappings and non-member Literal values …
Nov 12, 2026
a769f5d
test(outputs): pin only the possible values of the async subclass bas…
Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -293,8 +293,8 @@ A class a user defines is one of two kinds (developer, November 12, 2026, ITL-97
function annotated with a structural class may receive another class of the same
identity; a structural class promises its fields, not its methods. `from_instance` is the
only conversion, and Orcapod never calls it.
- A value is stored only as its own class (`UserClassLogicalType.accepts_instance`); never
store a subclass instance or a look-alike object in another class's column.
- A declared class stores its own instances and subclasses (`UserClassLogicalType.accepts_instance`);
only the declared class fields enter storage. Unrelated look-alike classes are refused.
- The identity record (`logical_types/user_class_identity.py`) is `{kind, name, fields}`
with JSON type tokens. The Arrow extension name is `name@<SHA-256 of the record>` and the
record is the extension metadata; together they are the logical type identifier. A nested
Expand Down Expand Up @@ -632,6 +632,11 @@ data. The rules:
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
every execution path (sync and async).
Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base.
Structural classes additionally accept equivalent structural identities. Numeric
narrowing and unrelated container types require explicit conversion. Base storage
omits subclass-only fields, and those fields do not enter the stored row hash.

- Every output column takes the large-normalized form of its input type
(arrow_utils.normalize_to_large_types: string to large_string, binary to large_binary, list
to large_list, recursively). Width, nullability and logical (extension) types are never
Expand Down
9 changes: 7 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -304,8 +304,8 @@ A class a user defines is one of two kinds (developer, November 12, 2026, ITL-97
function annotated with a structural class may receive another class of the same
identity; a structural class promises its fields, not its methods. `from_instance` is the
only conversion, and Orcapod never calls it.
- A value is stored only as its own class (`UserClassLogicalType.accepts_instance`); never
store a subclass instance or a look-alike object in another class's column.
- A declared class stores its own instances and subclasses (`UserClassLogicalType.accepts_instance`);
only the declared class fields enter storage. Unrelated look-alike classes are refused.
- The identity record (`logical_types/user_class_identity.py`) is `{kind, name, fields}`
with JSON type tokens. The Arrow extension name is `name@<SHA-256 of the record>` and the
record is the extension metadata; together they are the logical type identifier. A nested
Expand Down Expand Up @@ -674,6 +674,11 @@ data. The rules:
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
every execution path (sync and async).
Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base.
Structural classes additionally accept equivalent structural identities. Numeric
narrowing and unrelated container types require explicit conversion. Base storage
omits subclass-only fields, and those fields do not enter the stored row hash.

- Every output column takes the large-normalized form of its input type
(`arrow_utils.normalize_to_large_types`: `string` to `large_string`, `binary` to `large_binary`, `list`
to `large_list`, recursively). Width, nullability and logical (extension) types are never
Expand Down
36 changes: 36 additions & 0 deletions changelog.d/ITL-895.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
### Added

- `orcapod.errors.OutputTypeError` identifies function-output mismatches by function,
output column, declared type and returned type, with an explicit-conversion hint.

### Fixed

- Function outputs are validated before iteration, hashing or storage on standalone,
cached, sync and async paths (ITL-895, ITL-901). Float-for-int truncation and unordered
set-for-list conversion are rejected. Nested values follow the same checks; exact and
structural user classes reuse their logical type compatibility, and ndarray mismatches
use the same error. Diagnostics do not expose returned mapping keys or converter values.
- Same-kind numeric values are accepted silently only when represented exactly, including
NumPy scalars for Python numbers and in-range Python integers for NumPy integer types.
Numeric narrowing, non-subclass kind changes and iterable-to-list require explicit
conversion. Subclasses are accepted unchanged in memory; declared base storage omits
subclass-only fields and their contribution to row hashes. This includes bool for int
and datetime for date, whose stored form is the declared base type. Accepted numeric
exceptions are normalized to the declared type immediately, so downstream pods see the
value that is stored. A subclass value is the exception: downstream pods in memory
receive the object as returned (a `datetime` declared as `date`, or a user-class
subclass with its extra fields), while storage holds only the declared base form.
Conforming-data hashes are unchanged.
- A declared `dict` accepts only a `dict` instance (other mappings such as
`MappingProxyType` need `dict(value)`), and a declared `Literal` accepts only its
members, matched by type and value with no numeric exception.

### Known limitations

- Previously stored rows remain unchanged and are not automatically recomputed or repaired.
Python-value inputs (sources and datagrams) remain outside output validation (ITL-895).
- The declared type's logical type decides what is stored. A function that reads fields its
declared parameter type does not have (the extra fields of a subclass returned for a base
type, or the time of a `datetime` declared as `date`) gets results that can depend on the
execution path and can be reused across values that differ only in those fields
(ITL-978).
35 changes: 35 additions & 0 deletions design_issues/ITL-895-FO1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
## `src/orcapod/core/data_function.py` — returned values not checked against the declared type

### FO1 — A pod declared `-> int` stored `1.5` as `1`
**Status:** resolved
**Severity:** high
**Issue:** ITL-895

`PythonDataFunction._build_output_data` built `Data(output, python_schema=declared)` without
looking at the values. The Arrow table was built later with
`pa.Table.from_pylist(rows, schema=...)`, and the type converter passes primitives through
unchanged. pyarrow's integer converter calls `__int__` on a non-`int`, so `1.5`,
`numpy.float64(1.5)`, `Decimal` and `Fraction` were truncated. Numbers were read as epoch ticks
for date and timestamp fields, `bytes` and `str` were decoded and encoded into each other, and
`None` became a NULL in a not-null field. The `Data` kept the raw dict, so `as_dict()` gave
`1.5` while `as_table()`, the store and the hash gave `1`. Downstream pods read `as_dict()`, so
the same graph gave different results standalone, on the sync orchestrator and on the async
orchestrator. Present in every release up to v0.2.0. Root-cause evidence:
`superpowers/specs/itl-895-float-to-int/FINDINGS.md`.

**Fix (function outputs):** The shared `semantic_types/output_conformance.py` check
runs in `PythonDataFunction._build_output_data` for both sync and async execution. It
checks primitive and nested types, incorporates the ndarray guard, and delegates exact or
structural class acceptance to the active logical type. Mismatches raise `OutputTypeError`
with one diagnostic on standalone and stored paths. Returned values and mapping keys are
absent from diagnostic text and chained converter tracebacks.

The November 12 decision supersedes the earlier warning conversions: only exactly represented
same-kind numeric values are accepted across Python/NumPy types. Unrelated container types, numeric narrowing, and non-subclass kind changes require
explicit conversion. Python subclasses are accepted; the declared logical type controls
storage. Base storage omits subclass-only fields (and their contribution to row hashes);
`bool` for `int` and `datetime` for `date` also follow this subclass rule. Storage is validated eagerly; numeric exceptions are normalized while compatible
user-class objects retain their identity in memory.
Regressions: `tests/test_core/function_pod/test_output_conformance.py`.

Rows stored wrongly before the fix are served unchanged. They are not repaired automatically.
13 changes: 13 additions & 0 deletions design_issues/ITL-895-FO2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
## `src/orcapod/core/data_function.py` — returned values not checked against the declared type

### FO2 — Inputs built from Python values are not checked against their declared types
**Status:** open
**Severity:** medium
**Issue:** ITL-895

Inputs built from Python values (`DictSource` and `ListSource` with a declared schema,
`Datagram`, `Data` and `Tag` from dicts, `Datagram.update` / `with_columns`) still go through
the unchecked pyarrow conversion and truncate the same way (`1.5` for an `int` column is
stored as `1`). Evidence: `superpowers/specs/itl-895-float-to-int/04_input_sites.py`.
Deferred by the developer (2026-11-11): "generally a big implementation and given the nature
of Python, not sure if it's something I'd care to put explicit check".
32 changes: 32 additions & 0 deletions design_issues/ITL-895-FO3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
## `src/orcapod/core/data_function.py` — returned values not checked against the declared type

### FO3 — A function that reads subclass-only fields gets path-dependent, reusable results
**Status:** open
**Severity:** medium
**Issue:** ITL-895

A subclass returned for a declared base type is passed on in memory as it is and stored as
the base: the declared type's logical type decides what is stored, and the row hash covers
the stored fields only. A downstream function that reads a field its declared parameter type
does not have (`getattr(y, "extra", ...)` on a `Reading` parameter that received an
`ExtendedReading`) therefore gets results that depend on the execution path, and results that
are reused across values:

- standalone and the async orchestrator hand the function the in-memory subclass object
(within one async run a row may instead be served the result another row of the same
row hash stored first; which happens depends on timing, seen on Python 3.13 in CI);
- the sync orchestrator hands it the stored base value, without the extra field;
- two values that differ only in subclass-only fields have the same row hash, so a later row
is served the result computed for the earlier one, without a call.

The same holds for a `datetime` returned for a declared `date`. Also in scope: a `list` or
`dict` subclass returned for a declared `list[T]` / `dict[K, V]` is rebuilt as a plain
`list` / `dict` in memory.

Decision (developer, 2026-11-12): "this is a limitation of Python language by design and
... we should not try to fight it. How well this is handled is thus the job of the extension
logical type and we shouldn't over engineer a solution to restrict down things." No code
change before v0.3; follow-up ITL-978 (https://linear.app/metamorphic/issue/ITL-978).

Current behaviour is pinned by `TestSubclassOnlyFieldsDownstream` in
`tests/test_core/function_pod/test_output_conformance.py`.
8 changes: 7 additions & 1 deletion design_issues/ITL-972-UCK1.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ own class.

- An ordinary dataclass or pydantic model is an **exact** class: named by module path and
qualified name, never equivalent to another class. Python subclassing still applies at a
pod input. A value is stored only as its own class.
pod input. An output accepts a subclass, but the declared logical type controls its stored fields (ITL-895).
- `orcapod.dataclass` makes a **structural** class: named by its declared name (default the
class name); two structural classes with equal identity records are compatible in the
schema check and their instances compare equal. Orcapod converts nothing: a value is
Expand Down Expand Up @@ -55,3 +55,9 @@ converts the data) but not over Arrow IPC or the C stream interface, where Arrow
ITL-602; nested struct fields since this fix. Orcapod's own writers and Polars round trips
never produce non-large storage. Pinned by a strict xfail in
`tests/test_user_types/test_read_path.py`.

**Output-boundary follow-up (ITL-895):** a subclass returned for a base-class output is
accepted on every path. It passes unchanged in memory, while the declared base logical type
stores only base fields; extra fields are omitted from storage and row hashes. Compatible
structural objects are validated without replacing them; stored reads retain their registry
binding. Unrelated classes are rejected with `OutputTypeError` at the shared output boundary.
101 changes: 101 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,107 @@ The same step is available as a stream method: `wide.select_data_columns(["weigh
Tag columns are never affected by this rule -- a function pod passes all tags through
untouched.

## Return values must match the declared type

Each value the function returns is checked against its declared output type before it is
stored or hashed. A `float` returned for an output declared `int` is rejected; it is not
truncated. The `OutputTypeError` names the function, the output, the declared type and the
type of the value (never the value itself, which stays on the error's `value` attribute),
and says what to return instead:

```python
from orcapod.errors import OutputTypeError

@function_pod(output_keys="half_weight")
def halve(weight: float) -> int:
return weight / 2 # a float, although the function says it returns an int

weights = DictSource(data=[{"subject_id": "mouse_01", "weight": 25.3}], tag_columns=["subject_id"])

try:
list(halve.pod(weights).iter_data())
except OutputTypeError as err:
print(err)
# Function 'halve' returned float for output 'half_weight', which is declared int. The output does not conform. Return an int instead: round(value), or int(value) to truncate.
```

Fix the function (`return round(weight / 2)`), or declare the type it really returns
(`-> float`).

`OutputTypeError` is a `TypeError` (and a `ValueError`). Inside a `PipelineJob` it counts as
a failure of the function for that row, like an exception the function raises itself. With
the default `error_policy="continue"` the row is skipped and reported to the observer, and
the other rows are computed and stored. With `error_policy="fail_fast"` the synchronous
orchestrator stops the run; the asynchronous orchestrator does not stop for row failures yet
(ITL-879, see below).

The function-output rule is: the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base.

| Declared | Returned | Result |
|---|---|---|
| `int` / `float` / `bool` | same-kind NumPy scalar, represented exactly | accepted silently |
| NumPy integer type | Python or NumPy integer within range | accepted silently; out of range rejected |
| `int` / `float` | value-preserving numeric subclass such as `IntEnum` | accepted silently |
| `float` | `numpy.float32`, including `numpy.float32(0.1)` | accepted: widening preserves the original float32 value exactly |
| `numpy.float32` | Python `float` or `numpy.float64`, even `0.5` | rejected: explicitly return `numpy.float32(value)` |
| `float` | NumPy floating scalar wider than 64 bits | rejected: explicitly return `float(value)` |
| `list[T]` | tuple, NumPy array, range, deque, dict view, or iterator | rejected: return `list(value)` |
| `list[T]` | set or frozenset | rejected: return `sorted(value)` for deterministic order |
| `dict[K, V]` | another mapping, such as `MappingProxyType` or `UserDict` | rejected: return `dict(value)` |
| `Literal[...]` | a value that is not a member, or equals one with another type (`True`, `1.0` or `numpy.int64(1)` for `Literal[1]`) | rejected: a `Literal` names exact values, so no numeric exception applies |
| `date` | `datetime` | accepted as a Python subclass; storage keeps the date only |
| `int` | `bool` | accepted as a Python subclass; stored as an integer |
| `int` | float, Decimal, Fraction | rejected; choose an explicit conversion |
| `bool` | `int` | rejected: return `bool(value)` explicitly |
| floating type | integer or bool | rejected: return a floating value explicitly |
| non-optional type | `None` | rejected; declare `T \| None` if null is intended |
| ordinary dataclass or pydantic model | subclass | accepted; storage keeps only declared base fields |
| ordinary dataclass or pydantic model | unrelated class | rejected |
| structural class | another structural class with the same identity | accepted; fields follow the same recursive checks |

Python subclassing takes precedence: `bool` for `int` is accepted, while `int` for
`bool` is rejected. Numeric exceptions otherwise require the same kind.
Nested list elements, dictionary keys and values, and stored user-class fields follow the
same policy. A mismatch raises `OutputTypeError` at the common function-output boundary,
before standalone iteration, table materialization, or sync/async storage can diverge.
Errors name the function, output, declared type and returned type; nested locations show
structure without including returned dictionary keys or converter messages.

Outputs are validated against the declared storage schema immediately, and accepted numeric
exceptions are normalized so downstream pods, hashes and stored rows see the same value.
Eager validation never replaces a compatible user-class object; stored reads keep their
existing registry binding. A `Sub` returned for `Base` reaches the next in-memory pod
as that same `Sub` object. The stored column is `Base`: subclass-only fields are not
stored and do not enter the row hash. Two such values differing only in those extra
fields therefore have the same stored row hash. Likewise, a `datetime` declared as
`date` is passed on unchanged in memory, but storage contains only its date.

A consequence to know: the declared type's logical type decides what is stored, so a
function should read only what its declared parameter type has. A function that reads a
field its declared parameter type does not have (a subclass-only field, or the time of a
`datetime` declared as `date`) gets results that depend on the execution path: standalone
streams and the asynchronous orchestrator pass the in-memory object, while the synchronous
orchestrator and any later read pass the stored base value. Such results can also be reused
across values, because two values that differ only in what is not stored have the same row
hash and the second is served the first one's stored result. This follows from Python
subclassing and is left as it is (ITL-978).

No warning-and-convert path remains for function outputs. Make container conversions
explicit:

```python
@function_pod(output_keys="weights")
def twice(weight: float) -> list[float]:
return list((weight, weight * 2))

for tag, data in twice.pod(weights).iter_data():
print(data.as_dict())
# {'weights': [25.3, 50.6]}
```

Previously stored rows are served unchanged; this check does not repair or recompute them.
Input-source conformance remains outside this change.

## `PipelineJob` -- database-backed, cached execution

Applying a pod directly, as above, computes in memory and keeps nothing. To store results and
Expand Down
Loading
Loading