diff --git a/.zed/rules b/.zed/rules index ac56134d0..7e13538e2 100644 --- a/.zed/rules +++ b/.zed/rules @@ -293,8 +293,8 @@ A class a user defines is one of two kinds (developer, November 12, 2026, ITL-97 function annotated with a structural class may receive another class of the same identity; a structural class promises its fields, not its methods. `from_instance` is the only conversion, and Orcapod never calls it. -- A value is stored only as its own class (`UserClassLogicalType.accepts_instance`); never - store a subclass instance or a look-alike object in another class's column. +- A declared class stores its own instances and subclasses (`UserClassLogicalType.accepts_instance`); + only the declared class fields enter storage. Unrelated look-alike classes are refused. - The identity record (`logical_types/user_class_identity.py`) is `{kind, name, fields}` with JSON type tokens. The Arrow extension name is `name@` and the record is the extension metadata; together they are the logical type identifier. A nested @@ -632,6 +632,11 @@ data. The rules: expands or converts a type implicitly; the only exception is normalization to large variants. Inputs that disagree on a column's type are an error naming the column and both types, on every execution path (sync and async). + Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base. + Structural classes additionally accept equivalent structural identities. Numeric + narrowing and unrelated container types require explicit conversion. Base storage + omits subclass-only fields, and those fields do not enter the stored row hash. + - Every output column takes the large-normalized form of its input type (arrow_utils.normalize_to_large_types: string to large_string, binary to large_binary, list to large_list, recursively). Width, nullability and logical (extension) types are never diff --git a/CLAUDE.md b/CLAUDE.md index 6bb15d477..59048c0a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -304,8 +304,8 @@ A class a user defines is one of two kinds (developer, November 12, 2026, ITL-97 function annotated with a structural class may receive another class of the same identity; a structural class promises its fields, not its methods. `from_instance` is the only conversion, and Orcapod never calls it. -- A value is stored only as its own class (`UserClassLogicalType.accepts_instance`); never - store a subclass instance or a look-alike object in another class's column. +- A declared class stores its own instances and subclasses (`UserClassLogicalType.accepts_instance`); + only the declared class fields enter storage. Unrelated look-alike classes are refused. - The identity record (`logical_types/user_class_identity.py`) is `{kind, name, fields}` with JSON type tokens. The Arrow extension name is `name@` and the record is the extension metadata; together they are the logical type identifier. A nested @@ -674,6 +674,11 @@ data. The rules: expands or converts a type implicitly; the only exception is normalization to large variants. Inputs that disagree on a column's type are an error naming the column and both types, on every execution path (sync and async). + Function-output rule (developer, November 12, 2026): the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base. + Structural classes additionally accept equivalent structural identities. Numeric + narrowing and unrelated container types require explicit conversion. Base storage + omits subclass-only fields, and those fields do not enter the stored row hash. + - Every output column takes the large-normalized form of its input type (`arrow_utils.normalize_to_large_types`: `string` to `large_string`, `binary` to `large_binary`, `list` to `large_list`, recursively). Width, nullability and logical (extension) types are never diff --git a/changelog.d/ITL-895.md b/changelog.d/ITL-895.md new file mode 100644 index 000000000..9063dafac --- /dev/null +++ b/changelog.d/ITL-895.md @@ -0,0 +1,36 @@ +### Added + +- `orcapod.errors.OutputTypeError` identifies function-output mismatches by function, + output column, declared type and returned type, with an explicit-conversion hint. + +### Fixed + +- Function outputs are validated before iteration, hashing or storage on standalone, + cached, sync and async paths (ITL-895, ITL-901). Float-for-int truncation and unordered + set-for-list conversion are rejected. Nested values follow the same checks; exact and + structural user classes reuse their logical type compatibility, and ndarray mismatches + use the same error. Diagnostics do not expose returned mapping keys or converter values. +- Same-kind numeric values are accepted silently only when represented exactly, including + NumPy scalars for Python numbers and in-range Python integers for NumPy integer types. + Numeric narrowing, non-subclass kind changes and iterable-to-list require explicit + conversion. Subclasses are accepted unchanged in memory; declared base storage omits + subclass-only fields and their contribution to row hashes. This includes bool for int + and datetime for date, whose stored form is the declared base type. Accepted numeric + exceptions are normalized to the declared type immediately, so downstream pods see the + value that is stored. A subclass value is the exception: downstream pods in memory + receive the object as returned (a `datetime` declared as `date`, or a user-class + subclass with its extra fields), while storage holds only the declared base form. + Conforming-data hashes are unchanged. +- A declared `dict` accepts only a `dict` instance (other mappings such as + `MappingProxyType` need `dict(value)`), and a declared `Literal` accepts only its + members, matched by type and value with no numeric exception. + +### Known limitations + +- Previously stored rows remain unchanged and are not automatically recomputed or repaired. + Python-value inputs (sources and datagrams) remain outside output validation (ITL-895). +- The declared type's logical type decides what is stored. A function that reads fields its + declared parameter type does not have (the extra fields of a subclass returned for a base + type, or the time of a `datetime` declared as `date`) gets results that can depend on the + execution path and can be reused across values that differ only in those fields + (ITL-978). diff --git a/design_issues/ITL-895-FO1.md b/design_issues/ITL-895-FO1.md new file mode 100644 index 000000000..ff3cf468b --- /dev/null +++ b/design_issues/ITL-895-FO1.md @@ -0,0 +1,35 @@ +## `src/orcapod/core/data_function.py` — returned values not checked against the declared type + +### FO1 — A pod declared `-> int` stored `1.5` as `1` +**Status:** resolved +**Severity:** high +**Issue:** ITL-895 + +`PythonDataFunction._build_output_data` built `Data(output, python_schema=declared)` without +looking at the values. The Arrow table was built later with +`pa.Table.from_pylist(rows, schema=...)`, and the type converter passes primitives through +unchanged. pyarrow's integer converter calls `__int__` on a non-`int`, so `1.5`, +`numpy.float64(1.5)`, `Decimal` and `Fraction` were truncated. Numbers were read as epoch ticks +for date and timestamp fields, `bytes` and `str` were decoded and encoded into each other, and +`None` became a NULL in a not-null field. The `Data` kept the raw dict, so `as_dict()` gave +`1.5` while `as_table()`, the store and the hash gave `1`. Downstream pods read `as_dict()`, so +the same graph gave different results standalone, on the sync orchestrator and on the async +orchestrator. Present in every release up to v0.2.0. Root-cause evidence: +`superpowers/specs/itl-895-float-to-int/FINDINGS.md`. + +**Fix (function outputs):** The shared `semantic_types/output_conformance.py` check +runs in `PythonDataFunction._build_output_data` for both sync and async execution. It +checks primitive and nested types, incorporates the ndarray guard, and delegates exact or +structural class acceptance to the active logical type. Mismatches raise `OutputTypeError` +with one diagnostic on standalone and stored paths. Returned values and mapping keys are +absent from diagnostic text and chained converter tracebacks. + +The November 12 decision supersedes the earlier warning conversions: only exactly represented +same-kind numeric values are accepted across Python/NumPy types. Unrelated container types, numeric narrowing, and non-subclass kind changes require +explicit conversion. Python subclasses are accepted; the declared logical type controls +storage. Base storage omits subclass-only fields (and their contribution to row hashes); +`bool` for `int` and `datetime` for `date` also follow this subclass rule. Storage is validated eagerly; numeric exceptions are normalized while compatible +user-class objects retain their identity in memory. +Regressions: `tests/test_core/function_pod/test_output_conformance.py`. + +Rows stored wrongly before the fix are served unchanged. They are not repaired automatically. diff --git a/design_issues/ITL-895-FO2.md b/design_issues/ITL-895-FO2.md new file mode 100644 index 000000000..3445cd215 --- /dev/null +++ b/design_issues/ITL-895-FO2.md @@ -0,0 +1,13 @@ +## `src/orcapod/core/data_function.py` — returned values not checked against the declared type + +### FO2 — Inputs built from Python values are not checked against their declared types +**Status:** open +**Severity:** medium +**Issue:** ITL-895 + +Inputs built from Python values (`DictSource` and `ListSource` with a declared schema, +`Datagram`, `Data` and `Tag` from dicts, `Datagram.update` / `with_columns`) still go through +the unchecked pyarrow conversion and truncate the same way (`1.5` for an `int` column is +stored as `1`). Evidence: `superpowers/specs/itl-895-float-to-int/04_input_sites.py`. +Deferred by the developer (2026-11-11): "generally a big implementation and given the nature +of Python, not sure if it's something I'd care to put explicit check". diff --git a/design_issues/ITL-895-FO3.md b/design_issues/ITL-895-FO3.md new file mode 100644 index 000000000..d7d73561e --- /dev/null +++ b/design_issues/ITL-895-FO3.md @@ -0,0 +1,32 @@ +## `src/orcapod/core/data_function.py` — returned values not checked against the declared type + +### FO3 — A function that reads subclass-only fields gets path-dependent, reusable results +**Status:** open +**Severity:** medium +**Issue:** ITL-895 + +A subclass returned for a declared base type is passed on in memory as it is and stored as +the base: the declared type's logical type decides what is stored, and the row hash covers +the stored fields only. A downstream function that reads a field its declared parameter type +does not have (`getattr(y, "extra", ...)` on a `Reading` parameter that received an +`ExtendedReading`) therefore gets results that depend on the execution path, and results that +are reused across values: + +- standalone and the async orchestrator hand the function the in-memory subclass object + (within one async run a row may instead be served the result another row of the same + row hash stored first; which happens depends on timing, seen on Python 3.13 in CI); +- the sync orchestrator hands it the stored base value, without the extra field; +- two values that differ only in subclass-only fields have the same row hash, so a later row + is served the result computed for the earlier one, without a call. + +The same holds for a `datetime` returned for a declared `date`. Also in scope: a `list` or +`dict` subclass returned for a declared `list[T]` / `dict[K, V]` is rebuilt as a plain +`list` / `dict` in memory. + +Decision (developer, 2026-11-12): "this is a limitation of Python language by design and +... we should not try to fight it. How well this is handled is thus the job of the extension +logical type and we shouldn't over engineer a solution to restrict down things." No code +change before v0.3; follow-up ITL-978 (https://linear.app/metamorphic/issue/ITL-978). + +Current behaviour is pinned by `TestSubclassOnlyFieldsDownstream` in +`tests/test_core/function_pod/test_output_conformance.py`. diff --git a/design_issues/ITL-972-UCK1.md b/design_issues/ITL-972-UCK1.md index 29b7ac343..aee8c4866 100644 --- a/design_issues/ITL-972-UCK1.md +++ b/design_issues/ITL-972-UCK1.md @@ -23,7 +23,7 @@ own class. - An ordinary dataclass or pydantic model is an **exact** class: named by module path and qualified name, never equivalent to another class. Python subclassing still applies at a - pod input. A value is stored only as its own class. + pod input. An output accepts a subclass, but the declared logical type controls its stored fields (ITL-895). - `orcapod.dataclass` makes a **structural** class: named by its declared name (default the class name); two structural classes with equal identity records are compatible in the schema check and their instances compare equal. Orcapod converts nothing: a value is @@ -55,3 +55,9 @@ converts the data) but not over Arrow IPC or the C stream interface, where Arrow ITL-602; nested struct fields since this fix. Orcapod's own writers and Polars round trips never produce non-large storage. Pinned by a strict xfail in `tests/test_user_types/test_read_path.py`. + +**Output-boundary follow-up (ITL-895):** a subclass returned for a base-class output is +accepted on every path. It passes unchanged in memory, while the declared base logical type +stores only base fields; extra fields are omitted from storage and row hashes. Compatible +structural objects are validated without replacing them; stored reads retain their registry +binding. Unrelated classes are rejected with `OutputTypeError` at the shared output boundary. diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 06a48dec6..f7b7cb118 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -142,6 +142,107 @@ The same step is available as a stream method: `wide.select_data_columns(["weigh Tag columns are never affected by this rule -- a function pod passes all tags through untouched. +## Return values must match the declared type + +Each value the function returns is checked against its declared output type before it is +stored or hashed. A `float` returned for an output declared `int` is rejected; it is not +truncated. The `OutputTypeError` names the function, the output, the declared type and the +type of the value (never the value itself, which stays on the error's `value` attribute), +and says what to return instead: + +```python +from orcapod.errors import OutputTypeError + +@function_pod(output_keys="half_weight") +def halve(weight: float) -> int: + return weight / 2 # a float, although the function says it returns an int + +weights = DictSource(data=[{"subject_id": "mouse_01", "weight": 25.3}], tag_columns=["subject_id"]) + +try: + list(halve.pod(weights).iter_data()) +except OutputTypeError as err: + print(err) +# Function 'halve' returned float for output 'half_weight', which is declared int. The output does not conform. Return an int instead: round(value), or int(value) to truncate. +``` + +Fix the function (`return round(weight / 2)`), or declare the type it really returns +(`-> float`). + +`OutputTypeError` is a `TypeError` (and a `ValueError`). Inside a `PipelineJob` it counts as +a failure of the function for that row, like an exception the function raises itself. With +the default `error_policy="continue"` the row is skipped and reported to the observer, and +the other rows are computed and stored. With `error_policy="fail_fast"` the synchronous +orchestrator stops the run; the asynchronous orchestrator does not stop for row failures yet +(ITL-879, see below). + +The function-output rule is: the returned value must be an instance of the declared type, or an exact numeric equivalent; a subclass stands in for its base and is stored as the base. + +| Declared | Returned | Result | +|---|---|---| +| `int` / `float` / `bool` | same-kind NumPy scalar, represented exactly | accepted silently | +| NumPy integer type | Python or NumPy integer within range | accepted silently; out of range rejected | +| `int` / `float` | value-preserving numeric subclass such as `IntEnum` | accepted silently | +| `float` | `numpy.float32`, including `numpy.float32(0.1)` | accepted: widening preserves the original float32 value exactly | +| `numpy.float32` | Python `float` or `numpy.float64`, even `0.5` | rejected: explicitly return `numpy.float32(value)` | +| `float` | NumPy floating scalar wider than 64 bits | rejected: explicitly return `float(value)` | +| `list[T]` | tuple, NumPy array, range, deque, dict view, or iterator | rejected: return `list(value)` | +| `list[T]` | set or frozenset | rejected: return `sorted(value)` for deterministic order | +| `dict[K, V]` | another mapping, such as `MappingProxyType` or `UserDict` | rejected: return `dict(value)` | +| `Literal[...]` | a value that is not a member, or equals one with another type (`True`, `1.0` or `numpy.int64(1)` for `Literal[1]`) | rejected: a `Literal` names exact values, so no numeric exception applies | +| `date` | `datetime` | accepted as a Python subclass; storage keeps the date only | +| `int` | `bool` | accepted as a Python subclass; stored as an integer | +| `int` | float, Decimal, Fraction | rejected; choose an explicit conversion | +| `bool` | `int` | rejected: return `bool(value)` explicitly | +| floating type | integer or bool | rejected: return a floating value explicitly | +| non-optional type | `None` | rejected; declare `T \| None` if null is intended | +| ordinary dataclass or pydantic model | subclass | accepted; storage keeps only declared base fields | +| ordinary dataclass or pydantic model | unrelated class | rejected | +| structural class | another structural class with the same identity | accepted; fields follow the same recursive checks | + +Python subclassing takes precedence: `bool` for `int` is accepted, while `int` for +`bool` is rejected. Numeric exceptions otherwise require the same kind. +Nested list elements, dictionary keys and values, and stored user-class fields follow the +same policy. A mismatch raises `OutputTypeError` at the common function-output boundary, +before standalone iteration, table materialization, or sync/async storage can diverge. +Errors name the function, output, declared type and returned type; nested locations show +structure without including returned dictionary keys or converter messages. + +Outputs are validated against the declared storage schema immediately, and accepted numeric +exceptions are normalized so downstream pods, hashes and stored rows see the same value. +Eager validation never replaces a compatible user-class object; stored reads keep their +existing registry binding. A `Sub` returned for `Base` reaches the next in-memory pod +as that same `Sub` object. The stored column is `Base`: subclass-only fields are not +stored and do not enter the row hash. Two such values differing only in those extra +fields therefore have the same stored row hash. Likewise, a `datetime` declared as +`date` is passed on unchanged in memory, but storage contains only its date. + +A consequence to know: the declared type's logical type decides what is stored, so a +function should read only what its declared parameter type has. A function that reads a +field its declared parameter type does not have (a subclass-only field, or the time of a +`datetime` declared as `date`) gets results that depend on the execution path: standalone +streams and the asynchronous orchestrator pass the in-memory object, while the synchronous +orchestrator and any later read pass the stored base value. Such results can also be reused +across values, because two values that differ only in what is not stored have the same row +hash and the second is served the first one's stored result. This follows from Python +subclassing and is left as it is (ITL-978). + +No warning-and-convert path remains for function outputs. Make container conversions +explicit: + +```python +@function_pod(output_keys="weights") +def twice(weight: float) -> list[float]: + return list((weight, weight * 2)) + +for tag, data in twice.pod(weights).iter_data(): + print(data.as_dict()) +# {'weights': [25.3, 50.6]} +``` + +Previously stored rows are served unchanged; this check does not repair or recompute them. +Input-source conformance remains outside this change. + ## `PipelineJob` -- database-backed, cached execution Applying a pod directly, as above, computes in memory and keeps nothing. To store results and diff --git a/docs/concepts/your-own-data-types.md b/docs/concepts/your-own-data-types.md index 33444c868..3986af0d1 100644 --- a/docs/concepts/your-own-data-types.md +++ b/docs/concepts/your-own-data-types.md @@ -252,11 +252,11 @@ module path, the name and the fields are unchanged. Python's subclass rule holds: a pod annotated with a class accepts a column of a subclass. The subclass remains its own type, with its own fields and tables. -A column holds one class. To return a subclass from a pod, declare it: a pod annotated +A column has its declared logical type. A pod annotated `-> Child` produces a `Child` column with all of `Child`'s fields, and a downstream parameter `p: Base` accepts that column. A pod annotated `-> Base` that returns a `Child` -instance is an error (`TypeError`) when its output is turned into a table or stored: the -value is refused, never stored as a `Base` without its own fields. +instance is accepted: in memory it remains a `Child`, while the declared `Base` logical +type stores only the base fields. Subclass-only fields do not enter the stored row hash. A dataclass and a pydantic model are never the same type, and neither is ever the same type as an `op.dataclass`. @@ -298,9 +298,8 @@ Neither case is reported as missing data, and nothing stored is changed. - An ordinary dataclass or pydantic model defined inside a function cannot be stored: it has no module path to be found by. An `op.dataclass` may be defined anywhere. -- Each subclass is its own type with its own fields; it is never stored as its parent. A - value is stored only as its own class: a pod annotated `-> Base` that returns a `Child` - instance fails when the result is stored, instead of losing the child's fields. +- Each subclass has its own identity. Declaring a base-class output accepts subclass + instances but stores only base fields; declare the subclass to store its extra fields. - There is no structural pydantic model: a pydantic model is always identified by its exact class. - Results stored by Orcapod versions before 0.3 identified classes by module and class @@ -312,3 +311,11 @@ Neither case is reported as missing data, and nothing stored is changed. - [Function Pods](function-pods.md): the pods that take and return these classes. - [Identity & Hashing](identity.md): how a type's identity enters pipeline and content hashes. + +Function outputs accept instances of their declared class and subclasses (ITL-895). +A subclass object is passed unchanged to the next pod in memory. When returned for a +base-class output, only the base fields are stored and read back; extra subclass fields +are not included in the row hash. Two subclass instances differing only in extra fields +therefore have the same stored row hash. Declare the subclass to retain all its fields. +Compatible structural objects also pass unchanged in memory; stored reads use the registry +binding for their structural identity. diff --git a/src/orcapod/core/data_function.py b/src/orcapod/core/data_function.py index 15eaa2180..abcd22861 100644 --- a/src/orcapod/core/data_function.py +++ b/src/orcapod/core/data_function.py @@ -7,7 +7,6 @@ import uuid from abc import abstractmethod from collections.abc import Callable, Iterable, Sequence -import types import typing from typing import TYPE_CHECKING, Any, ClassVar, Generic, Literal, TypeVar @@ -22,12 +21,17 @@ get_function_components, ) from orcapod.core.result_cache import ResultCache +from orcapod.errors import OutputTypeError from orcapod.protocols.core_protocols import DataFunctionProtocol, DataProtocol from orcapod.protocols.core_protocols.executor import ( DataFunctionExecutorProtocol, PythonFunctionExecutorProtocol, ) from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.semantic_types.output_conformance import ( + check_conformance, + normalize_output_numbers, +) from orcapod.types import DataValue, Schema, SchemaLike from orcapod.utils import schema_utils from orcapod.utils.canonical_type_names import stabilize_type_names @@ -37,6 +41,9 @@ from orcapod.protocols.observability_protocols import DataExecutionLoggerProtocol if TYPE_CHECKING: + from orcapod.logical_types.protocols import ( + TypeConverterProtocol as LogicalTypeConverterProtocol, + ) import pyarrow as pa import pyarrow.compute as pc else: @@ -112,52 +119,6 @@ def parse_function_outputs( E = TypeVar("E", bound=DataFunctionExecutorProtocol) -def _declared_ndarray_type(declared: Any) -> type | None: - """Return the ndarray class a declared output type requires, if any. - - Handles ``np.ndarray`` (or a subclass), parameterized aliases such as - ``npt.NDArray[np.float32]``, and ``Optional``/``| None`` of either. - NumPy is never imported here: a class counts as an ndarray when its MRO - contains ``numpy.ndarray``. - """ - candidates = [declared] - if typing.get_origin(declared) in (typing.Union, types.UnionType): - candidates = [a for a in typing.get_args(declared) if a is not type(None)] - for candidate in candidates: - # A parameterized alias such as npt.NDArray[np.float32] has origin np.ndarray. - candidate = typing.get_origin(candidate) or candidate - if isinstance(candidate, type) and any( - base.__module__ == "numpy" and base.__qualname__ == "ndarray" - for base in candidate.__mro__ - ): - return candidate - return None - - -def _check_declared_ndarray_outputs( - output_data: dict[str, Any], output_schema: Schema -) -> None: - """Refuse a non-array returned for an output declared as a NumPy array (ITL-883). - - A NumPy scalar (for example ``x * 2`` on a 0-d array) is not an array, and - Orcapod never wraps one implicitly. - - Raises: - TypeError: Naming the output key, the declared type and the returned type. - """ - for key, value in output_data.items(): - if value is None: - continue - required = _declared_ndarray_type(output_schema.get(key)) - if required is not None and not isinstance(value, required): - raise TypeError( - f"Output {key!r} is declared as {required.__module__}.{required.__qualname__}" - f" but the function returned {type(value).__module__}." - f"{type(value).__qualname__} ({value!r}); return an array, for example" - " with np.asarray(...)" - ) - - class DataFunctionBase(TraceableBase, Generic[E]): """Abstract base class for DataFunctionProtocol. @@ -448,7 +409,6 @@ def __init__( self._function = function self._is_async = inspect.iscoroutinefunction(function) - # Reject functions with variadic parameters -- PythonDataFunction maps # data keys to named parameters, so the full parameter set must be fixed. _sig = inspect.signature(function) @@ -628,14 +588,45 @@ def is_async(self) -> bool: def _build_output_data(self, values: Any) -> DataProtocol: """Build an output Data from raw function return values. + Every returned value is checked against its declared output type and validated + against Arrow storage eagerly. Numeric exceptions are normalized; compatible + subclass and structural objects pass unchanged in memory. The declared logical + type controls their stored representation (ITL-895). + Without the check pyarrow would coerce some mismatches silently, for example + truncating ``1.5`` returned for ``-> int``. The shared conformance policy + permits only value-preserving numeric exceptions. + Args: values: Raw return value from the wrapped function. Returns: A Data containing the parsed outputs with source info. + + Raises: + OutputTypeError: If a returned value does not conform to its declared type, + or cannot be converted to it. """ output_data = parse_function_outputs(self._output_keys, values) - _check_declared_ndarray_outputs(output_data, self.output_data_schema) + output_schema = self.output_data_schema + for key, value in output_data.items(): + mismatch = check_conformance( + value, + output_schema[key], + # DataContext's legacy protocol omits the logical-type operations. + typing.cast( + "LogicalTypeConverterProtocol", + typing.cast(object, self.data_context.type_converter), + ), + ) + if mismatch is not None: + raise OutputTypeError( + self.canonical_function_name, + key, + output_schema[key], + value, + mismatch.describe(), + mismatch.hint, + ) def combine(*components: tuple[str, ...]) -> str: inner_parsed = [":".join(component) for component in components] @@ -646,14 +637,69 @@ def combine(*components: tuple[str, ...]) -> str: k: combine(self.uri, (new_record_uuid.hex,), (k,)) for k in output_data } + if not output_data: + return Data( + output_data, + source_info=source_info, + record_uuid=new_record_uuid, + python_schema=output_schema, + data_context=self.data_context, + ) + # Validate storage eagerly, but retain compatible user-class instances. Reading + # the table back here would rebind structural objects to the registry's class. + self._convert_outputs(output_data, output_schema) + normalized = { + key: typing.cast( + DataValue, normalize_output_numbers(value, output_schema[key]) + ) + for key, value in output_data.items() + } return Data( - output_data, + normalized, + python_schema=output_schema, source_info=source_info, record_uuid=new_record_uuid, - python_schema=self.output_data_schema, data_context=self.data_context, ) + def _convert_outputs( + self, output_data: dict[str, DataValue], output_schema: Schema + ) -> pa.Table: + """Convert checked outputs to a one-row Arrow table with the declared types. + + Args: + output_data: The parsed outputs, already checked against ``output_schema``. + output_schema: The declared output types. + + Returns: + The one-row table, typed by the declared schema. + + Raises: + OutputTypeError: If an output cannot be converted, for example an integer + out of range for its declared width; it names that output. + """ + converter = self.data_context.type_converter + try: + return converter.python_dicts_to_arrow_table( + [output_data], python_schema=output_schema + ) + except Exception as exc: + # Find the output that fails on its own, so the error can name it. + for key, value in output_data.items(): + try: + _ = converter.python_dicts_to_arrow_table( + [{key: value}], python_schema={key: output_schema[key]} + ) + except Exception: + raise OutputTypeError( + self.canonical_function_name, + key, + output_schema[key], + value, + type(exc).__name__, + ) from None + raise + def _call_async_function_sync(self, data: DataProtocol, **kwargs: Any) -> Any: """Run the wrapped async function synchronously. diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 94ada08a4..b06fae878 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -67,6 +67,100 @@ class DataFunctionUnavailableError(RuntimeError): """ +def type_text(annotation: object) -> str: + """Return a short, readable name for a type or annotation, for error messages. + + Builtin classes are shown bare (``int``); other classes with their module + (``numpy.int8``, ``decimal.Decimal``); generic aliases and unions as Python prints + them (``list[int]``, ``int | None``). Never use this for hashing or identity. + + Args: + annotation: A class or a typing annotation. + + Returns: + The display name. + """ + if isinstance(annotation, type) and not hasattr(annotation, "__origin__"): + module = annotation.__module__ + if module in ("builtins", "__main__"): + return annotation.__qualname__ + return f"{module}.{annotation.__qualname__}" + return str(annotation).replace("typing.", "") + + +class OutputTypeError(TypeError, ValueError): + """A function returned a value that does not match its declared output type. + + Orcapod rejects incompatible returned values (ITL-895): pyarrow would + otherwise truncate a ``float`` returned for an output declared ``int``. The data + function raises this before anything is stored or hashed, so a pipeline treats it + like any other failure of the function: a row failure under ``error_policy``, + reported to the observer's ``on_data_crash``. + + It subclasses both ``TypeError`` (a value of the wrong type) and ``ValueError`` + (it also wraps conversion failures of a value of the right type, such as an + integer out of range for the declared width or a naive ``datetime`` under the + strict timezone policy, which were ``ValueError`` before). + + Attributes: + function_name: Name of the function pod's function. + output_key: Name of the output column. + declared_type: The declared type of that output. + value: The value the function returned for that output. + """ + + def __init__( + self, + function_name: str, + output_key: str, + declared_type: object, + value: object, + detail: str | None = None, + hint: str | None = None, + ) -> None: + """Create the error for one output of one call. + + Args: + function_name: Name of the function pod's function. + output_key: Name of the output column. + declared_type: The declared type of that output. + value: The value the function returned for that output. + detail: Where inside the value the mismatch is, or why the conversion + failed, when that is not the value as a whole. + hint: What the function should return instead, when there is a specific + fix (``Return a float instead: 0.0 or float(value)``). + """ + self.function_name: str = function_name + self.output_key: str = output_key + self.declared_type: object = declared_type + self.value: object = value + self.detail: str | None = detail + self.hint: str | None = hint + where = f": {detail}" if detail else "" + fix = f". {hint}." if hint else "; convert the value in the function." + # Types only, never the value: pipelines log this message, and a returned value + # can be large or sensitive. The value itself stays on ``self.value``. + super().__init__( + f"Function {function_name!r} returned {type_text(type(value))} " + + f"for output {output_key!r}, which is declared " + + f"{type_text(declared_type)}{where}. The output does not conform{fix}" + ) + + def __reduce__(self) -> tuple[object, ...]: + """Pickle with the constructor arguments (executors send errors across processes).""" + return ( + type(self), + ( + self.function_name, + self.output_key, + self.declared_type, + self.value, + self.detail, + self.hint, + ), + ) + + class FieldNotResolvableError(LookupError): """ Raised when a source cannot resolve a field value for a given record ID. diff --git a/src/orcapod/logical_types/structural_logical_type_factory.py b/src/orcapod/logical_types/structural_logical_type_factory.py index 6f4ed19f6..2d4eb5815 100644 --- a/src/orcapod/logical_types/structural_logical_type_factory.py +++ b/src/orcapod/logical_types/structural_logical_type_factory.py @@ -48,7 +48,7 @@ def accepts_instance(self, value: Any, converter: TypeConverterProtocol) -> bool class with the same identity record. """ cls = type(value) - if cls is self._python_type: + if isinstance(value, self._python_type): return True if not uci.is_structural(cls): return False diff --git a/src/orcapod/logical_types/user_class_logical_type.py b/src/orcapod/logical_types/user_class_logical_type.py index f03c7d3b7..92ad15ca7 100644 --- a/src/orcapod/logical_types/user_class_logical_type.py +++ b/src/orcapod/logical_types/user_class_logical_type.py @@ -195,7 +195,7 @@ def python_to_storage( #: What ``accepts_instance`` requires, for the error message. instance_requirement: str = ( "an ordinary dataclass or pydantic model is stored only from instances of " - "that exact class. Build an instance of the class, or annotate the value " + "that class or a subclass. Build an instance of the class, or annotate the value " "with its own class." ) @@ -207,10 +207,10 @@ def accepts_instance(self, value: Any, converter: TypeConverterProtocol) -> bool converter: The active converter. Returns: - For an exact class: whether ``type(value)`` is the class object. A subclass - instance is refused, because its own fields would be lost. + Whether the value is an instance of the declared class or a subclass. + Storage uses the declared class's fields, excluding subclass-only fields. """ - return type(value) is self._python_type + return isinstance(value, self._python_type) def storage_to_python( self, storage_value: Any, converter: TypeConverterProtocol | None diff --git a/src/orcapod/semantic_types/output_conformance.py b/src/orcapod/semantic_types/output_conformance.py new file mode 100644 index 000000000..28a5aa601 --- /dev/null +++ b/src/orcapod/semantic_types/output_conformance.py @@ -0,0 +1,345 @@ +"""Function-output conformance (ITL-895, ITL-901). + +the returned value must be an instance of the declared type, or an exact numeric +equivalent; a subclass stands in for its base and is stored as the base. + +Logical types retain ownership of their compatibility rules. The caller adds function +and output context to a mismatch; this module never formats returned values. +""" + +from __future__ import annotations + +import dataclasses +import datetime as dt +import enum +import types +import typing +from collections.abc import Mapping +from dataclasses import dataclass + +from orcapod.errors import type_text +from orcapod.logical_types import user_class_identity +from orcapod.logical_types.user_class_logical_type import UserClassLogicalType + +if typing.TYPE_CHECKING: + from orcapod.logical_types.protocols import TypeConverterProtocol + + +class Treatment(enum.Enum): + """Whether a classified near miss is accepted or rejected.""" + + ACCEPT = "accept" + REJECT = "reject" + + +class NearMiss(enum.Enum): + """The explicit exceptions and rejected near misses of the output contract.""" + + NUMPY_SCALAR_SAME_KIND = "numpy_scalar_same_kind" + INT_FOR_FLOAT = "int_for_float" + BOOL_FOR_FLOAT = "bool_for_float" + NONE_FOR_NON_OPTIONAL = "none_for_non_optional" + ITERABLE_FOR_LIST = "iterable_for_list" + FLOAT_NARROWING = "float_narrowing" + + +OUTPUT_CONFORMANCE: Mapping[NearMiss, Treatment] = types.MappingProxyType( + { + case: Treatment.ACCEPT + if case is NearMiss.NUMPY_SCALAR_SAME_KIND + else Treatment.REJECT + for case in NearMiss + } +) + + +@dataclass(frozen=True) +class Nonconformance: + """A mismatch, with structural location and an explicit-conversion hint.""" + + path: str + expected: object + value: object + hint: str | None = None + + def describe(self) -> str | None: + """Describe nested types without exposing any returned content.""" + if not self.path: + return None + return f"{self.path} is {type_text(type(self.value))}, expected {type_text(self.expected)}" + + +def _at(mismatch: Nonconformance | None, path: str) -> Nonconformance | None: + """Attach the structural location of a nested mismatch.""" + return ( + dataclasses.replace(mismatch, path=path + mismatch.path) if mismatch else None + ) + + +def _numpy_scalar(value: object) -> tuple[str, int] | None: + """Return a NumPy scalar's dtype kind and byte width without importing NumPy eagerly.""" + if not type(value).__module__.startswith("numpy"): + return None + import numpy as np + + if isinstance(value, np.generic): + return value.dtype.kind, value.dtype.itemsize + return None + + +def _numpy_type(declared: object) -> tuple[str, int] | None: + """Return the kind and byte width of a declared NumPy scalar type.""" + if not isinstance(declared, type) or not declared.__module__.startswith("numpy"): + return None + import numpy as np + + if issubclass(declared, np.generic): + dtype = np.dtype(declared) + return dtype.kind, dtype.itemsize + return None + + +def _declared_ndarray_type(declared: object) -> type | None: + """Resolve an ndarray annotation or parameterized alias without importing NumPy.""" + candidates = ( + typing.get_args(declared) + if typing.get_origin(declared) in (typing.Union, types.UnionType) + else (declared,) + ) + for annotation in candidates: + candidate = typing.get_origin(annotation) or annotation + if isinstance(candidate, type) and any( + base.__module__ == "numpy" and base.__qualname__ == "ndarray" + for base in candidate.__mro__ + ): + return candidate + return None + + +class _Checker: + """Apply the output policy recursively using the active logical-type converter.""" + + def __init__(self, converter: TypeConverterProtocol) -> None: + """Use the active converter for user-class compatibility.""" + self.converter = converter + + def near( + self, case: NearMiss, value: object, declared: object + ) -> Nonconformance | None: + """Apply the single decision table and provide an explicit remedy.""" + if OUTPUT_CONFORMANCE[case] is Treatment.ACCEPT: + return None + if case in (NearMiss.INT_FOR_FLOAT, NearMiss.BOOL_FOR_FLOAT): + # A fixed example avoids disclosing the returned numeric value. + hint = ( + "Return a float instead: 0.0 or float(value)" + if declared is float + else f"Return {type_text(declared)}(value)" + ) + elif case is NearMiss.NONE_FOR_NON_OPTIONAL: + hint = f"Return a {type_text(declared)}, or declare the output {type_text(declared)} | None" + elif case is NearMiss.ITERABLE_FOR_LIST: + hint = "Return a list instead: list(value)" + if isinstance(value, (set, frozenset)): + hint = "Return a list in a fixed order instead, e.g. sorted(value)" + else: + hint = f"Return {type_text(declared)}(value) instead" + return Nonconformance("", declared, value, hint) + + def check(self, value: object, declared: object) -> Nonconformance | None: + """Return the first mismatch; unknown logical types are checked by conversion.""" + origin = typing.get_origin(declared) + args = typing.get_args(declared) + if origin in (typing.Union, types.UnionType): + non_none = [a for a in args if a is not type(None)] + if value is None and len(non_none) < len(args): + return None + if len(non_none) == 1: + return self.check(value, non_none[0]) + return None # unsupported complex unions are rejected by the converter + if origin is typing.Literal: + # A Literal names exact values: a member matches by type and value, so + # True is not 1 and no numeric exception applies. + if any(type(value) is type(member) and value == member for member in args): + return None + return Nonconformance( + "", declared, value, "Return one of the declared literal values" + ) + if value is None: + return self.near(NearMiss.NONE_FOR_NON_OPTIONAL, value, declared) + array_type = _declared_ndarray_type(declared) + if array_type is not None: + return ( + None + if isinstance(value, array_type) + else Nonconformance( + "", + declared, + value, + "Return an array, for example with np.asarray(...)", + ) + ) + numpy_type = _numpy_type(declared) + kind = ( + numpy_type[0] + if numpy_type + else "b" + if declared is bool + else "i" + if declared is int + else "f" + if declared is float + else None + ) + if kind in ("b", "i", "u", "f"): + return self.check_number(value, declared, kind, numpy_type) + if declared in (str, bytes, dt.date, dt.datetime, dt.time, dt.timedelta): + return ( + None + if isinstance(declared, type) and isinstance(value, declared) + else Nonconformance("", declared, value) + ) + if origin is list: + if not isinstance(value, list): + return self.near(NearMiss.ITERABLE_FOR_LIST, value, declared) + for index, item in enumerate(value): + mismatch = self.check(item, args[0] if args else object) + if mismatch: + return _at(mismatch, f"[{index}]") + return None + if origin is dict: + if not isinstance(value, dict): + return Nonconformance( + "", declared, value, "Return a dict instead: dict(value)" + ) + key_type, value_type = args if len(args) == 2 else (object, object) + for key, item in value.items(): + mismatch = _at(self.check(key, key_type), ".keys()") or _at( + self.check(item, value_type), "[*]" + ) + if mismatch: + return mismatch + return None + logical_type = self.converter.get_logical_type_for_python_type(declared) + if isinstance(logical_type, UserClassLogicalType): + if not logical_type.accepts_instance(value, self.converter): + return Nonconformance( + "", declared, value, logical_type.instance_requirement + ) + cls = logical_type.python_type + hints = user_class_identity.type_hints(cls) + for name in user_class_identity.stored_field_names(cls): + mismatch = self.check(getattr(value, name), hints[name]) + if mismatch: + return _at(mismatch, f".{name}") + return None + + def check_number( + self, + value: object, + declared: object, + kind: str, + numpy_type: tuple[str, int] | None, + ) -> Nonconformance | None: + """Require the same numeric kind and prevent overflow or floating narrowing.""" + if isinstance(declared, type) and isinstance(value, declared): + return None # Python subclassing is the primary acceptance rule. + scalar = _numpy_scalar(value) + actual_kind = ( + scalar[0] + if scalar + else ( + "b" + if isinstance(value, bool) + else "i" + if isinstance(value, int) + else "f" + if isinstance(value, float) + else None + ) + ) + if kind in ("i", "u"): + if actual_kind not in ("i", "u"): + return Nonconformance( + "", + declared, + value, + "Return an int instead: round(value), or int(value) to truncate", + ) + if numpy_type is not None: + import numpy as np + + limits = np.iinfo(typing.cast(type, declared)) + number = int(typing.cast(typing.SupportsInt, value)) + if not limits.min <= number <= limits.max: + return Nonconformance( + "", + declared, + value, + f"Return a value within the range of {type_text(declared)}", + ) + elif kind == "b": + if actual_kind != "b": + return Nonconformance( + "", declared, value, "Return bool(value) explicitly" + ) + elif kind == "f": + if actual_kind in ("i", "u", "b"): + case = ( + NearMiss.BOOL_FOR_FLOAT + if actual_kind == "b" + else NearMiss.INT_FOR_FLOAT + ) + return self.near(case, value, declared) + if actual_kind != "f": + return Nonconformance("", declared, value) + declared_width = numpy_type[1] if numpy_type else 8 + actual_width = scalar[1] if scalar else 8 + if actual_width > declared_width: + return self.near(NearMiss.FLOAT_NARROWING, value, declared) + if type(value) is declared or (numpy_type is None and scalar is None): + return None + return self.near(NearMiss.NUMPY_SCALAR_SAME_KIND, value, declared) + + +def check_conformance( + value: object, declared: object, converter: TypeConverterProtocol +) -> Nonconformance | None: + """Check one returned value using its declared type and active logical-type rules.""" + return _Checker(converter).check(value, declared) + + +def normalize_output_numbers(value: object, declared: object) -> object: + """Normalize already-validated numeric values without rebuilding user-class objects. + + This performs no compatibility decisions: check_conformance and declared-schema + storage validation must succeed first. Containers are copied, never mutated. + """ + origin = typing.get_origin(declared) + args = typing.get_args(declared) + if value is None: + return None + if origin in (typing.Union, types.UnionType): + non_none = [a for a in args if a is not type(None)] + if len(non_none) == 1: + return normalize_output_numbers(value, non_none[0]) + numpy_type = _numpy_type(declared) + kind = numpy_type[0] if numpy_type else None + if isinstance(declared, type) and isinstance(value, declared): + return value # Preserve subclass objects; the declared converter owns storage. + if declared is bool or kind == "b": + return bool(value) + if declared is int or kind in ("i", "u"): + return int(typing.cast(typing.SupportsInt, value)) + if declared is float or kind == "f": + return float(typing.cast(typing.SupportsFloat, value)) + if origin is list and isinstance(value, list): + return [normalize_output_numbers(item, args[0]) for item in value] + if origin is dict and isinstance(value, dict): + return { + normalize_output_numbers(key, args[0]): normalize_output_numbers( + item, args[1] + ) + for key, item in value.items() + } + return value diff --git a/src/orcapod/semantic_types/universal_converter.py b/src/orcapod/semantic_types/universal_converter.py index 28b679490..4fdb1873b 100644 --- a/src/orcapod/semantic_types/universal_converter.py +++ b/src/orcapod/semantic_types/universal_converter.py @@ -1675,6 +1675,11 @@ def _convert_datetime(dt: datetime) -> datetime: origin = get_origin(python_type) args = get_args(python_type) + if python_type is int: + # bool is an int subclass, but Arrow refuses it for an integer field. + # Preserve the Python object in memory; encode its declared base here. + return lambda value: int(value) if isinstance(value, bool) else value + if python_type in {int, float, str, bool, bytes} or origin is None: # Basic types - no conversion needed return lambda value: value diff --git a/superpowers/plans/2026-11-11-itl-895-decisions.md b/superpowers/plans/2026-11-11-itl-895-decisions.md new file mode 100644 index 000000000..88dde2112 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-895-decisions.md @@ -0,0 +1,165 @@ +# ITL-895 Developer Decisions (D1-D7, ITL-901) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +Follows `2026-11-11-itl-895-float-to-int.md` (the draft PR #299). That plan built the +mechanism; this one applies the developer's decisions of 2026-11-11. + +## Status log (for a successor) + +- 2026-11-11: Plan written. The warning mechanism (section "Design") and the open question + Q2 went to the coordinator through `ask`. +- 2026-11-11: Coordinator: Q1 approved as designed (the name `OutputConversionWarning` is a + new public name it confirms with the developer; keep a rename one line). Q2: (a), + provisionally, structured so (b) is a small switch (`iterable_for_list` in the table). PR #300 + (tensors) will import `orcapod.errors.OutputConversionWarning` too; whichever PR merges + second reconciles. Finding: `tuple[...]` is not a supported output annotation at all + ("Unsupported annotation"), so the reverse-of-D5 case cannot occur; the checker's + `tuple[...]` branch is dead code and is removed. + +- 2026-11-11: Developer, final: the category is `orcapod.errors.ValueConversionWarning` + (plain `UserWarning` subclass, docstring verbatim from the coordinator, shared with PR #300; + the caller builds the message), deduplicated explicitly per function instance and (output, + location, near miss), marked only after `warnings.warn` returns; Q2 option (a) final. + Done: tests (red, then green), rule table, warning, explicit dedup, docs table, CHANGELOG, + DESIGN_ISSUES (FO1 resolved, FO2 inputs open/deferred), merge of origin/main (ITL-878, + ITL-821), no-debug-only-parameters refactor of the checker (containers annotate results). + Dedup evidence: `superpowers/specs/itl-895-float-to-int/09_warning_dedup.py`. Finding: + `PipelineJob` runs grow superlinearly with rows on main too (1,000 rows: minutes), so the + pipeline dedup scenarios use 1,000 rows. **Next:** full suite on the final head, PR body, + mark ready, CI, CodeRabbit (review bodies too), worker_done. + +**Goal:** `OUTPUT_CONFORMANCE` applies the developer's decisions: each near-miss value is +either accepted silently, accepted and converted with an `OutputConversionWarning`, or rejected +with an `OutputTypeError` that says what to return instead. + +**Architecture:** The rule table's fields become a three-valued `Treatment` (`ACCEPT`, +`CONVERT_WITH_WARNING`, `REJECT`). The checker returns the first mismatch and the conversions +it found. `PythonDataFunction._build_output_data` raises for a mismatch, and otherwise warns +once per conversion from a single `warnings.warn` call before the eager conversion that +already exists. The stored value and the value downstream pods receive stay the same object +path (the eager Arrow conversion), so they cannot diverge. + +**Tech Stack:** Python 3.11-3.13, pyarrow, numpy, pytest, `uv run`. + +--- + +## Decisions (developer, 2026-11-11) + +| Case | Declared | Returned | Treatment | +|---|---|---|---| +| D1 | `float` (any float type) | `int` / NumPy integer | reject; message says `return 0.0` / `float(x)` | +| D2 | `int` / `bool` / `float` (or NumPy type of that kind) | NumPy scalar of the same kind, value-exact | accept | +| D3 | `float` | `bool` / `numpy.bool_` | reject | +| D4 | not optional | `None` | reject (unchanged) | +| D5 | `list[T]` | `tuple` | accept, converted to `list`, warning | +| D6 | `date` | `datetime` | accept, time dropped, warning (reverses the draft) | +| D7 | `numpy.float32` | Python `float` / wider NumPy float | accept, rounded to 32 bits, warning | +| ITL-901 | `list[T]` | `set` / `frozenset` | reject (order differs between processes) | + +Everything else in the draft stands. Inputs (sources, `Datagram`) are **deferred** by the +developer: no change; DESIGN_ISSUES records it as open/deferred. + +Finding while planning: under the draft, other iterables for `list[T]` pass **unchecked**, and +pyarrow truncates inside them: `numpy.array([1.5, 2.0])` returned for `list[int]` is stored as +`[1, 2]`. See Q2. + +## Design (sent to the coordinator) + +- `orcapod.errors.OutputConversionWarning(UserWarning)`, next to `OutputTypeError`. It is + built from `function_name`, `output_key`, `location`, `declared_type`, `returned_type`, + `conversion` and `hint` and keeps them as attributes. Message, with **no per-row value**: + - D5: `Function 'f' returned tuple for output 'y', which is declared list[int]; Orcapod + converted it to a list. Return a list (list(value)) to avoid this warning.` + - D6: `Function 'f' returned datetime.datetime for output 'd', which is declared + datetime.date; Orcapod dropped the time of day. Return value.date() to avoid this warning.` + - D7: `Function 'f' returned float for output 'y', which is declared numpy.float32; Orcapod + rounded it to 32 bits. Return numpy.float32(value) to avoid this warning.` + - Nested values name a structural location, never an index or key that varies by row: + `for output 'y' at [*]` (list element, dict value), `.field` (dataclass field), `[0]` + (position in a fixed `tuple[...]`). +- Raised from one line (`warnings.warn(warning, stacklevel=1)` in a helper of + `data_function.py`), so the default `"default"` action shows it once per distinct message + per process: one warning per function, output, location and conversion kind, however many + rows. `-W error::orcapod.errors.OutputConversionWarning` turns it into an exception, which is + then a row failure like `OutputTypeError` (a strict mode for free). `-W always` shows every + row. +- The checker is pure: it returns `ConformanceReport(mismatch, conversions)`, and the caller + (`_build_output_data`, which knows the function and output) builds the error / warning. No + helper receives a value only for a message. +- Cached rows are not recomputed, so they do not warn again. +- Async orchestrator: the function runs in an executor thread, but `_build_output_data` runs on + the event loop thread after it returns; the module registry is shared, so dedup holds. +- Tests: `pytest.warns(errors.OutputConversionWarning, match=...)`; dedup tested with + `warnings.catch_warnings(record=True)` + `simplefilter("default")` over many rows (pytest + resets the registry per test). A 10,000-row run, sync and async, is verified by a script and + reported. +- Assumption: a `list` returned for a declared `tuple[...]` is the reverse of D5 and gets the + same treatment (accept, converted, warning). + +**Q2** (asked): other iterables for `list[T]` (NumPy array, `deque`, `range`, dict views, +generators). Recommended (a): re-iterable ordered containers are accepted with the D5 warning, +their elements checked (an ndarray by its dtype, not per element); one-shot iterators +(generators) are rejected with `return list(...)`; `set` / `frozenset` rejected (ITL-901). +(b) strict: only `list`, plus `tuple` with a warning. (c) draft: unchecked. + +## Files + +- Modify: `src/orcapod/semantic_types/output_conformance.py` (Treatment, table, report, hints). +- Modify: `src/orcapod/errors.py` (`OutputConversionWarning`; `OutputTypeError` takes a hint). +- Modify: `src/orcapod/core/data_function.py` (`_build_output_data` warns). +- Modify: `tests/test_core/function_pod/test_output_conformance.py`. +- Modify: `docs/concepts/function-pods.md`, `CHANGELOG.md`, `DESIGN_ISSUES.md`, PR body. + +## Task 1: Tests first (after the coordinator's reply) + +- [ ] Replace `TestUndecidedCasesKeepTodaysBehaviour` with: + - `REJECTED_BY_DECISION`: `float<-0` (D1; message contains `0.0` and `float(`), + `float<-np.int64(1)` (D1), `np.float32<-1` (D1), `float<-True`, `float<-np.bool_(True)` + (D3), `list[int]<-{1, 2}`, `list[int]<-frozenset({1})` (ITL-901), plus Q2 cases. + - `ACCEPTED_SILENTLY` (assert no `OutputConversionWarning` via + `warnings.catch_warnings(record=True)` + `simplefilter("always")`): `int<-np.int64(3)`, + `bool<-np.bool_(True)`, `float<-np.float64(1.5)`, `float<-np.float32(0.5)`, + `np.float32<-np.float32(0.1)`, every conforming value. + - `CONVERTED_WITH_WARNING`: `list[int]<-(1, 2)` -> `[1, 2]`, `dt.date<-datetime(2020,1,1,12)` + -> `date(2020,1,1)`, `np.float32<-0.1` -> `float(np.float32(0.1))`, + `tuple[int, int]<-[1, 2]`, nested `dict[str, list[int]]<-{"a": (1,)}` (location `[*]`). + Each: `pytest.warns(..., match=, , declared, returned type)`, stored value. + - Downstream sees the converted value, standalone and `PipelineJob` sync/async x + memory/delta: tuple -> `list` seen by `down`, datetime -> `date` (not `datetime`), + `0.1` -> `float(np.float32(0.1))`. + - Dedup: 1,000 rows under `simplefilter("default")` -> exactly one warning per kind. + - Rejections in `PipelineJob` sync/async x memory/delta: a row failure with nothing stored. +- [ ] Run, show the failures, commit `test(function_pod): developer decisions D1-D7 (ITL-895)`. + +## Task 2: Rule table and checker + +- [ ] `Treatment` enum; `ConformanceRule` fields `int_for_float`, `numpy_scalar_same_kind`, + `bool_for_float`, `none_for_non_optional`, `tuple_for_list`, `datetime_for_date`, + `float_for_float32`, `set_for_list`, `iterable_for_list` (Q2); the table per the decisions. +- [ ] `check_conformance(value, declared, rule) -> ConformanceReport` (replaces + `find_nonconformance`); `Nonconformance` gains `hint`; `Conversion(location, expected, + returned_type, conversion, hint)`; conversions deduplicated within one value. +- [ ] Run the tests; commit `feat(output_conformance): apply ITL-895 decisions`. + +## Task 3: Warning class and `_build_output_data` + +- [ ] `OutputConversionWarning` in `errors.py`; `OutputTypeError` appends the hint. +- [ ] `_build_output_data`: raise on mismatch; `_warn_conversions` issues each warning from one + line; then the existing eager conversion. Run tests; commit. + +## Task 4: Docs, CHANGELOG, DESIGN_ISSUES + +- [ ] `docs/concepts/function-pods.md`: the full table (accepted / converted with a warning / + rejected), a runnable warning sample, the dedup and `-W error` notes. +- [ ] CHANGELOG `[0.3.0] — unreleased`: update the ITL-895 entry, mention ITL-901. +- [ ] DESIGN_ISSUES FO1: resolved for outputs; inputs open/deferred with the developer's reason. + +## Task 5: Verify and finish + +- [ ] 10,000-row script, sync and async, default filters: count warnings; record in this file. +- [ ] No-debug-only-parameters review of the whole diff. +- [ ] ruff check, ruff format --check, basedpyright (exit 0), `uv run pytest tests/`, + `uv run pytest test-objective`, golden tests unchanged. +- [ ] Merge origin/main (PR #287 landed), rerun, push, PR body (Fixes ITL-895, Fixes ITL-901), + mark ready, CI green, CodeRabbit rounds (review bodies too), worker_done. diff --git a/superpowers/plans/2026-11-11-itl-895-float-to-int.md b/superpowers/plans/2026-11-11-itl-895-float-to-int.md new file mode 100644 index 000000000..8dffe14d7 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-895-float-to-int.md @@ -0,0 +1,204 @@ +# ITL-895 Function Output Conformance Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +## Status log (for a successor) + +- 2026-11-11: Step 1 (root cause) is done. See `superpowers/specs/itl-895-float-to-int/FINDINGS.md`; + commit `07037243`. +- 2026-11-11: Coordinator reply. The developer decides D1-D7, the option and the input-side + scope; the coordinator recommends (c). For now: + - Build the mechanism of (c) for **function outputs only**, with the conformance rule as one + table so that each of D1-D7 is a one-line switch. + - Every **undecided** D keeps today's behaviour. + - Decided cases: D4 (`None` for a non-optional output: reject). The coordinator also listed + datetime -> date among the lossy pairs to fail, so D6 is reject. + - Do not change `DictSource` / `ListSource` / `Datagram` input behaviour. + - Keep the PR a draft. + - Report the out-of-scope items in worker_done. +- 2026-11-11: Tasks 1-6 done. Draft PR #299 is open, and CI is green (`run-tests`, objective + tests, ruff, typecheck). `test-objective/unit/test_data_function.py::test_call_returns_none_propagates` + encoded the old `None` behaviour; its function now declares `-> int | None`, and a sibling + test checks the rejection. CodeRabbit skipped the draft. + **Next:** when the developer decides D1-D7, flip the entries in `OUTPUT_CONFORMANCE` + (`semantic_types/output_conformance.py`) and the matching tests in + `TestUndecidedCasesKeepTodaysBehaviour`. Extend the check to inputs if they are in scope. + Merge main after PR #287 / PR #289 land; PR #289's ndarray check in `_build_output_data` can + then raise `OutputTypeError`. Mark the PR ready, then run the CodeRabbit rounds. + +**Goal:** A function pod rejects a returned value that does not conform to its declared output +type, before anything is stored. The rejection names the pod, the output, the declared type +and the value's type. It is a row failure under `error_policy`. The output `Data` holds exactly +what is stored and hashed. + +**Architecture:** A new module, `semantic_types/output_conformance.py`, holds the rule. It has +one frozen `ConformanceRule` table with a field per D-decision, plus `find_nonconformance(value, +declared, rule)`, which walks orcapod's type grammar (primitives, numpy scalar types, `list`, +`tuple`, `dict`, `Optional`, dataclass fields, date/time) and returns the first mismatch. +Logical types and anything else stay with their own converters. + +`PythonDataFunction._build_output_data` runs the check, then converts the outputs to Arrow +**eagerly** through the data context's type converter, and builds the `Data` from that table. +So `as_dict()`, `as_table()`, the store and the hash all see the same values. Any failure, +from the check or from the conversion, raises the new `orcapod.errors.OutputTypeError`. +`OutputTypeError` subclasses both `TypeError` and `ValueError`, so existing handlers and tests +for the conversion's `ValueError`s still apply. It is raised inside the data function's `call`, +so every orchestrator already treats it as a function failure (row failure under +`error_policy`, `on_data_crash`). + +**Tech Stack:** Python 3.11-3.13, pyarrow, pytest, `uv run`. + +## Files + +- Create: `src/orcapod/semantic_types/output_conformance.py`. Rule table and checker. +- Modify: `src/orcapod/errors.py`. Add `OutputTypeError`. +- Modify: `src/orcapod/core/data_function.py`. In `_build_output_data` (line ~569), add the + check and the eager conversion. +- Create: `tests/test_core/function_pod/test_output_conformance.py`. Public-interface + regressions. +- Modify: `docs/concepts/function-pods.md`, `CHANGELOG.md` (new `[0.3.0] — unreleased` + section), `DESIGN_ISSUES.md` (one appended entry). + +## Task 1: Failing regression tests (public interface) + +**Files:** Create `tests/test_core/function_pod/test_output_conformance.py` + +- [ ] Write the tests (full code in the file). They cover: + - Standalone pod: each lossy pair raises `OutputTypeError` from `iter_data()`, and the + message names the pod, the output, the declared type and the value's type. The pairs are + int<-1.5, int<-np.float64(1.5), int<-Decimal('1.5'), int<-Fraction(3, 2), np.int8<-1.5, + `int | None`<-1.5, list[int]<-[1, 1.5], dict[str, int]<-{'a': 1.5}, tuple[int, int]<-(1, 1.5), + date<-1.5, datetime<-1.5, date<-datetime, str<-b'x', bytes<-'x', int<-None, float<-None, + list[int]<-None, and a dataclass field int<-1.5. + - The standalone `as_table()` over mixed results raises `OutputTypeError`, not the + "Complex unions" `ValueError`. + - `function_pod(result_database=...)`: raises, and nothing is stored. + - `PipelineJob` (sync and async orchestrators x in-memory and Delta stores): the run + completes, the failed row is absent, the conforming rows are stored, and an observer's + `on_data_crash` receives the `OutputTypeError`. With `error_policy="fail_fast"`, the sync + orchestrator raises `OutputTypeError`. + - Divergence: in the chain `up` (`-> float`, returns the int `1`, accepted under D1 until + decided) into `down(y: float)`, `down` receives the stored value (`1.0`, a `float`) + standalone and on both orchestrators. `as_dict()` equals `as_table()`. + - Undecided cases stay accepted with today's stored value: np.int64 for int, 1 for float, + True for float, (1, 2) for list[int], 0.1 for np.float32, np.bool_ for bool. + - Conforming outputs are unchanged: the content hash of a pod output equals that of + `Data({'y': 1}, python_schema={'y': int})`. +- [ ] Run `uv run pytest tests/test_core/function_pod/test_output_conformance.py -q`. Expected: + the rejection tests FAIL (no error raised, value truncated); the accepted-case tests pass. +- [ ] Commit `test(function_pod): failing regressions for output type conformance (ITL-895)`. + +## Task 2: `OutputTypeError` + +**Files:** Modify `src/orcapod/errors.py` + +```python +class OutputTypeError(TypeError, ValueError): + """A function returned a value that does not match its declared output type (ITL-895). + + Orcapod never converts a returned value implicitly: a ``float`` for an output declared + ``int`` would otherwise be truncated by pyarrow. Raised by the data function before + anything is stored or hashed, so a pipeline treats it like any other function failure + (a row failure under ``error_policy``). + + Subclasses both ``TypeError`` and ``ValueError``: a type mismatch is a ``TypeError``, + and conversion failures the error also wraps (an integer out of range, a naive + datetime under the strict timezone policy) were ``ValueError`` before. + + Attributes: + function_name: Name of the function pod's function. + output_key: Name of the output column. + declared_type: The declared output type. + value: The value the function returned for that output. + """ +``` + +The constructor takes `function_name`, `output_key`, `declared_type`, `value` and `detail`, and +builds the message: `Function 'f' returned float (1.5) for output 'y', which is declared int: +. Orcapod does not convert types implicitly; convert the value in the function.` + +## Task 3: Conformance rule + +**Files:** Create `src/orcapod/semantic_types/output_conformance.py` + +- `ConformanceRule` is a frozen dataclass with these fields: `int_for_float` (D1), + `numpy_scalar_same_kind` (D2), `bool_for_float` (D3), `none_for_non_optional` (D4), + `sequence_for_list` (D5), `datetime_for_date` (D6) and `float_for_float32` (D7). +- `OUTPUT_CONFORMANCE` is the one table, one line per decision. D1, D2, D3, D5 and D7 are True + (today's behaviour, undecided). D4 and D6 are False (decided). +- `find_nonconformance(value, declared, rule) -> Nonconformance | None`, where + `Nonconformance` has `path`, `expected` and `value`. + - `Optional` / `X | None`: `None` passes; otherwise check `X`. + - `None` for a non-optional type: rejected unless D4. + - `bool`: exact `bool`, or a numpy bool (D2). + - `int`: an `int` that is not a `bool` (subclasses such as `IntEnum` keep their value), or a + numpy integer (D2). Any float, Decimal, Fraction or numpy float is rejected. + - numpy integer dtypes (`np.int8`, ...): Python `int`, the declared numpy type, or another + numpy integer (D2). The range is checked by the conversion. + - `float` / `np.float64`: a `float` (a numpy float64 is a `float` subclass); a numpy float of + another width (D2); an `int` (D1); a numpy integer (D1 and D2); a `bool` (D3). + - `np.float32`: an `np.float32`; a Python `float` or another numpy float (D7); an `int` + (D1); a `bool` (D3). + - `str`: an `str`. `bytes`: a `bytes`. + - `date`: a `date` that is not a `datetime` (D6). `datetime`: a `datetime`. + `time` / `timedelta`: by `isinstance`. + - `list[T]`: a `list`, or a `tuple` (D5), each element checked against `T`. Any other + non-str iterable is passed through unchecked under D5 (today) and rejected otherwise. + - `tuple[T1, ..., Tn]`: a `tuple`, or a `list` (D5), of length n, each element checked. + - `dict[K, V]`: a `Mapping`, each key and value checked. + - A dataclass declared type, when the value is an instance: each field checked against + `get_type_hints`. + - Anything else (logical types, pydantic, `Any`, `Literal`, complex unions): `None`. Those + are left to their own converter. + +## Task 4: Check and eager conversion in `_build_output_data` + +**Files:** Modify `src/orcapod/core/data_function.py` + +```python +output_data = parse_function_outputs(self._output_keys, values) +schema = self.output_data_schema +for key, value in output_data.items(): + problem = find_nonconformance(value, schema[key]) + if problem is not None: + raise OutputTypeError(self.canonical_function_name, key, schema[key], value, problem.describe()) +... +if not output_data: + return Data(output_data, ...) # unchanged: no columns to convert +try: + table = self.data_context.type_converter.python_dicts_to_arrow_table([output_data], python_schema=schema) +except Exception as exc: + key = + raise OutputTypeError(..., detail=f"{type(exc).__name__}: {exc}") from exc +return Data(table, source_info=source_info, record_uuid=new_record_uuid, data_context=self.data_context) +``` + +- [ ] Run the Task 1 tests. Expected: all pass. +- [ ] Run `uv run pytest tests/test_core tests/test_pipeline -q -x`, then the golden suite + `uv run pytest tests/test_hashing/golden -q`. Expected: green, golden unchanged (644). +- [ ] Commit `fix(data_function): reject function outputs that do not match the declared type (ITL-895)`. + +## Task 5: Docs, CHANGELOG, DESIGN_ISSUES + +- [ ] `docs/concepts/function-pods.md`: a new section, "Return values must match the declared + type". It gives a runnable sample, `# ...` output comments, and the list of what is accepted + while D1-D7 are open. +- [ ] `CHANGELOG.md`: a `## [0.3.0] — unreleased` section with `### Fixed` above `[0.2.0]`. +- [ ] `DESIGN_ISSUES.md`: one appended entry, `FO1`, status `in progress` (it becomes + `resolved` when D1-D7 and the input-side scope are decided). +- [ ] `uv run pytest tests/test_docs -q`. Commit `docs: ...`. + +## Task 6: Verification and PR + +- [ ] ruff check, ruff format --check and basedpyright (exit 0). Full `uv run pytest tests/` + on 3.11, 3.12 and 3.13. +- [ ] Merge origin/main, rerun, push, and open a **draft** PR ("Fixes ITL-895"). Then run + `orca linear attach` and `orca worktree set --pr`. + +## Self-review + +- Spec coverage: lossy pairs (Task 1); None (D4); divergence; row failure on sync and async, + in-memory and Delta; pod-level error naming; undecided defaults; no hash change (golden); + inputs untouched (no converter change). Covered. +- Known gap, by design until the developer decides: logical types (Path, UUID, ndarray, + pydantic) keep their own conversion rules. PR #289 adds the ndarray check. diff --git a/superpowers/plans/2026-11-12-itl-895-refresh.md b/superpowers/plans/2026-11-12-itl-895-refresh.md new file mode 100644 index 000000000..b1e635d45 --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-895-refresh.md @@ -0,0 +1,98 @@ +# ITL-895 / ITL-901 PR 299 Refresh Plan + +> **For agentic workers:** Use sensei:executing-plans to execute this plan inline, with coordinator approval before the real merge and any policy changes. + +**Goal:** Refresh PR 299 onto main with one function-output conformance boundary and consistent errors across standalone, stored, sync and async paths. + +**Architecture:** Investigate main and the PR before selecting an implementation. Keep user-class compatibility owned by its logical type and apply function context at the function-output boundary. Preserve all current hashes and stored schemas. + +**Tech Stack:** Python 3.11–3.13, uv, pytest, PyArrow, ruff, basedpyright. + +## Investigation and decision gate +- [x] Abort interrupted scratch merge, check out real PR branch, delete old scratch branch, pull without rebasing. +- [x] Read main CLAUDE.md and PR body; fetch all review bodies and GraphQL review threads. +- [ ] Read all review history, including body-only findings and previous replies. +- [ ] Preserve PR tests in scratch evidence; run rejection cases on main through public APIs and record actual failures/coercions. +- [ ] Dry-run merge origin/main on a scratch branch, inventory every conflict and propose its resolution. +- [ ] Ask coordinator to approve resolution, strictness reconciliation, and outstanding review fixes before real merge. + +## Approved integration (only after decision) +- [ ] Extend this plan with exact implementation and regression cases from the approved proposal. +- [ ] Write public regression tests and show failures before fixing. +- [ ] Merge origin/main with a merge commit; consolidate conformance and error wording at the common output boundary. +- [ ] Migrate CHANGELOG additions verbatim to changelog.d/ITL-895.md and design entries to individual design_issues files; keep main CHANGELOG. +- [ ] Update documentation and PR body to actual behavior. +- [ ] Run uv run python scripts/assemble_changelog.py --check and uv run python scripts/design_issues.py --check. +- [ ] Run uv run pytest tests/test_hashing/golden unchanged; compare golden files with origin/main (no PR-introduced movement expected). +- [ ] Run full uv run --python 3.12 pytest tests/, then 3.11, then 3.13; record skip/failure names. +- [ ] Run uv run --with ruff ruff check ., uv run --with ruff ruff format --check ., uv run --with basedpyright basedpyright (exit 0). +- [ ] Fetch main again; merge and rerun if it moved. Commit all changes and push once per approved round. +- [ ] Verify CI and CodeRabbit review on exact head; read all threads and body-only findings, route nontrivial revision plans through ask. +- [ ] Report exact reviewed SHA, hand-resolved files, golden evidence, and remaining gaps with worker_done exactly once. + +## Initial findings +- Head 5537c0a6 includes conversions-with-warnings approved on November 11, while the current task explicitly prohibits implicit conversion/widening. Route this discrepancy through dry-run ask. +- Main has both the ITL-883 ndarray guard and ITL-972 declared-class conversion. Reconcile both with PR conformance. +- Review thread PRRT_kwDOOf7mTs62WdUw has no reply: converter exception messages and traceback expose returned datetime values. Earlier mapping-key privacy thread has a reply but its promised fix only shortened keys; evaluate in proposed review plan. + +## Dry-run evidence (2026-11-12) +Main tested: b46549fd6e13059be795141684436a2e27e2771b. PR head: 5537c0a6b043b4c307b675c035648e08eeda0016. +- Original PR test file on main: 96 failed, 22 passed, 1 skipped; many failures are missing OutputTypeError/ValueConversionWarning, so not sufficient behavioral evidence alone. +- Reused original LOSSY matrix in a probe with built-in TypeError: all 30 values pass iter_data unchecked. On as_table, 27 pass, number-for-datetime raises AttributeError, numpy.bool-for-float and mapping-for-list raise ArrowInvalid. None has the unified contract. +- Added main-guard probes: subclass-for-base is accepted by iter_data and rejected by as_table with TypeError; scalar-for-ndarray is rejected on both. Total 61 failed, 3 passed (expected red suite). +- Evidence files: /tmp/itl895-main-original.log, /tmp/itl895-main-behavior.log, /tmp/test_itl895_main_behavior.py, /tmp/itl895-original-tests.py. +- Scratch merge conflicts only in DESIGN_ISSUES.md (modify/delete) and src/orcapod/core/data_function.py (conformance loop vs ndarray guard). Scratch merge aborted; real branch restored without changing tracked code. + +## Proposed decision for coordinator +1. Keep PR eager output construction as common boundary; absorb ndarray detection into conformance and raise OutputTypeError there, retaining main data-context changes. +2. Reuse logical-type class acceptance rather than duplicate exact/structural rules. Reject wrong classes before output emission, with caller-owned function/output context; storage guards remain as general converter invariants, not a second pod rule. +3. Interpret current strictness instruction as superseding old warning conversions (tuple/iterable to list, datetime to date, wider floats, same-kind NumPy-to-Python conversion); update tests/docs accordingly, pending explicit coordinator decision. +4. Migrate own FO entries into per-entry files and changelog into one fragment, keeping main legacy-file deletion and changelog unchanged. +5. Accept still-valid review privacy fixes: structural dict paths, no converter message/chained value in output error. Add regression tests first. Reply to old unanswered thread and body-only mapping-key finding after approval. +6. No golden changes or schema/hash changes authorized. + +## Coordinator approval and exact implementation steps +Coordinator approved merge/consolidation and privacy fixes; conversion policy remains pending. Preserve all existing conversion decisions until the coordinator replies. +- [ ] Resolve data_function conflict initially retaining both existing checks to establish a runnable red baseline. +- [ ] Add tests in test_output_conformance.py for ndarray scalar error identity, exact-class subclass rejection, structurally equivalent classes with invalid nested fields, and privacy of mapping/converter errors. Compare standalone vs sync/async stored errors. +- [ ] Move `_declared_ndarray_type` from data_function.py to output_conformance.py; have `_Checker.check` return `_mismatch` for wrong ndarray values and remove `_check_declared_ndarray_outputs` and its call. +- [ ] Pass the active converter to `check_conformance`; `_Checker.check` asks `converter.get_logical_type_for_python_type(declared)` for class handling, delegates `UserClassLogicalType.accepts_instance`, and recursively validates its declared field annotations. The converter is used for compatibility, not diagnostics. +- [ ] Keep `OutputTypeError` construction in PythonDataFunction; remove raw converter text and suppress chained exception (`type(exc).__name__`, `from None`). +- [ ] Replace mapping key paths with structural `.keys()` / `[*]`, remove `_key_text`, and change the old key-shortening assertion to require no key content. +- [ ] Run targeted tests before and after changes; migrate FO1/FO2 and changelog text verbatim, then update prose only where integration changes facts. + +## Developer policy decision and implementation +Coordinator delivered developer decision at 2026-11-12 14:12 UTC: preserve exactly represented same-kind numeric conversions silently; reject container changes, datetime-to-date, and numeric narrowing even when a particular narrowing value happens to fit. Bool/int remain distinct kinds. Update both CLAUDE.md and .zed/rules. +- Consolidation red: 11 failed, 4 passed; green before policy update: 133 passed, 1 skipped. +- Policy red: 36 failed, 20 passed, 4 platform skips; policy implementation: 233 passed, 4 skips, 3 old diagnostic assertions updated (container type now named before inspecting elements, static float hint). +- Removed output warning machinery; retained one ACCEPT/REJECT decision table. +- Existing PR tests that relied on removed behavior: TestConvertedWithWarning, former tuple/date/narrowing DOWNSTREAM cases. Replaced with full standalone/table/async/sync/memory/Delta rejection matrix and numeric acceptance/downstream matrix. +- Existing docs sample `twice` in function-pods.md returned a tuple for list; now explicitly returns a list. Historical probe 09_warning_dedup.py must be marked historical, not executed as current-policy evidence. + +## Structural preservation decision +Coordinator approved preserving compatible user-class objects unchanged after eager storage validation; stored reads retain existing registry binding. Keep test_values_are_passed_as_they_are unchanged. Update the earlier Child-for-Base iteration test to reject. Implementation: validate outputs through the declared Arrow schema, then construct the output Data from validated Python values with only numeric exceptions normalized; never reconstruct or mutate user-class objects. Lists/dicts recurse into numeric values while retaining user-class leaves. This preserves the no-conversion contract from #325. + +## HOLD received at 14:22 UTC +Coordinator reports developer disagrees with previous subclass/container decision. Freeze those points pending clarification; do not push. Already implemented under earlier explicit approvals: shared-boundary subclass rejection, iterable rejection, structural object preservation. Sent storage analysis: Base representation stores Base fields only and hash excludes subclass extras; preserving extras requires a different schema/representation. Lossy rejection and numeric acceptance remain authorized. + +## Final developer wording (14:29 UTC; hold lifted) +Accept instances/subclasses of the declared type; declared logical type owns storage. Accept exact same-kind numeric equivalents and structural identity equivalence. Reject unrelated classes/containers and numeric narrowing. Consequently bool-for-int and datetime-for-date are accepted; user-class subclass fields outside the declared base are omitted from storage and the stored row hash. Never rebuild compatible user objects in memory. Updated the logical type acceptance rule for exact/structural subclass instances, public tests, docs and paired agent instructions. +- Final-policy subclass regressions: 16 failures before implementing acceptance. +- Stopped obsolete full-suite process before final policy implementation; final suite must be rerun. +- Developer withdrew custom UV_CACHE_DIR/--no-sync guidance: use plain uv run and request cache/network permission as needed; keep all artifacts inside .untracked. + +## Final verification checkpoint (14:45 UTC) +- The final policy is the 14:29 decision above; earlier proposals and checklists are retained as history, not active policy. Investigation, full review-history reading, dry-run approval, merge, fragment migration, conformance consolidation, privacy fixes, and final docs are implemented. +- Public output suite: 342 passed, 8 platform skips. Additional coordinator-requested probes verify list/optional/nested base savers, pydantic subclasses, rejection of incompatible overridden fields, and value-class inference in DictSource. Nested Arrow fields are physical structs under the enclosing extension, so tests inspect storage_type once at the outer boundary. +- ruff check and format check pass; changelog (19 fragments) and design checks (118 entries) pass; basedpyright exits 0 (11089 warnings). +- origin/main still b46549fd; full Python 3.12 suite running in .untracked/full-312-approved.log, then 3.11 and 3.13. No golden regeneration or JSON movement. +- Remaining: full suites, final commit and single push, PR body replacement, all review replies/body findings, CI and CodeRabbit verification on final head. + +## Current-main integration checkpoint +- Full Python 3.12 and 3.11 on b46549fd each passed: 9708 passed, 109 skipped, 19 expected failures. Affected objective data-function tests: 22 passed. +- Main advanced to c8e9eb21 (#302, async operator readback); merged cleanly without hand resolution. Stopped the superseded 3.13 run before modifying files. +- Full Python 3.12 on c8e9eb21 integration: 9784 passed, 109 skipped, 23 expected failures. Static checks all pass (604 formatted files, 20 changelog fragments, 119 design entries, basedpyright zero errors). +- Coordinator's follow-up specifies affected tests plus goldens for this main advance. Running those on 3.11, then the full 3.13 suite; the original full 3.11 evidence remains recorded above. +- Exact skip and expected-failure names are retained in `.untracked/full-312-current.log` and `.untracked/full-311-approved.log`. Skips cover PostgreSQL opt-in (77), SpiralDB opt-in (9), unavailable MinIO (8), unavailable Graphviz (3), platform longdouble width (8), and deltalake path support (4). + +- Final local verification: Python 3.11 affected suites/goldens 2105 passed, 8 skipped, 7 expected failures; full Python 3.13 9784 passed, 109 skipped, 23 expected failures. No unexpected failures or golden changes. All local gates complete; pushing once, then CI/CodeRabbit and reply audit. diff --git a/superpowers/plans/2026-11-12-itl-895-review-round-c7f53bc0.md b/superpowers/plans/2026-11-12-itl-895-review-round-c7f53bc0.md new file mode 100644 index 000000000..95ad0818e --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-895-review-round-c7f53bc0.md @@ -0,0 +1,37 @@ +# ITL-895 / PR #299: review round on c7f53bc0 (successor worker) + +State on takeover: head c7f53bc0 pushed, CI green, CodeRabbit review 5876929622 on c7f53bc0 +with six inline findings and no body-only findings. Scratch and repro scripts: `.untracked/r2/`. + +## Findings (all reproduced through pods / PipelineJob before any change) + +| # | Thread (comment id) | Verdict | Evidence | Action | +|---|---|---|---|---| +| 1 | changelog.d/ITL-895.md:19 (4552331351) | accept | "downstream values agree with stored values" contradicts the subclass sentence before it | reword: numeric normalization gives agreement; subclass / datetime-for-date keep more in memory than storage | +| 2 | design_issues/ITL-895-FO2.md:4 (4552331374) | accept | `**Status:** open (deferred)`; checker passes it by prefix, convention lists `open` | `open`, deferral stays in body | +| 3 | user_class_logical_type.py:213 (4552331387) | real, design decision | `.untracked/r2/repro3b.py`: run 1 (async) x=20 stores z=20; run 2 x=20,99: r99 served z=20 with zero calls. First-run sync gives z=-1, async z=20 | no code change without the developer; document + design issue entry, pending coordinator answer | +| 4 | output_conformance.py:204 (4552331409) | accept | `MappingProxyType`, `UserDict` accepted for `dict[str,int]` and rebuilt as dict | `isinstance(value, dict)` in check and normalize | +| 5 | output_conformance.py:213 (4552331439) | accept | `Literal[1]` returning 1.5 stores 1; `Literal["yes"]` returning "no" stored; `Literal["yes", None]` returning None rejected | Literal branch: member by type and value; None allowed when None is a member | +| 6 | FINDINGS.md:227 (4552331469) | accept | text only | add the reproducibility caveat | + +Extra observation (not from CodeRabbit): `normalize_output_numbers` rebuilds list and dict +subclass instances as plain list / dict in memory (`.untracked/r2/repro7.py`). + +## Steps after coordinator approval + +1. Failing tests in `tests/test_core/function_pod/test_output_conformance.py` (public pods): + non-dict mappings rejected, dict subclass accepted; Literal membership (standalone and stored). +2. Fix `output_conformance.py`; doc edits (1, 2, 6, and whatever is approved for 3). +3. Merge origin/main if moved; full suite 3.12, 3.11, 3.13; ruff, format, basedpyright; golden + check (no movement expected: conforming values are untouched). +4. One commit, one push, signed replies on all six threads; wait for CI and CodeRabbit on the new head. + +## Outcome notes + +- F3 decided by the developer (Nov 12): keep the rule, document, follow-up ITL-978. F7 is a known gap in ITL-978's scope. +- Pushed 0bc9cc1b; CI failed on Python 3.13 only: the async baseline test asserted `{r20: 20, r99: 99}` + but got `{r20: 20, r99: 20}`. Within one async run the second row can be served the first + row's stored result (same row hash), depending on timing. The test now pins only the set of + possible values. Second push for this test fix. +- CodeRabbit on 0bc9cc1b: one finding, PYI061 on `Literal["yes", None]` test fixtures; PYI is not + in the project's ruff selection (`ruff check .` passes), answered without a code change. diff --git a/superpowers/specs/itl-895-float-to-int/01_pyarrow_minimal.py b/superpowers/specs/itl-895-float-to-int/01_pyarrow_minimal.py new file mode 100644 index 000000000..b57638cb1 --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/01_pyarrow_minimal.py @@ -0,0 +1,163 @@ +"""Claim 1: pyarrow's Python-to-Arrow conversion itself truncates float -> integer. + +Pure pyarrow (plus numpy / stdlib values); no orcapod import. Shows: + +1. The minimal reproduction of the exact orcapod call: + ``pa.Table.from_pylist([{"y": 1.5}], schema=pa.schema([("y", pa.int64())]))``. +2. A matrix of Python value x declared Arrow type through the same call, so the + asymmetry (float -> int truncates, the other mismatches raise) is visible. +3. The same call through ``pa.array(values, type=...)``, and the explicit safe cast + (``pc.cast``), which does refuse 1.5 -> int64. + +Why (from pyarrow's own C++ source, shipped in the wheel under +``pyarrow/src/arrow/python``): + +* ``python_to_arrow.cc`` ``PyValue::Convert(IntegerType)`` calls ``CIntFromPython``. +* ``helpers.cc`` ``CIntFromPython`` rejects ``bool`` explicitly, then + ``CIntFromPythonImpl``: if the object is not a ``PyLong`` it calls ``PyObjectToPyInt``, + which tries ``__index__`` and, when that fails, FALLS BACK TO ``__int__`` + (``nb_int``). ``float.__int__`` truncates toward zero, so 1.5 -> 1, -1.5 -> -1. + Range is checked afterwards (overflow raises). Anything with ``__int__`` is accepted + the same way: ``Decimal``, ``Fraction``, ``numpy.float64``. +* ``Convert(DoubleType)`` accepts float, numpy float, and ints only through + ``IntegerScalarToDoubleSafe`` (|v| <= 2**53, else raises). +* ``Convert(FloatType)`` (float32) does ``static_cast(double)``: rounds silently. +* ``Convert(BooleanType)`` accepts only ``True``/``False``/``numpy.bool_``. +* String/binary converters reject non-str/bytes. +* Timestamp/date/time/duration converters fall through to ``CIntFromPython`` for + non-datetime objects, so a float there truncates too (1.5 -> 1 tick). + +No conversion option changes this: ``pa.array``/``from_pylist`` take no ``safe=`` for +Python sequences (``safe`` only applies to pandas/numpy inputs), and ``from_pandas`` +does not change the integer converter. + +Run: uv run python superpowers/specs/itl-895-float-to-int/01_pyarrow_minimal.py + uv run --no-project --with pyarrow==20.0.0 --with numpy \ + python superpowers/specs/itl-895-float-to-int/01_pyarrow_minimal.py +""" + +from __future__ import annotations + +import datetime as dt +from decimal import Decimal +from fractions import Fraction + +import numpy as np +import pyarrow as pa +import pyarrow.compute as pc +import warnings + +warnings.simplefilter("ignore", DeprecationWarning) + +print(f"pyarrow {pa.__version__}, numpy {np.__version__}\n") + +print("1. minimal reproduction of the orcapod call") +t = pa.Table.from_pylist([{"y": 1.5}], schema=pa.schema([("y", pa.int64())])) +print( + f" pa.Table.from_pylist([{{'y': 1.5}}], schema=int64) -> {t['y'].to_pylist()}\n" +) + + +def through(value, arrow_type) -> str: + try: + tbl = pa.Table.from_pylist( + [{"v": value}], schema=pa.schema([("v", arrow_type)]) + ) + out = tbl["v"].to_pylist()[0] + if (pa.types.is_temporal(arrow_type)) and not isinstance( + value, (dt.date, dt.timedelta) + ): + tag = "REINTERPRETED" # a number taken as ticks/days since the epoch + elif out == value and type(out) is type(value): + tag = "kept" + elif out == value: + tag = "kept, type changed" + else: + tag = "LOSSY" + return f"{tag:18} -> {out!r}" + except Exception as e: # noqa: BLE001 - we want every error class + return f"{'raises':18} {type(e).__name__}: {str(e)[:70]}" + + +CASES = [ + # (label, value, arrow type) + ("float 1.5 -> int64", 1.5, pa.int64()), + ("float 2.9 -> int64", 2.9, pa.int64()), + ("float -1.5 -> int64", -1.5, pa.int64()), + ("float 1.5 -> int8", 1.5, pa.int8()), + ("float 1.5 -> uint32", 1.5, pa.uint32()), + ("float 2.0 -> int64", 2.0, pa.int64()), + ("float nan -> int64", float("nan"), pa.int64()), + ("float inf -> int64", float("inf"), pa.int64()), + ("float 1e30 -> int64", 1e30, pa.int64()), + ("np.float64(1.5) -> int64", np.float64(1.5), pa.int64()), + ("np.float32(1.5) -> int64", np.float32(1.5), pa.int64()), + ("Decimal('1.5') -> int64", Decimal("1.5"), pa.int64()), + ("Fraction(3, 2) -> int64", Fraction(3, 2), pa.int64()), + ("np.int64(3) -> int64", np.int64(3), pa.int64()), + ("np.int64(3) -> int8", np.int64(3), pa.int8()), + ("True -> int64", True, pa.int64()), + ("np.bool_(True) -> int64", np.bool_(True), pa.int64()), + ("'1' -> int64", "1", pa.int64()), + ("300 -> int8", 300, pa.int8()), + ("-1 -> uint8", -1, pa.uint8()), + ("1 -> bool", 1, pa.bool_()), + ("1.0 -> bool", 1.0, pa.bool_()), + ("np.bool_(True) -> bool", np.bool_(True), pa.bool_()), + ("1 -> float64", 1, pa.float64()), + ("2**53+1 -> float64", 2**53 + 1, pa.float64()), + ("True -> float64", True, pa.float64()), + ("Decimal('1.5') -> float64", Decimal("1.5"), pa.float64()), + ("np.int64(3) -> float64", np.int64(3), pa.float64()), + ("np.float32(0.1) -> float64", np.float32(0.1), pa.float64()), + ("0.1 -> float32", 0.1, pa.float32()), + ("2**24+1 -> float32", 2**24 + 1, pa.float32()), + ("1 -> large_string", 1, pa.large_string()), + ("b'x' -> large_string", b"x", pa.large_string()), + ("'x' -> large_binary", "x", pa.large_binary()), + ("1.5 -> timestamp[us]", 1.5, pa.timestamp("us")), + ("1 -> timestamp[us]", 1, pa.timestamp("us")), + ("1.5 -> date32", 1.5, pa.date32()), + ("1.5 -> duration[us]", 1.5, pa.duration("us")), + ("date(2020,1,1) -> timestamp[us]", dt.date(2020, 1, 1), pa.timestamp("us")), + ( + "datetime naive -> timestamp[us,UTC]", + dt.datetime(2020, 1, 1, 12), + pa.timestamp("us", "UTC"), + ), + ( + "datetime aware -> timestamp[us]", + dt.datetime(2020, 1, 1, 12, tzinfo=dt.timezone(dt.timedelta(hours=2))), + pa.timestamp("us"), + ), + ("datetime -> date32", dt.datetime(2020, 1, 1, 12), pa.date32()), + ("1.5 -> decimal128(5,1)", 1.5, pa.decimal128(5, 1)), + ("Decimal('1.25') -> decimal128(5,1)", Decimal("1.25"), pa.decimal128(5, 1)), + ("1 -> decimal128(5,1)", 1, pa.decimal128(5, 1)), + ("[1.5] -> large_list", [1.5], pa.large_list(pa.int64())), + ( + "{'a': 1.5} -> struct", + {"a": 1.5}, + pa.struct([("a", pa.int64())]), + ), +] + +print("2. Table.from_pylist(rows, schema=...) for each (value, declared Arrow type)") +for label, value, typ in CASES: + print(f" {label:40} {through(value, typ)}") + +print("\n3. other entry points") +print( + f" pa.array([1.5], type=int64) -> {pa.array([1.5], type=pa.int64()).to_pylist()}" +) +print( + f" pa.scalar(1.5, type=int64) -> {pa.scalar(1.5, type=pa.int64()).as_py()}" +) +try: + pc.cast(pa.array([1.5]), pa.int64()) +except pa.ArrowInvalid as e: + print(f" pc.cast(pa.array([1.5]), int64) [safe] -> raises ArrowInvalid: {e}") +print( + f" pc.cast(..., safe=False) -> {pc.cast(pa.array([1.5]), pa.int64(), safe=False).to_pylist()}" +) +print(f" int(1.5) (what nb_int does) -> {int(1.5)}") diff --git a/superpowers/specs/itl-895-float-to-int/02_call_chain.py b/superpowers/specs/itl-895-float-to-int/02_call_chain.py new file mode 100644 index 000000000..1ccea1602 --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/02_call_chain.py @@ -0,0 +1,138 @@ +"""Claim 2: the orcapod call chain from the function's return value to the stored value. + +For every execution path, run a pod declared ``-> int`` that returns ``1.5`` with the +tracer installed, and print (a) each pyarrow call that received a non-int value for an +integer field, with the orcapod frames that led to it, and (b) what ends up stored / +yielded. + +Run: uv run python superpowers/specs/itl-895-float-to-int/02_call_chain.py +""" + +from __future__ import annotations + +import importlib + +import asyncio +import sys +import tempfile +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) + +_tracer = importlib.import_module("_tracer") # the sibling helper, via sys.path + +from orcapod import PipelineJob, function_pod # noqa: E402 +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase # noqa: E402 +from orcapod.pipeline import AsyncPipelineOrchestrator # noqa: E402 +from orcapod.sources import DictSource # noqa: E402 + +_tracer.install() + + +def make_src() -> DictSource: + return DictSource(data=[{"k": "a", "x": 1}], tag_columns=["k"], source_id="src") + + +def banner(title: str) -> None: + print(f"\n=== {title} ===") + _tracer.HITS.clear() + + +def show_iter(stream) -> None: + for _tag, data in stream.iter_data(): + print( + f" iter_data: as_dict()={data.as_dict()} as_table()={data.as_table().to_pylist()}" + f" arrow type={data.arrow_schema().field('y').type}" + ) + + +# --- A. standalone FunctionPod stream, sync function -------------------------------- +@function_pod(output_keys="y") +def f_sync(x: int) -> int: + return x + 0.5 # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +banner("A1 standalone FunctionPod, iter_data()") +show_iter(f_sync.pod(make_src())) + +banner("A2 standalone FunctionPod, as_table()") +t = f_sync.pod(make_src()).as_table() +print(f" as_table: y={t['y'].to_pylist()} type={t.schema.field('y').type}") + + +# --- B. standalone FunctionPod stream, async function ------------------------------- +@function_pod(output_keys="y") +async def f_async(x: int) -> int: + return x + 0.5 # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +banner("B1 standalone FunctionPod with an async function, iter_data()") +show_iter(f_async.pod(make_src())) + +banner("B2 standalone FunctionPod with an async function, as_table()") +t = f_async.pod(make_src()).as_table() +print(f" as_table: y={t['y'].to_pylist()} type={t.schema.field('y').type}") + +# --- C. cached function pod: result_database (CachedDataFunction) ------------------- +db_c = InMemoryArrowDatabase() + + +@function_pod(output_keys="y", result_database=db_c) +def f_cached(x: int) -> int: + return x + 0.5 # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +banner("C1 function_pod(result_database=...) -> CachedDataFunction, as_table()") +t = f_cached.pod(make_src()).as_table() +print(f" as_table: y={t['y'].to_pylist()} type={t.schema.field('y').type}") +banner("C2 CachedDataFunction, second run (served from the store)") +show_iter(f_cached.pod(make_src())) + +# --- D. cached function pod: pod_cache_database (CachedFunctionPod) ----------------- +db_d = InMemoryArrowDatabase() + + +@function_pod(output_keys="y", pod_cache_database=db_d) +def f_podcached(x: int) -> int: + return x + 0.5 # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +banner("D1 function_pod(pod_cache_database=...) -> CachedFunctionPod, as_table()") +t = f_podcached.pod(make_src()).as_table() +print(f" as_table: y={t['y'].to_pylist()} type={t.schema.field('y').type}") + + +# --- E. FunctionNode in a PipelineJob ----------------------------------------------- +def run_job(store, orchestrator=None, fn=f_sync) -> None: + job = PipelineJob(name="itl895", store=store) + with job: + fn.pod(make_src(), label="f") + job.run(orchestrator=orchestrator) + node = job.nodes["f"] + t = node.as_table() + print(f" node.as_table: y={t['y'].to_pylist()} type={t.schema.field('y').type}") + recs = node.get_all_records() + if recs is not None: + print( + f" stored record y={recs['y'].to_pylist()} type={recs.schema.field('y').type}" + ) + + +banner("E1 PipelineJob sync, InMemoryArrowDatabase") +run_job(InMemoryArrowDatabase()) + +with tempfile.TemporaryDirectory() as d: + banner("E2 PipelineJob sync, DeltaTableDatabase") + run_job(DeltaTableDatabase(Path(d) / "sync")) + +banner("E3 PipelineJob async orchestrator, InMemoryArrowDatabase") +run_job(InMemoryArrowDatabase(), AsyncPipelineOrchestrator()) + +with tempfile.TemporaryDirectory() as d: + banner("E4 PipelineJob async orchestrator, DeltaTableDatabase") + run_job(DeltaTableDatabase(Path(d) / "async"), AsyncPipelineOrchestrator()) + +banner("E5 PipelineJob async orchestrator, async function, InMemoryArrowDatabase") +run_job(InMemoryArrowDatabase(), AsyncPipelineOrchestrator(), fn=f_async) + +_ = asyncio # imported for readers running parts interactively diff --git a/superpowers/specs/itl-895-float-to-int/03_split_datagram.py b/superpowers/specs/itl-895-float-to-int/03_split_datagram.py new file mode 100644 index 000000000..28b5b44ad --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/03_split_datagram.py @@ -0,0 +1,69 @@ +"""Claim 3: the output Data is internally inconsistent, and downstream pods see 1.5. + +``PythonDataFunction._build_output_data`` builds ``Data({"y": 1.5}, python_schema={"y": int})`` +without checking the value. The Data keeps the raw dict; its Arrow table (used for +``content_hash()`` and for storage) is built lazily and truncates to 1. So: + +* ``data.as_dict()`` says 1.5, ``data.as_table()`` / the stored record / the hash say 1. +* A downstream pod consumes ``as_dict()``. Chained standalone it receives 1.5; in a + PipelineJob it receives 1 on the sync orchestrator and 1.5 on the async one. The same + graph gives z=15 or z=10 depending on how it runs. In the standalone chain the upstream + Data is never converted to Arrow before the downstream function runs. + +Run: uv run python superpowers/specs/itl-895-float-to-int/03_split_datagram.py +""" + +from __future__ import annotations + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator +from orcapod.sources import DictSource + +seen: list[object] = [] + + +@function_pod(output_keys="y") +def up(x: int) -> int: + return x + 0.5 # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +@function_pod(output_keys="z") +def down(y: int) -> int: + seen.append(y) + return y * 10 + + +def src() -> DictSource: + return DictSource(data=[{"k": "a", "x": 1}], tag_columns=["k"], source_id="src") + + +print("=== one output Data, two answers ===") +for _tag, data in up.pod(src()).iter_data(): + print(f" as_dict() -> {data.as_dict()}") + print(f" as_table() -> {data.as_table().to_pylist()}") + print(f" python_schema-> {dict(data.schema())}") + print(f" content_hash -> {data.content_hash().to_hex()[:16]}...") + from orcapod.core.datagrams import Data + + same_as_int = Data({"y": 1}, python_schema={"y": int}).content_hash().to_hex()[:16] + print( + f" content_hash of Data({{'y': 1}}) -> {same_as_int}... (identical: hash is of the truncated value)" + ) + +print("\n=== standalone chain: what does the downstream function receive? ===") +seen.clear() +t = down.pod(up.pod(src())).as_table() +print(f" down received y={seen!r}; z={t['z'].to_pylist()}") + +for label, orch in [("sync", None), ("async", AsyncPipelineOrchestrator())]: + seen.clear() + job = PipelineJob(name=f"itl895_chain_{label}", store=InMemoryArrowDatabase()) + with job: + down.pod(up.pod(src(), label="up"), label="down") + job.run(orchestrator=orch) + print(f"\n=== PipelineJob ({label}) ===") + print( + f" down received y={seen!r}; z={job.nodes['down'].as_table()['z'].to_pylist()}; " + f"up stored y={job.nodes['up'].as_table()['y'].to_pylist()}" + ) diff --git a/superpowers/specs/itl-895-float-to-int/04_input_sites.py b/superpowers/specs/itl-895-float-to-int/04_input_sites.py new file mode 100644 index 000000000..4d318e2aa --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/04_input_sites.py @@ -0,0 +1,186 @@ +"""Claim 4: the same truncation is reachable for INPUT data, not only function outputs. + +Every place orcapod turns Python values into Arrow against a declared type goes through +``UniversalTypeConverter.python_dicts_to_arrow_table`` (``pa.Table.from_pylist`` with a +schema, ``universal_converter.py:1106``), whose primitive converter is a passthrough +(``universal_converter.py:1586-1588``). Paths that start from Arrow (Arrow/Delta/CSV/ +DataFrame sources, operators) use ``Table.cast``, which is a SAFE cast and raises. + +Each probe below prints the stored value and type, or the error, and the tracer's site. + +Run: uv run python superpowers/specs/itl-895-float-to-int/04_input_sites.py +""" + +from __future__ import annotations + +import importlib + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) + +_tracer = importlib.import_module("_tracer") # the sibling helper, via sys.path + +import polars as pl # noqa: E402 +import pyarrow as pa # noqa: E402 + +from orcapod.core.datagrams import Data, Datagram, Tag # noqa: E402 +from orcapod.operators import Batch, PolarsFilter # noqa: E402 +from orcapod.sources import ( # noqa: E402 + ArrowTableSource, + DataFrameSource, + DictSource, + ListSource, +) + +_tracer.install() + + +def probe(label: str, fn) -> None: + _tracer.HITS.clear() + try: + result = fn() + verdict = "TRUNCATED" if _tracer.HITS else "no truncation" + print(f"- {label}\n {verdict}: {result}") + except Exception as e: # noqa: BLE001 + print(f"- {label}\n raises {type(e).__name__}: {str(e)[:110]}") + for site in _tracer.summary(): + print(f" site: {site}") + + +def col(stream, name): + t = stream.as_table() + return f"{name}={t[name].to_pylist()} ({t.schema.field(name).type})" + + +print("== sources built from Python values ==") +probe( + "DictSource, data_schema={'k': str, 'x': int}, x=1.5", + lambda: col( + DictSource( + data=[{"k": "a", "x": 1.5}], + tag_columns=["k"], + data_schema={"k": str, "x": int}, + ), + "x", + ), +) +probe( + "DictSource, data_schema=pa.schema(x: int64), x=1.5", + lambda: col( + DictSource( + data=[{"k": "a", "x": 1.5}], + tag_columns=["k"], + data_schema=pa.schema([("k", pa.large_string()), ("x", pa.int64())]), + ), + "x", + ), +) +probe( + "DictSource, TAG column declared int, k=1.5", + lambda: col( + DictSource( + data=[{"k": 1.5, "x": 1}], + tag_columns=["k"], + data_schema={"k": int, "x": int}, + ), + "k", + ), +) +probe( + "DictSource, no schema, x = 1 then 1.5 (inferred)", + lambda: col( + DictSource(data=[{"k": "a", "x": 1}, {"k": "b", "x": 1.5}], tag_columns=["k"]), + "x", + ), +) +probe( + "ListSource, data_schema={'v': int}, element 1.5", + lambda: col( + ListSource(name="v", data=[1.5], data_schema={"v": int, "element_index": int}), + "v", + ), +) + +print("\n== sources built from Arrow / DataFrames (Table.cast, safe) ==") +probe( + "DataFrameSource, float column, schema=int64", + lambda: col( + DataFrameSource( + pl.DataFrame({"k": ["a"], "x": [1.5]}), + tag_columns=["k"], + schema=pa.schema([("k", pa.large_string()), ("x", pa.int64())]), + ), + "x", + ), +) +probe( + "ArrowTableSource, float64 column (type kept, no declared type to disagree with)", + lambda: col( + ArrowTableSource(pa.table({"k": ["a"], "x": [1.5]}), tag_columns=["k"]), "x" + ), +) + +print("\n== datagrams built from Python values ==") +probe( + "Datagram({'x': 1.5}, python_schema={'x': int}).as_table()", + lambda: Datagram({"x": 1.5}, python_schema={"x": int}).as_table().to_pylist(), +) +probe( + "Data({'x': 1.5}, python_schema={'x': int}).as_table()", + lambda: Data({"x": 1.5}, python_schema={"x": int}).as_table().to_pylist(), +) +probe( + "Tag({'k': 1.5}, python_schema={'k': int}).as_table()", + lambda: Tag({"k": 1.5}, python_schema={"k": int}).as_table().to_pylist(), +) + + +def _update_arrow_backed(): + d = Datagram({"x": 1}, python_schema={"x": int}) + d.as_table() # Arrow table loaded: update() takes its Arrow path + return d.update(x=1.5).as_table().to_pylist() + + +def _update_dict_backed(): + d = Datagram({"x": 1}, python_schema={"x": int}) + return d.update(x=1.5).as_table().to_pylist() + + +probe("Datagram.update(x=1.5) on an int column, Arrow-backed", _update_arrow_backed) +probe("Datagram.update(x=1.5) on an int column, dict-backed", _update_dict_backed) +probe( + "Datagram.with_columns(column_types={'z': int}, z=1.5)", + lambda: ( + Datagram({"x": 1}, python_schema={"x": int}) + .with_columns(column_types={"z": int}, z=1.5) + .as_table() + .to_pylist() + ), +) +probe( + "Datagram.with_columns(column_types={'z': pa.int64()}, z=1.5)", + lambda: ( + Datagram({"x": 1}, python_schema={"x": int}) + .with_columns(column_types={"z": pa.int64()}, z=1.5) + .as_table() + .to_pylist() + ), +) +probe( + "Data source_info value 1.5 (type derived from the value, no declared type)", + lambda: ( + Data({"y": 1}, source_info={"y": 1.5}) # pyright: ignore[reportArgumentType] # deliberate wrong value: probes the conversion + .as_table(columns={"source": True}) + .to_pylist() + ), +) + +print("\n== operators (Arrow in, Arrow out) ==") +src = DictSource(data=[{"k": "a", "x": 1}, {"k": "b", "x": 2}], tag_columns=["k"]) +probe("Batch(batch_size=2) over int column", lambda: col(Batch(batch_size=2)(src), "x")) +probe( + "PolarsFilter(x > 1) over int column", + lambda: col(PolarsFilter([pl.col("x") > 1])(src), "x"), +) diff --git a/superpowers/specs/itl-895-float-to-int/05_pod_matrix.py b/superpowers/specs/itl-895-float-to-int/05_pod_matrix.py new file mode 100644 index 000000000..a5a2dd48f --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/05_pod_matrix.py @@ -0,0 +1,146 @@ +"""Claim 5: declared output type x returned value, through a real pod, today. + +For each pair, one pod is built and run two ways: + +* ``standalone``: ``f.pod(src).as_table()`` (no store) +* ``job``: ``PipelineJob`` (sync orchestrator, in-memory store), reading the stored record + +Each cell is ``kept``, ``LOSSY`` (stored value differs from the returned one), +``converted`` (equal value, different Python type), or ``raises ``. The last +section shows the standalone ``as_table()`` "Complex unions" error from the issue. + +Run: uv run python superpowers/specs/itl-895-float-to-int/05_pod_matrix.py +""" + +from __future__ import annotations + +import datetime as dt +import logging +import warnings +from decimal import Decimal +from typing import Any + +import numpy as np + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + +warnings.simplefilter("ignore") +logging.disable(logging.CRITICAL) + +UTC = dt.timezone.utc +PLUS2 = dt.timezone(dt.timedelta(hours=2)) + +CASES: list[tuple[Any, Any]] = [ + (int, 1.5), + (int, 2.0), + (int, np.float64(1.5)), + (int, Decimal("1.5")), + (int, np.int64(3)), + (int, True), + (int, np.bool_(True)), + (int, "1"), + (int, None), + (int, 2**63), + (np.int8, 1.5), + (np.int8, 300), + (float, 1), + (float, True), + (float, 2**53 + 1), + (float, np.float32(0.1)), + (float, Decimal("1.5")), + (float, "1.0"), + (float, None), + (bool, 1), + (bool, np.bool_(True)), + (bool, 1.0), + (str, 1), + (str, b"x"), + (bytes, "x"), + (dt.datetime, dt.datetime(2020, 1, 1, 12, tzinfo=PLUS2)), + (dt.datetime, dt.datetime(2020, 1, 1, 12)), + (dt.datetime, dt.date(2020, 1, 1)), + (dt.datetime, 1.5), + (dt.date, dt.datetime(2020, 1, 1, 12)), + (dt.date, 1.5), + (list[int], [1.5]), + (list[int], (1, 2)), + (int | None, 1.5), +] + + +def make_pod(declared, value, label): + def fn(x: int): + return value + + fn.__annotations__["return"] = declared + fn.__name__ = fn.__qualname__ = label + return function_pod(output_keys="y")(fn) + + +def classify(stored: Any, value: Any) -> str: + if stored == value and type(stored) is type(value): + return f"kept {stored!r}" + try: + equal = stored == value + if isinstance(equal, np.ndarray): + equal = bool(equal.all()) + except Exception: # noqa: BLE001 + equal = False + if equal and not isinstance(value, dt.datetime): + return f"converted {stored!r}" + return f"LOSSY {stored!r}" + + +def run(declared, value, i) -> tuple[str, str]: + f = make_pod(declared, value, f"f{i}") + src = DictSource(data=[{"k": "a", "x": 1}], tag_columns=["k"], source_id="src") + out = [] + try: + out.append(classify(f.pod(src).as_table()["y"].to_pylist()[0], value)) + except Exception as e: # noqa: BLE001 + out.append(f"raises {type(e).__name__}") + try: + job = PipelineJob(name=f"m{i}", store=InMemoryArrowDatabase()) + with job: + f.pod(src, label="f") + job.run() + recs = job.nodes["f"].get_all_records() + out.append( + classify(recs["y"].to_pylist()[0], value) + if recs is not None and recs.num_rows + else "no row stored" + ) + except Exception as e: # noqa: BLE001 + out.append(f"raises {type(e).__name__}") + return out[0], out[1] + + +def name(t) -> str: + return getattr(t, "__name__", None) or str(t) + + +print( + f"{'declared':10} {'returned':42} {'standalone as_table':34} {'PipelineJob stored'}" +) +for i, (declared, value) in enumerate(CASES): + s, j = run(declared, value, i) + print(f"{name(declared):10} {repr(value)[:40]:42} {s[:34]:34} {j[:40]}") + +print( + "\n== the issue's 'Complex unions' error: standalone as_table() over mixed results ==" +) + + +@function_pod(output_keys="y") +def g(x: int) -> int: + return x if x == 1 else x + 0.5 # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +src2 = DictSource(data=[{"k": "a", "x": 1}, {"k": "b", "x": 2}], tag_columns=["k"]) +print(" iter_data:", [d.as_dict() for _, d in g.pod(src2).iter_data()]) +try: + g.pod(src2).as_table() +except Exception as e: # noqa: BLE001 + print(f" as_table raises {type(e).__name__}: {e}") diff --git a/superpowers/specs/itl-895-float-to-int/06_released_versions.py b/superpowers/specs/itl-895-float-to-int/06_released_versions.py new file mode 100644 index 000000000..04eca1d8d --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/06_released_versions.py @@ -0,0 +1,31 @@ +"""Claim 6: released versions have the bug (version-agnostic reproduction). + +Runs the issue's reproduction against whatever ``orcapod`` is importable, using only the +API every release since v0.1.0 shares (``function_pod``, ``DictSource``, ``.pod(...)``, +``as_table()``). + +Run against a release without touching this worktree: + + git archive v0.1.4 | tar -x -C /v0.1.4 + uv run --no-project --python 3.12 --with /v0.1.4 \ + python -I superpowers/specs/itl-895-float-to-int/06_released_versions.py +""" + +from __future__ import annotations + +from importlib.metadata import version + +from orcapod import function_pod +from orcapod.sources import DictSource + + +@function_pod(output_keys="y") +def f(x: int) -> int: + return x + 0.5 # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +src = DictSource(data=[{"k": "a", "x": 1}], tag_columns=["k"]) +t = f.pod(src).as_table() +print( + f"orcapod {version('orcapod')}: y={t['y'].to_pylist()} type={t.schema.field('y').type}" +) diff --git a/superpowers/specs/itl-895-float-to-int/07_blast_radius.py b/superpowers/specs/itl-895-float-to-int/07_blast_radius.py new file mode 100644 index 000000000..139cb1744 --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/07_blast_radius.py @@ -0,0 +1,79 @@ +"""Claim 7: blast radius for existing stores, and how to find affected rows. + +1. A truncated stored row is byte-identical to a genuine one: the stored value, the + output data hash and the record are those of ``1``; nothing records that the function + returned ``1.5``. Stored data alone cannot tell the two apart. +2. Later runs reuse the stored row without calling the function (Phase 1 of + ``FunctionNode``). A fix that rejects floats therefore does NOT touch rows already + stored: they keep being served until the user removes them or bumps the pod version. +3. Recipe to find affected rows: for each pod whose declared output contains an integer + type (or another type pyarrow coerces: ``date``/``datetime``, ``str``/``bytes``, + ``float32``, lists/structs of these), call the raw function again on the node's stored + inputs and flag rows whose returned value is not exactly the declared type. + +Run: uv run python superpowers/specs/itl-895-float-to-int/07_blast_radius.py +""" + +from __future__ import annotations + +import logging + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + +logging.disable(logging.CRITICAL) +calls: list[int] = [] + + +def raw_half(x: int) -> int: + calls.append(x) + return ( # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + x + 0.5 if x % 2 else x + ) # odd inputs return a float + + +half = function_pod(output_keys="y")(raw_half) + +rows = [{"k": f"r{i}", "x": i} for i in range(4)] +db = InMemoryArrowDatabase() + + +def build() -> PipelineJob: + job = PipelineJob(name="blast", store=db) + with job: + src = DictSource(data=rows, tag_columns=["k"], source_id="src") + half.pod(src, label="half") + return job + + +job = build() +job.run() +recs = job.nodes["half"].get_all_records(columns={"meta": True}) +print("1. stored rows (odd x returned x + 0.5):") +for k, y in zip(recs["k"].to_pylist(), recs["y"].to_pylist()): + x = int(k[1:]) + print(f" {k}: x={x} stored y={y!r}") +print( + f" stored column type: {recs.schema.field('y').type}; meta columns: " + f"{[c for c in recs.column_names if c.startswith('__')][:4]} ..." +) + +calls.clear() +job2 = build() +job2.run() +print( + f"\n2. second run over the same store: function calls={len(calls)}; " + f"y={job2.nodes['half'].as_table()['y'].to_pylist()} (served from the store)" +) + +print( + "\n3. finding affected rows: re-call the raw function on the node's stored inputs" +) +src_stream = job2.nodes["src"] +affected = [] +for tag, data in src_stream.iter_data(): + returned = raw_half(**data.as_dict()) + if type(returned) is not int: + affected.append((tag.as_dict()["k"], returned)) +print(f" affected rows (tag, value the function returns): {affected}") diff --git a/superpowers/specs/itl-895-float-to-int/08_none_output.py b/superpowers/specs/itl-895-float-to-int/08_none_output.py new file mode 100644 index 000000000..d08d988db --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/08_none_output.py @@ -0,0 +1,93 @@ +"""Claim 8: ``None`` returned for a non-optional output (coordinator follow-up from PR #289). + +Same family as float -> int: the returned value is never checked against the declared +output type. ``-> int`` maps to ``int64 not null``; ``pa.Table.from_pylist`` writes a +NULL into a ``not null`` field without error, and ``Table.validate(full=True)`` does not +check field nullability either. What happens next depends on the path: + +* standalone ``as_table()``: returns a table whose ``not null`` field holds a NULL; +* in-memory store: the run completes; the NULL is stored; reading the node back fails + in ``restore_schema_nullability`` (``Table.cast``: "Casting field 'y' with null values + to non-nullable"); +* Delta store: the Delta writer refuses the batch at flush ("declared as non-nullable + but contains null values"); today that is logged as a row failure and the final flush + raises ``RuntimeError`` (ITL-878 / PR #287 turns it into ``StoreWriteError``). + +Run: uv run python superpowers/specs/itl-895-float-to-int/08_none_output.py +""" + +from __future__ import annotations + +import logging +import tempfile +from pathlib import Path + +import pyarrow as pa + +from orcapod import PipelineJob, function_pod +from orcapod.core.datagrams import Data +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator +from orcapod.sources import DictSource + +logging.disable(logging.CRITICAL) + +print("== pyarrow only ==") +t = pa.Table.from_pylist( + [{"y": None}], schema=pa.schema([pa.field("y", pa.int64(), nullable=False)]) +) +print( + f" from_pylist(None, 'y: int64 not null') -> {t.to_pylist()} field={t.schema.field('y')} " + f"null_count={t['y'].null_count}" +) +t.validate(full=True) +print(" Table.validate(full=True) -> passes (Arrow does not check field nullability)") + +print("\n== orcapod Data ==") +d = Data({"y": None}, python_schema={"y": int}) +print( + f" Data({{'y': None}}, python_schema={{'y': int}}).as_table() -> {d.as_table().to_pylist()} " + f"field={d.as_table().schema.field('y')}" +) + + +@function_pod(output_keys="y") +def f(x: int) -> int: + return None # pyright: ignore[reportReturnType] # deliberate wrong return type: reproduces ITL-895 + + +def src() -> DictSource: + return DictSource(data=[{"k": "a", "x": 1}], tag_columns=["k"], source_id="s") + + +print("\n== standalone pod ==") +st = f.pod(src()).as_table() +print(f" as_table() -> y={st['y'].to_pylist()} field={st.schema.field('y')}") + +print("\n== PipelineJob ==") +for label, kind, orch in [ + ("in-memory, sync", "mem", None), + ("in-memory, async", "mem", AsyncPipelineOrchestrator()), + ("Delta, sync", "delta", None), + ("Delta, async", "delta", AsyncPipelineOrchestrator()), +]: + with tempfile.TemporaryDirectory() as tmp: + store = ( + DeltaTableDatabase(Path(tmp) / "db") + if kind == "delta" + else InMemoryArrowDatabase() + ) + job = PipelineJob(name="none_out", store=store) + with job: + f.pod(src(), label="f") + try: + job.run(orchestrator=orch) + run = "run completes" + except Exception as e: # noqa: BLE001 + run = f"run raises {type(e).__name__}: {str(e)[:80]}" + try: + recs = job.nodes["f"].get_all_records() + read = "no rows" if recs is None else f"stored y={recs['y'].to_pylist()}" + except Exception as e: # noqa: BLE001 + read = f"read raises {type(e).__name__}: {str(e)[:70]}" + print(f" {label:18} {run}\n {'':18} {read}") diff --git a/superpowers/specs/itl-895-float-to-int/09_warning_dedup.py b/superpowers/specs/itl-895-float-to-int/09_warning_dedup.py new file mode 100644 index 000000000..e12127601 --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/09_warning_dedup.py @@ -0,0 +1,82 @@ +"""How many ValueConversionWarnings does a long run show? (ITL-895) + +Each scenario runs in a fresh interpreter and counts what ``warnings.showwarning`` is asked +to show, under Python's default filters and under ``-W always`` (``simplefilter("always")``, +which disables the warning registry). A pod with an output declared ``list[int]`` returns +a tuple (D5) and an output declared ``datetime.date`` returns a datetime (D6) on every row. +The function instance deduplicates explicitly, so the expected count is two per function +instance, under either filter: one per output x conversion. + +The standalone scenarios use 10,000 rows. The ``PipelineJob`` scenarios use 1,000 rows, +because a ``PipelineJob`` run grows faster than linearly with the row count on main too +(1,000 rows took minutes), which is not what this script measures; the deduplication does not +depend on the row count. + +Run: ``uv run python superpowers/specs/itl-895-float-to-int/09_warning_dedup.py`` +""" + +from __future__ import annotations + +import datetime as dt +import subprocess +import sys +import warnings + +SCENARIOS = ("standalone", "standalone-twice", "sync", "async") + + +def run(scenario: str, always: bool) -> None: + from orcapod import PipelineJob, function_pod + from orcapod.databases import InMemoryArrowDatabase + from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator + from orcapod.sources import DictSource + + shown: list[str] = [] + original = warnings.showwarning + + def count(message, category, filename, lineno, file=None, line=None): # noqa: ANN001, ANN202 + shown.append(f"{category.__name__}: {message}") + + warnings.showwarning = count + if always: + warnings.simplefilter("always") + rows = 10_000 if scenario.startswith("standalone") else 1_000 + + @function_pod(output_keys=("pair", "day")) + def near_miss(x: int) -> tuple[list[int], dt.date]: + return (x, x), dt.datetime(2020, 1, 1, x % 24) # pyright: ignore[reportReturnType] + + source = DictSource( + data=[{"k": f"r{x}", "x": x} for x in range(rows)], tag_columns=["k"] + ) + stored = 0 + if scenario.startswith("standalone"): + for _ in range(2 if scenario == "standalone-twice" else 1): + stored = near_miss.pod(source).as_table().num_rows + else: + job = PipelineJob(name="dedup", store=InMemoryArrowDatabase()) + with job: + near_miss.pod(source, label="near_miss") + orchestrator = ( + SyncPipelineOrchestrator() + if scenario == "sync" + else AsyncPipelineOrchestrator() + ) + job.run(orchestrator=orchestrator) + stored = job.nodes["near_miss"].as_table().num_rows + warnings.showwarning = original + filters = "always" if always else "default" + print(f"{scenario} ({filters} filter): {stored} rows, {len(shown)} warnings shown") + for line in shown: + print(f" {line}") + + +if __name__ == "__main__": + if len(sys.argv) > 1: + run(sys.argv[1], always=sys.argv[2] == "always") + else: + for name in SCENARIOS: + for filters in ("default", "always"): + _ = subprocess.run( + [sys.executable, __file__, name, filters], check=True + ) diff --git a/superpowers/specs/itl-895-float-to-int/FINDINGS.md b/superpowers/specs/itl-895-float-to-int/FINDINGS.md new file mode 100644 index 000000000..37a4be7db --- /dev/null +++ b/superpowers/specs/itl-895-float-to-int/FINDINGS.md @@ -0,0 +1,328 @@ +> Historical investigation and November 11 decisions. The November 12 policy in +> `docs/concepts/function-pods.md` supersedes warning conversions. Probe +> `09_warning_dedup.py` measures commit `5537c0a6`, not the current branch. + +# ITL-895 root cause: a pod declared `-> int` stores `1.5` as `1` + +Investigated on `b197544e` (origin/main, November 11, 2026). Each claim has a script in this +directory, and its captured output is in `outputs/`. Run a script with +`uv run python superpowers/specs/itl-895-float-to-int/