Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions changelog.d/ITL-1257.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
### Breaking changes and migration

- A function with no `output_keys` that returns a value other than `None` raises
`NoOutputsDeclaredError` (an `OutputTypeError`) instead of having the value dropped
(ITL-1257). To keep the value, declare an output key with its type (a return
annotation or `output_schema`); otherwise return nothing.

### Fixed

- A function annotated `-> None` with no `output_keys` is accepted as a function with no
outputs, like a function with no return annotation (ITL-1257). It was refused ("has a
return type annotation, but no return keys were specified") because `typing.get_type_hints`
reads `-> None` as `NoneType`. `-> None` with `output_keys` is now refused when the pod is
created; it was accepted and gave an output column of type `NoneType`.
- A function with a positional-only parameter is refused when the pod is created, naming
the parameter (ITL-1258). A pod passes its inputs by name, so a call that supplied
such a parameter failed with a `TypeError` (one with a default worked only while the
input never held it, and the parameter could never be given).
16 changes: 15 additions & 1 deletion docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,18 @@ def compute_bmi(weight: float, height: float) -> float:
Key points:

- **`output_keys`** names the output data column(s). A single string means one output
column; a list of strings means the function returns multiple values.
column; a list of strings means the function returns multiple values. A function with
no outputs (it only reads its inputs) is annotated `-> None`, or not annotated, and
takes no `output_keys`; its pod has no output columns. `-> None` together with
`output_keys` raises `ValueError` when the pod is created, and a value other than
`None` returned by a function with no outputs raises `NoOutputsDeclaredError` (an
`OutputTypeError`): to keep the value, declare an output key with its type (a return
annotation or `output_schema`), or return nothing. Such a pod is
limited for now: in a job its rows fail, over data-tags and before an operator it
raises (ITL-1260); for a function run for its effect, use a `SideEffectPod`. An
output that may be
missing is not "no outputs": it is one output key typed `T | None`, and a returned
`None` is stored as a null.
- **Function parameters** are matched to the input stream's data columns by name: one data
column per parameter. A parameter with a default value may be left out of the stream, and
the default is used. Data columns the function does not take are dropped when the pod is
Expand All @@ -34,6 +45,9 @@ Key points:
dataclass instance, raises `TypeError` when the pod is created, naming the parameter. Use
a hashable default and build the object inside the function, or supply the value as an
input column. See [Function values and parameter defaults](../reference/hashing.md#1d-function-values-and-parameter-defaults).
- **Inputs are passed by name.** A parameter may be keyword-only (after `*`), but a
positional-only parameter (before `/`) and `*args` / `**kwargs` raise `ValueError`
when the pod is created.
- **Parameter types** are used to validate schema compatibility. Give each parameter a type
annotation, or supply its type through the decorator's `input_schema` mapping. A parameter
with neither raises `ValueError` when the pod is created.
Expand Down
23 changes: 23 additions & 0 deletions docs/dev/design_issues/ITL-1257-NR1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## `src/orcapod/core/data_function.py` — returned values not checked against the declared type

### NR1 — `-> None` with no output keys was refused; a value returned with no outputs was dropped
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1257 (related: ITL-1258)

`extract_function_schemas` (`utils/schema_utils.py`) has a branch for a function annotated
`-> None`: with no output keys it means no outputs, with output keys it is refused. The
branch was never reached, because `typing.get_type_hints` reads `-> None` as `NoneType`:
`-> None` with no output keys was refused ("has a return type annotation, but no return
keys were specified"), while `-> None` with one output key was accepted with an output
column of type `NoneType`. Separately, a function with no output keys that returned a
value had the value dropped silently (`parse_function_outputs`).

**Fix:** `extract_function_schemas` reads `NoneType` as `-> None`, so `-> None` with no
output keys means no outputs (as no return annotation does) and `-> None` with output keys
is refused at construction. A function with no output keys that returns a value other
than `None` raises `NoOutputsDeclaredError`, an `OutputTypeError`, at call time, on the
sync and the async path (developer: "declared `-> None` (or lack of any return type) should
be treated as empty return"). Tests: `tests/test_core/data_function/test_none_return.py`.
In the same change, a positional-only parameter is refused at construction (ITL-1258: a
call that supplied it failed; `test_positional_only.py`).
27 changes: 25 additions & 2 deletions src/orcapod/core/data_function.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
get_function_components,
)
from orcapod.core.result_cache import ResultCache
from orcapod.errors import OutputTypeError
from orcapod.errors import NoOutputsDeclaredError, OutputTypeError
from orcapod.protocols.core_protocols import DataFunctionProtocol, DataProtocol
from orcapod.protocols.core_protocols.executor import (
DataFunctionExecutorProtocol,
Expand Down Expand Up @@ -80,7 +80,8 @@ def parse_function_outputs(
Map raw function return values to a keyed output dict.

Rules:
- ``output_keys = []``: return value is ignored; empty dict returned.
- ``output_keys = []``: empty dict returned. ``PythonDataFunction`` refuses a
returned value other than ``None`` before this is called (ITL-1257).
- ``output_keys = ["result"]``: any value (including iterables) is stored as-is
under the single key.
- ``output_keys = ["a", "b", ...]``: ``values`` must be iterable and its length
Expand Down Expand Up @@ -439,11 +440,27 @@ def __init__(
f"parameters (*args / **kwargs). "
f"Offending parameters: {_variadic!r}."
)
# A pod passes its inputs by name, so a positional-only parameter could never
# be given (ITL-1258).
_positional_only = [
name
for name, param in _sig.parameters.items()
if param.kind is inspect.Parameter.POSITIONAL_ONLY
]
if _positional_only:
raise ValueError(
f"PythonDataFunction does not support functions with positional-only "
f"parameters: a function pod passes its inputs by name. "
f"Offending parameters: {_positional_only!r}; remove the '/' from the "
f"signature."
)

if output_keys is None:
output_keys = []
if isinstance(output_keys, str):
output_keys = [output_keys]
# A copy taken once: a later change to the caller's list changes nothing.
output_keys = tuple(output_keys)
self._output_keys = output_keys
if function_name is None:
if hasattr(self._function, "__name__"):
Expand Down Expand Up @@ -617,9 +634,15 @@ def _build_output_data(self, values: Any) -> DataProtocol:
A Data containing the parsed outputs with source info.

Raises:
NoOutputsDeclaredError: If the function declares no outputs and returned a
value other than ``None``.
OutputTypeError: If a returned value does not conform to its declared type,
or cannot be converted to it.
"""
if not self._output_keys and values is not None:
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
# No output keys declare an empty return (ITL-1257): a returned value
# contradicts the declaration and is refused, never dropped.
raise NoOutputsDeclaredError(self.canonical_function_name, values)
output_data = parse_function_outputs(self._output_keys, values)
output_schema = self.output_data_schema
for key, value in output_data.items():
Expand Down
57 changes: 54 additions & 3 deletions src/orcapod/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -305,9 +305,14 @@ class OutputTypeError(TypeError, ValueError):

Attributes:
function_name: Name of the function pod's function.
output_key: Name of the output column.
declared_type: The declared type of that output.
output_key: Name of the output column; ``None`` for ``NoOutputsDeclaredError``
(the function declares no output).
declared_type: The declared type of that output; ``None`` for
``NoOutputsDeclaredError``.
value: The value the function returned for that output.
detail: Where inside the value the mismatch is, when that is not the whole
value; ``None`` otherwise.
hint: What the function should return instead, when there is a specific fix.
"""

def __init__(
Expand All @@ -332,7 +337,7 @@ def __init__(
fix (``Return a float instead: 0.0 or float(value)``).
"""
self.function_name: str = function_name
self.output_key: str = output_key
self.output_key: str | None = output_key
self.declared_type: object = declared_type
self.value: object = value
self.detail: str | None = detail
Expand Down Expand Up @@ -362,6 +367,52 @@ def __reduce__(self) -> tuple[object, ...]:
)


class NoOutputsDeclaredError(OutputTypeError):
"""A function that declares no outputs returned a value (ITL-1257).

A function with no ``output_keys``, annotated ``-> None`` or not annotated, declares
an empty return: the number of outputs is the number of output keys. A returned value
other than ``None`` contradicts the declaration, so it is refused rather than
dropped. It is an ``OutputTypeError``, so a pipeline treats it as one: a row failure
under ``error_policy``.

It keeps the attributes of ``OutputTypeError``: ``output_key`` and
``declared_type`` are ``None`` (there is no output), ``detail`` is ``None`` and
``hint`` says how to fix it.

Attributes:
function_name: Name of the function pod's function.
value: The value the function returned.
"""

def __init__(self, function_name: str, value: object) -> None:
"""Create the error for one call.

Args:
function_name: Name of the function pod's function.
value: The value the function returned.
"""
self.function_name: str = function_name
self.output_key: str | None = None
self.declared_type: object = None
self.value: object = value
self.detail: str | None = None
self.hint: str | None = (
"Declare an output key with its type (a return "
"annotation or output_schema), or return nothing"
)
# Types only, never the value (see ``OutputTypeError``).
TypeError.__init__(
self,
Comment thread
Copilot marked this conversation as resolved.
Resolved
f"Function {function_name!r} declares no outputs (no output_keys) but "
+ f"returned {type_text(type(value))}. {self.hint}.",
)

def __reduce__(self) -> tuple[object, ...]:
"""Pickle with the constructor arguments (executors send errors across processes)."""
return (type(self), (self.function_name, self.value))


class UnboundSourceError(RuntimeError):
"""Raised when a data-producing method is called on an unbound SourceNode.

Expand Down
23 changes: 23 additions & 0 deletions src/orcapod/utils/schema_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,25 @@ def schema_incompatibility(
return None


def _declares_none(annotation: object) -> bool:
"""Whether a return annotation, as read, is ``-> None``.

``typing.get_type_hints`` turns ``-> None`` into ``NoneType``. When it fails on
another annotation, a postponed annotation (``from __future__ import annotations``)
is still text: ``"None"``, or ``"'None'"`` when it was written as a string
(``-> "None"``).

Args:
annotation: The return annotation as ``get_type_hints`` or the signature gives it.

Returns:
True when it is ``None`` written in one of those forms.
"""
if annotation is type(None):
return True
return isinstance(annotation, str) and annotation.strip("'\"") == "None"


def extract_function_schemas(
func: Callable,
output_keys: Collection[str],
Expand Down Expand Up @@ -259,6 +278,10 @@ def extract_function_schemas(

# get_type_hints stores the return annotation under the key 'return'
return_annot = resolved_hints.get("return", signature.return_annotation)
# ``-> None`` declares no outputs (ITL-1257), which is not an output whose value is
# None: the number of outputs is the number of output keys.
if _declares_none(return_annot):
return_annot = None
inferred_output_types: Schema = {}
if return_annot is not inspect.Signature.empty and return_annot is not None:
output_item_types = []
Expand Down
39 changes: 39 additions & 0 deletions tests/test_core/data_function/string_annotated_functions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
"""Functions whose annotations are strings (``from __future__ import annotations``).

Used by ``test_none_return.py`` (ITL-1257): ``-> None`` read through
``typing.get_type_hints`` from a string annotation.
"""

from __future__ import annotations


def log_value(x: int) -> None:
"""Declares no outputs."""
del x


def log_value_unannotated(x: int): # no return annotation on purpose
"""Declares no outputs by having no return annotation."""
del x


def maybe(x: int) -> int | None:
"""One output that may be null."""
return None if x < 0 else x


def unresolvable_input(x: Missing) -> None: # noqa: F821 # pyright: ignore[reportUndefinedVariable] # unresolvable on purpose
"""Declares no outputs; its parameter annotation cannot be resolved.

``typing.get_type_hints`` fails on it, so the return annotation stays the text
``"None"``; the input type is supplied with ``input_schema``.
"""
del x


def unresolvable_input_quoted(x: Missing) -> "None": # noqa: F821 # pyright: ignore[reportUndefinedVariable] # unresolvable on purpose
"""As ``unresolvable_input``, with the return annotation written as a string.

The postponed annotation is then the text ``"'None'"``.
"""
del x
4 changes: 2 additions & 2 deletions tests/test_core/data_function/test_data_function.py
Original file line number Diff line number Diff line change
Expand Up @@ -222,9 +222,9 @@ def test_output_data_schema_has_correct_types(self, add_pf):
schema = add_pf.output_data_schema
assert schema["result"] is int

def test_output_keys_string_normalised_to_list(self):
def test_output_keys_string_normalised_to_tuple(self):
pf = PythonDataFunction(add, output_keys="result")
assert pf._output_keys == ["result"]
assert pf._output_keys == ("result",)

def test_output_keys_collection_preserved(self):
pf = PythonDataFunction(multi, output_keys=["sum", "product"])
Expand Down
Loading
Loading