Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,26 @@ data. Never inspects tags. Two execution models:
- FunctionPod → FunctionPodStream: lazy, in-memory
- FunctionNode: DB-backed, two-phase (yield cached results first, then compute missing)

Calling a function pod (developer, November 12, 2026, ITL-807). pod(a, b) is the
convenient call and pod.process(a, b) the exact primitive:

- pod(a, b) joins the inputs as process does and, when the joined stream carries data
columns that are not parameters of the function, places one automatic SelectDataColumns
after the automatic join and before the pod (_FunctionPodBase.__call__, the one place
that decides it). The select is an ordinary operator step: graph, hashes and stored tables
must equal those of a pipeline with that select written by hand (the stream's
`select_data_columns` method builds the same operator, and `a.join(b)` the same `Join`).
Nothing to drop means no step. One info log names the dropped columns; it is not a warning.
- pod(a, b, strict=True) and pod.process(a, b) reject an extra column. strict is part of
no identity. process never takes a strict argument and never drops a column.
- A missing required column is an error in every form. Tags are never dropped. A data column
shared by two inputs fails at the join, used or not: no special handling.
- No select is added when the data function's accepts_arbitrary_keywords is true (no kind
today; PythonDataFunction rejects variadic keyword arguments).
- Function pods only. An operator's and a SideEffectPod's __call__ stay aliases of
process with no strict argument. There is no global setting.
- A test of the exact input contract uses process or strict=True.

Operator (core/operators/) — structural pod transforming streams without synthesizing new
data values. All subclass StaticOutputPod:
- UnaryOperator — 1 input (Batch, Select/Drop columns, Map, Filter)
Expand Down
20 changes: 20 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,26 @@ data. Never inspects tags. Two execution models:
- `FunctionPod` → `FunctionPodStream`: lazy, in-memory
- `FunctionNode`: DB-backed, two-phase (yield cached results first, then compute missing)

**Calling a function pod** (developer, November 12, 2026, ITL-807). `pod(a, b)` is the
convenient call and `pod.process(a, b)` the exact primitive:

- `pod(a, b)` joins the inputs as `process` does and, when the joined stream carries data
columns that are not parameters of the function, places one automatic `SelectDataColumns`
after the automatic join and before the pod (`_FunctionPodBase.__call__`, the one place
that decides it). The select is an ordinary operator step: graph, hashes and stored tables
must equal those of a pipeline with that select written by hand (the stream's
`select_data_columns` method builds the same operator, and `a.join(b)` the same `Join`).
Nothing to drop means no step. One info log names the dropped columns; it is not a warning.
- `pod(a, b, strict=True)` and `pod.process(a, b)` reject an extra column. `strict` is part of
no identity. `process` never takes a `strict` argument and never drops a column.
- A missing required column is an error in every form. Tags are never dropped. A data column
shared by two inputs fails at the join, used or not: no special handling.
- No select is added when the data function's `accepts_arbitrary_keywords` is true (no kind
today; `PythonDataFunction` rejects `**kwargs`).
- Function pods only. An operator's and a `SideEffectPod`'s `__call__` stay aliases of
`process` with no `strict` argument. There is no global setting.
- A test of the exact input contract uses `process` or `strict=True`.

**Operator** (`core/operators/`) — structural pod transforming streams without synthesizing new
data values. All subclass `StaticOutputPod`:
- `UnaryOperator` — 1 input (Batch, Select/Drop columns, Map, Filter)
Expand Down
15 changes: 15 additions & 0 deletions changelog.d/ITL-807.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
### Changed

- Calling a function pod, `pod(a, b)`, now drops the data columns its function does not take
(ITL-807, ITL-193). The inputs are joined as before; when the joined stream carries data
columns that are not parameters of the function, one automatic `SelectDataColumns` is
placed after the automatic join and before the pod. It is an ordinary operator step: the
graph, the hashes and the stored tables are those of a pipeline with that select written by
hand (`pod(wide.select_data_columns([...]))`, or `pod(a.join(b).select_data_columns([...]))`), so a change in an unused column no longer causes a recomputation. When nothing would
be dropped no step is added and the pipeline is the one built before. One message at `INFO`
level names the dropped columns. `pod(a, b, strict=True)` rejects an extra column, and
`pod.process(a, b)` is the exact primitive and always rejects it. Code that relied on
`pod(wide)` raising for an extra column must use `strict=True` or `process`. A missing
column for a parameter without a default is an error in every form; tag columns are never
dropped; two inputs sharing a data column still fail at the join. Operators and
`SideEffectPod` are unchanged: their `__call__` stays an alias of `process`.
24 changes: 24 additions & 0 deletions design_issues/ITL-807-F16.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## `src/orcapod/core/function_pod.py`

### F16 — Docs said a function pod extracts the columns it needs; the code rejected any extra column
**Status:** resolved
**Severity:** medium
**Issue:** ITL-807

`docs/concepts/function-pods.md` once said that a function pod picks the columns it needs out
of a wider stream, while `_FunctionPodBase._validate_input_schema` rejected every data column
that was not a parameter of the function. Users had to put a `SelectDataColumns` in front of
each pod by hand (ITL-193 asked for the automatic drop). `__call__` was a plain alias of
`process`.

**Fix:** Developer decision of November 12, 2026. `_FunctionPodBase.__call__(*streams,
label=None, strict=False)` is the convenient call: it joins the inputs as `process` does and,
when the joined stream carries data columns that are not parameters, applies one
`SelectDataColumns(columns=<parameters present>)` after the automatic join and before the
pod, then calls `process`. The select is an ordinary operator step, so the graph, hashes and
stored tables equal those of a pipeline with the select written by hand (the stream methods
`select_data_columns` and `join` build the same operators; pinned by a test); when nothing would
be dropped no step is added. One info message names the dropped columns. `strict=True` and
`process` reject an extra column as before; `strict` is in no identity. No select is added for
a data function whose `accepts_arbitrary_keywords` is true (none today). Operators and
`SideEffectPod` keep `__call__` as an alias of `process`.
117 changes: 90 additions & 27 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@ Key points:

- **`output_keys`** names the output data column(s). A single string means one output
column; a list of strings means the function returns multiple values.
- **Function parameters** must match the input stream's data columns exactly: one data
column per parameter, with the same names. A parameter with a default value may be left out
of the stream, and the default is used. See [Input columns must match exactly](#input-columns-must-match-exactly).
- **Function parameters** are matched to the input stream's data columns by name: one data
column per parameter. A parameter with a default value may be left out of the stream, and
the default is used. Data columns the function does not take are dropped when the pod is
called. See [Input columns](#input-columns).
- **Default values** are part of the function's signature hash, so a default must be a value
Orcapod can hash: `None`, numbers, strings, `bytes`, and lists, tuples, sets and dicts of
them. A default of another type, such as a `Path`, an `Enum` member, a `datetime` or a
Expand Down Expand Up @@ -71,7 +72,7 @@ source = DictSource(
)

# Apply the function pod to the source stream
result = compute_bmi.pod(source) # shorthand for compute_bmi.pod.process(source)
result = compute_bmi.pod(source)

# Inspect the output schema -- tags pass through, data are replaced
tag_schema, data_schema = result.output_schema()
Expand All @@ -90,18 +91,16 @@ for tag, data in result.iter_data():
The function pod preserves tags and replaces data columns with the function's output.

!!! tip
All standard pods support `__call__` as a shorthand for `.process()`, so
`compute_bmi.pod(source)` is equivalent to `compute_bmi.pod.process(source)`.
Calling a function pod, `compute_bmi.pod(source)`, is the convenient form of
`compute_bmi.pod.process(source)`: it first drops the data columns the function does not
take (see [Input columns](#input-columns)). When the stream has exactly the function's
columns, as here, the two are the same call.

## Input columns must match exactly
## Input columns

A function pod is strict about its input: the stream's data columns must be exactly the
function's parameters. Orcapod does **not** pick the columns it needs out of a wider stream.
A stream with an extra data column, a missing column for a parameter without a default, or a
column whose type is incompatible with its parameter is rejected with `ValueError` as soon as
the pod is applied. Types need to be compatible, not identical: a `str` column is accepted for
a `str | Path` parameter but rejected for an `int` one. (A parameter that has a default value
may be left out of the stream; the function then uses the default.)
A function takes one data column per parameter, matched by name. Calling the pod,
`compute_bmi.pod(stream)`, is the convenient form: data columns that are not parameters of
the function are dropped, and the function receives the rest. Tag columns are never dropped.

```python
wide = DictSource(
Expand All @@ -112,8 +111,51 @@ wide = DictSource(
tag_columns=["subject_id"],
)

result = compute_bmi.pod(wide) # 'age' is not a parameter: it is dropped
for tag, data in result.iter_data():
print(f"{tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_01'} -> {'bmi': 1756.9444444444446}
# {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995}
```

The drop is not hidden. Orcapod places one `SelectDataColumns` [operator](operators.md)
between the stream and the pod, exactly as if you had written it yourself with the stream's
`select_data_columns` method, so the two pipelines below are the same pipeline: same hashes,
same graph, same stored tables. `SelectDataColumns` is the name that appears in the graph.

```python
narrow = wide.select_data_columns(["weight", "height"])
by_hand = compute_bmi.pod(narrow)

print(result.pipeline_hash() == by_hand.pipeline_hash())
# True
print(result.content_hash() == by_hand.content_hash())
# True
```

Because the unused columns are cut before the function's input is hashed, a change in an
unused column does not cause a recomputation. When the stream already has exactly the
function's columns, no step is added.

Each call that drops columns logs one message at `INFO` level (logger
`orcapod.core.function_pod`) naming them. It is worth reading when a parameter has a default
value: a misspelt column (`Height` for a parameter `height=1.0`) is dropped like any other
unused column and the default is used, and that message is the only trace.

### Exact calls: `strict=True` and `.process()`

To have an extra column rejected instead of dropped, pass `strict=True`, or call
`.process()`, which is the exact primitive and never drops a column:

```python
try:
compute_bmi.pod(wide, strict=True)
except ValueError as err:
print(type(err).__name__)
# ValueError

try:
compute_bmi.pod(wide)
compute_bmi.pod.process(wide)
except ValueError as err:
print(type(err).__name__)
# ValueError
Expand All @@ -124,19 +166,20 @@ The error message names both schemas and the first column that does not fit, for
compatible with expected input schema Schema({'weight': float, 'height': float}): column
'age' is not a parameter of the function`.

Narrow the stream first with the `SelectDataColumns` [operator](operators.md) (or drop the
extras with `DropDataColumns`):
`strict` decides only whether an extra column is an error. It is not part of any hash: when
there is nothing to drop, `pod(stream)`, `pod(stream, strict=True)` and `pod.process(stream)`
build the same pipeline.

```python
from orcapod.operators import SelectDataColumns
### What is an error in every form

narrow = SelectDataColumns(columns=["weight", "height"])(wide)
result = compute_bmi.pod(narrow)
for tag, data in result.iter_data():
print(f"{tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_01'} -> {'bmi': 1756.9444444444446}
# {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995}
```
A missing column for a parameter without a default, or a column whose type is incompatible
with its parameter, is rejected with `ValueError` as soon as the pod is applied, however it
is applied. Types need to be compatible, not identical: a `str` column is accepted for a
`str | Path` parameter but rejected for an `int` one. (A parameter that has a default value
may be left out of the stream; the function then uses the default.)

A stream must keep at least one data column, so a stream none of whose data columns is a
parameter is rejected as well.

The same step is available as a stream method: `wide.select_data_columns(["weight", "height"])`.
Tag columns are never affected by this rule -- a function pod passes all tags through
Expand Down Expand Up @@ -615,7 +658,27 @@ for tag, data in result.iter_data():
```

The join happens on shared tag columns, and the merged data columns are fed to the function.
The [exact-match rule](#input-columns-must-match-exactly) applies to the joined stream.
The [input column rules](#input-columns) apply to the joined stream: the join comes first,
then one automatic `SelectDataColumns` drops the joined columns the function does not take.
Written by hand, that is `a.join(b).select_data_columns([...])` passed to the pod.
Two consequences:

- Two inputs that carry the same data column are rejected by the join, whether or not the
function takes that column. Drop it from one input yourself.
- A stream can be passed to restrict the rows by its tags: it is joined, and its data
columns, which the function does not take, are then dropped.

```python
chosen = DictSource(
data=[{"subject_id": "mouse_02", "cohort": "b"}],
tag_columns=["subject_id"],
)

result = compute_bmi.pod(weights, heights, chosen)
for tag, data in result.iter_data():
print(f"{tag.as_dict()} -> {data.as_dict()}")
# {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995}
```

## DataFunction internals

Expand Down
13 changes: 7 additions & 6 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,16 +111,16 @@ A few things to note about function pods:

- **Parameter names match data column names.** The function's parameter names
(`temperature`, `pressure`) must match the data column names from the input stream
exactly -- one parameter per data column. A stream with extra data columns is rejected;
narrow it first with `source.select_data_columns([...])`
(see [Function Pods](concepts/function-pods.md#input-columns-must-match-exactly)).
exactly -- one parameter per data column. Data columns the function does not take are
dropped when you call the pod; `pod(stream, strict=True)` and `pod.process(stream)`
reject them instead (see [Function Pods](concepts/function-pods.md#input-columns)).
- **Type annotations define the schemas.** Orcapod uses them to build the output schema and
to validate compatibility when the pod is applied. A parameter without an annotation needs
its type in the decorator's `input_schema` mapping instead.
- **`output_keys` names the output columns.** Since the function returns a tuple of two
values, `output_keys` must be a list of two names. For a function returning a single
value, pass a single string (e.g., `output_keys="result"`).
- **The decorator creates a `.pod` attribute.** You call `.pod.process()` to apply
- **The decorator creates a `.pod` attribute.** You call `.pod(stream)` to apply
the function to a stream.

Now apply it:
Expand All @@ -130,8 +130,9 @@ result = analyze_conditions.pod(source)
```

!!! tip
All standard pods support `__call__` as a shorthand for `.process()`, so
`pod(stream)` is equivalent to `pod.process(stream)`.
For an operator, `op(stream)` is a shorthand for `op.process(stream)`. For a function
pod, `pod(stream)` also drops the data columns the function does not take, while
`pod.process(stream)` is exact and rejects them.

The `result` is a new stream. Tags are preserved from the input; the data columns
are replaced with the function's outputs.
Expand Down
15 changes: 15 additions & 0 deletions src/orcapod/core/data_function.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,17 @@ def input_data_schema(self) -> Schema:
"""Schema describing the input data this function accepts."""
...

@property
def accepts_arbitrary_keywords(self) -> bool:
"""Whether the function takes input columns beyond ``input_data_schema``.

``False`` for every data function kind today: the parameters are a fixed
set (``PythonDataFunction`` rejects ``**kwargs``). A kind that passes every
input column to its function overrides this to return ``True``; a function
pod then never drops a column for it (ITL-807).
"""
return False

@property
@abstractmethod
def output_data_schema(self) -> Schema:
Expand Down Expand Up @@ -958,6 +969,10 @@ def canonical_function_name(self) -> str:
def input_data_schema(self) -> Schema:
return self._data_function.input_data_schema

@property
def accepts_arbitrary_keywords(self) -> bool:
return self._data_function.accepts_arbitrary_keywords

@property
def output_data_schema(self) -> Schema:
return self._data_function.output_data_schema
Expand Down
Loading
Loading