Skip to content
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,16 @@
cannot be turned into a record (a naive `datetime`, a value not matching the declared
type) is still a row failure governed by `error_policy`, like a function that raised. See
*When the store cannot be written* in the function-pods guide. (ITL-878 #287)
- `orcapod.enable_file_hash_caching` is exported at the top level (ITL-821).
- A source built from data with no rows and no schema (e.g. `DictSource(data=[])`) raises
`ValueError: Table is empty` instead of `tag_columns not found in table` (ITL-821).
- Applying a function pod to a stream whose data schema does not fit no longer prints a stray
`Key ... not found in parameter types.` line, and the error names the offending column,
e.g. `Incoming data schema ... is not compatible with expected input schema ...: column
'age' is not a parameter of the function` (it used to say `Incoming data data type`)
(ITL-821).
- `examples/save_and_load_pipelines.py` runs again, rewritten around `PipelineJob`; the test
suite now runs every example (ITL-821).

## [0.2.0] — 2026-11-10

Expand Down
35 changes: 35 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2293,3 +2293,38 @@ xfail. The async path checks key types on the first row of each input, because
So when either input is empty (left or right), a key type mismatch that the sync path reports
yields an empty result on the async path instead of an error. `Join`'s async key check
(`_check_shared_key_types`) has the same limit.

---

## Newcomer papercuts found by the docs audit

### NP1 — Missing export, misleading errors, inconsistent defaults, broken example, write noise
**Status:** in progress
**Severity:** medium
**Issue:** ITL-821

The docs audit (PR #271) ran every sample against `main` and found behavior that trips a
newcomer. Re-checked on `b197544e` (v0.2.0 + #291):

1. `orcapod.enable_file_hash_caching` was not exported. **Fix:** re-exported from `orcapod`.
2. `DictSource(data=[], tag_columns=[...])` raised `tag_columns not found in table: {...}.
Available columns: []`. **Fix:** `SourceStreamBuilder.build` raises `Table is empty` for
a table with no columns; a zero-row source with `data_schema=` still builds.
3. A pod applied to a stream with an extra column printed `Key 'age' not found in parameter
types.` (a `logger.warning` reaching logging's last-resort stderr handler) before raising
`Incoming data data type ...`. **Fix:** `schema_utils.schema_incompatibility` returns the
reason instead of logging; `FunctionPod` and `FunctionNode` raise `Incoming data schema
... : column 'age' is not a parameter of the function`.
4. Defaults that disagree: `@function_pod` `version="v0.0"` vs `FunctionPod.from_fn` /
`side_effect_function_pod` `"v1.0"` (the major version is in the data function's `uri`, so
it moves identity hashes and the result-store path); `MapData`/`MapTags`
`drop_unmapped=False` vs `stream.map_data`/`map_tags` `True` (in the operator identity).
**Open:** awaiting the developer's choice of default.
5. `examples/save_and_load_pipelines.py` passed a database to `Pipeline()` and failed.
**Fix:** rewritten around `PipelineJob`; `tests/test_examples/test_examples.py` runs every
example in a subprocess so an example cannot break silently again.
6. Noise: the protobuf gencode `UserWarning`s on import no longer reproduce (every
`orcapod` module imported with and without all extras, `-W always`; protobuf is never
loaded), so that half is closed. deltalake's `WARN deltalake_core::writer::stats Skipping
column ... because it's a binary field` lines on every Delta write still appear.
**Open:** awaiting the developer's choice of how to silence them.
7 changes: 4 additions & 3 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,10 @@ except ValueError as err:
# ValueError
```

The error message names both schemas, for example `... Schema({'weight': float, 'height':
float, 'age': int}) is not compatible with expected input schema Schema({'weight': float,
'height': float})`.
The error message names both schemas and the first column that does not fit, for example
`Incoming data schema Schema({'weight': float, 'height': float, 'age': int}) is not
compatible with expected input schema Schema({'weight': float, 'height': float}): column
'age' is not a parameter of the function`.

Narrow the stream first with the `SelectDataColumns` [operator](operators.md) (or drop the
extras with `DropDataColumns`):
Expand Down
4 changes: 4 additions & 0 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ Two construction rules apply to every source and are the most common first error
- At least one column must be left over as a data column, otherwise it raises
`ValueError: No data columns found in the table`.

Data with no rows has no columns to read, so `DictSource(data=[], ...)` raises
`ValueError: Table is empty`. To build a source with no rows, pass `data_schema=` so the
columns are known.

### `DerivedSource`

A `DerivedSource` reads the stored results of a node of a
Expand Down
51 changes: 32 additions & 19 deletions examples/save_and_load_pipelines.py
Original file line number Diff line number Diff line change
@@ -1,34 +1,47 @@
from orcapod import Pipeline, function_pod
from orcapod import sources, databases
"""Save a pipeline job to JSON, then load it back to inspect its stored results.

database = databases.DeltaTableDatabase("./local_database")
source1 = sources.DictSource(
Run from any directory: the database and the saved job are written to the
current working directory.
"""

from orcapod import PipelineJob, function_pod
from orcapod.databases import DeltaTableDatabase
from orcapod.sources import DictSource

database = DeltaTableDatabase("./local_database")

source1 = DictSource(
[{"id": 0, "x": 5}, {"id": 1, "x": 10}, {"id": 2, "x": 15}],
tag_columns=["id"],
label="source1",
source_id="source1",
)
source1 = source1.cached(database)
source2 = sources.DictSource(
source2 = DictSource(
[{"id": 0, "y": 3}, {"id": 2, "y": 6}, {"id": 4, "y": 9}],
tag_columns=["id"],
label="source2",
source_id="source2",
)
source2 = source2.cached(database)


pipeline = Pipeline("my_pipeline", database)


@function_pod("sum")
@function_pod(output_keys="sum")
def take_sum(x: int, y: int) -> int:
return x + y


with pipeline:
result = take_sum.pod(source1.join(source2))

path = "./pipeline.json"
pipeline.save(path)
# Record the pipeline: nothing is computed inside the ``with`` block.
job = PipelineJob(name="my_pipeline", store=database)
with job:
take_sum.pod(source1.join(source2), label="sum")

Check warning on line 33 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Result of call expression is of type "StreamProtocol" and is not used; assign to variable "_" if this is intentional (reportUnusedCallResult)

job.run()

Check warning on line 35 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Result of call expression is of type "PipelineJob" and is not used; assign to variable "_" if this is intentional (reportUnusedCallResult)
print(job.nodes["sum"].as_table().to_pandas())

Check warning on line 36 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Argument type is Any   Argument corresponds to parameter "values" in function "print" (reportAny)

Check warning on line 36 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "to_pandas" is Any (reportAny)

Check warning on line 36 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "as_table" is Any (reportAny)

pipeline2 = Pipeline.load(path)
# Save the job (pipeline topology and store configuration) to JSON ...
path = "./pipeline.json"
job.save(path)

# ... and load it back, for example in a later session, to inspect the stored
# results. A loaded job is read-only: its nodes read the records in the store
# with get_all_records(), and the job cannot run again. In-memory sources such
# as DictSource are not saved, so loading logs that they were skipped.
job2 = PipelineJob.load(path)
print(job2.nodes["sum"].get_all_records().to_pandas())

Check warning on line 47 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Argument type is Any   Argument corresponds to parameter "values" in function "print" (reportAny)

Check warning on line 47 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "to_pandas" is Any (reportAny)

Check warning on line 47 in examples/save_and_load_pipelines.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "get_all_records" is Any (reportAny)
2 changes: 2 additions & 0 deletions src/orcapod/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
OrcapodConfig,
load_config,
)
from .contexts import enable_file_hash_caching
from .core.function_pod import (
FunctionPod,
function_pod,
Expand Down Expand Up @@ -78,6 +79,7 @@
"HashingConfig",
"OrcapodConfig",
"load_config",
"enable_file_hash_caching",
"FunctionPod",
"function_pod",
"InvocationContext",
Expand Down
8 changes: 4 additions & 4 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -240,12 +240,12 @@ def _validate_input_schema(self, input_schema: Schema) -> None:
# check_schema_compatibility uses beartype.door.is_subhint which accepts
# any concrete branch: is_subhint(str, str | Path) → True,
# is_subhint(int, str | Path) → False.
if not schema_utils.check_schema_compatibility(
input_schema, expected_data_schema
):
reason = schema_utils.schema_incompatibility(input_schema, expected_data_schema)
if reason is not None:
# TODO: use custom exception type for better error handling
raise ValueError(
f"Incoming data data type {input_schema} is not compatible with expected input schema {expected_data_schema}"
f"Incoming data schema {input_schema} is not compatible with "
f"expected input schema {expected_data_schema}: {reason}"
)

def process_data(
Expand Down
10 changes: 6 additions & 4 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,12 +186,14 @@ def __init__(
# Use the pod's exposed input schema (ctx excluded) rather than the
# data function's full schema, because ctx is auto-injected per row.
expected_data_schema = function_pod.input_data_schema
if not schema_utils.check_schema_compatibility(
reason = schema_utils.schema_incompatibility(
incoming_data_types, expected_data_schema
):
)
if reason is not None:
raise ValueError(
f"Incoming data data type {incoming_data_types} from {input_stream} "
f"is not compatible with expected input schema {expected_data_schema}"
f"Incoming data schema {incoming_data_types} from {input_stream} "
f"is not compatible with expected input schema "
f"{expected_data_schema}: {reason}"
)

self._input_stream = input_stream
Expand Down
13 changes: 11 additions & 2 deletions src/orcapod/core/sources/stream_builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,15 +146,24 @@ def build(
SourceStreamResult with enriched stream and metadata.

Raises:
ValueError: If tag_columns or record_id_column are not in table.
ValueError: If the table has no columns ("Table is empty"), or if
tag_columns or record_id_column are not in the table.
"""
tag_columns_tuple = tuple(tag_columns)
system_tag_columns_tuple = tuple(system_tag_columns)

# 1. Drop system columns from raw input.
table = arrow_utils.drop_system_columns(table)

# 2. Validate tag_columns.
# 2. Validate tag_columns. A table with no columns at all (e.g. a
# DictSource built from no rows and no schema) gets its own error, since
# "tag column not found" would point at the wrong problem.
if not table.column_names:
raise ValueError(
"Table is empty: the input has no columns, so its tag and data "
"columns cannot be found. To build a source with no rows, give it "
"an explicit schema (for example data_schema= on DictSource)."
)
missing_tags = set(tag_columns_tuple) - set(table.column_names)
if missing_tags:
raise ValueError(
Expand Down
54 changes: 46 additions & 8 deletions src/orcapod/utils/schema_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,25 +39,63 @@ def verify_data_schema(data: dict, schema: SchemaLike) -> bool:
def check_schema_compatibility(
incoming_types: SchemaLike, receiving_types: Schema
) -> bool:
"""Return whether ``incoming_types`` can be passed to ``receiving_types``.

See ``schema_incompatibility`` for the rules and for the reason when the
answer is no.

Args:
incoming_types: Schema of the data offered (e.g. a stream's data schema).
receiving_types: Schema of the parameters that receive it.

Returns:
True if every incoming column is a parameter of a compatible type and
every required parameter has a column.
"""
return schema_incompatibility(incoming_types, receiving_types) is None


def _type_label(type_info: Any) -> str:
"""Return a short, readable label for a type in an error message."""
if isinstance(type_info, type) and not get_args(type_info):
return type_info.__name__
return str(type_info)


def schema_incompatibility(
incoming_types: SchemaLike, receiving_types: Schema
) -> str | None:
"""Explain why ``incoming_types`` cannot be passed to ``receiving_types``.

Every incoming column must be a parameter, its type must be a subhint of
the parameter's type, and every parameter without a default must have a
column. Nothing is logged: the caller decides how to report the reason.

Args:
incoming_types: Schema of the data offered (e.g. a stream's data schema).
receiving_types: Schema of the parameters that receive it.

Returns:
``None`` when the schemas are compatible, otherwise a one-line reason
naming the first offending column or parameter.
"""
from beartype.door import is_subhint

for key, type_info in incoming_types.items():
if key not in receiving_types:
logger.warning(f"Key '{key}' not found in parameter types.")
return False
return f"column '{key}' is not a parameter of the function"
if not is_subhint(type_info, receiving_types[key]):
logger.warning(
f"Type mismatch for key '{key}': expected {receiving_types[key]}, got {type_info}."
return (
f"column '{key}' has type {_type_label(type_info)}, "
f"the function expects {_type_label(receiving_types[key])}"
)
return False

# Every receiving key must be present in incoming OR be optional (has a default)
for key in receiving_types:
if key not in incoming_types and key not in receiving_types.optional_fields:
logger.warning(f"Required key '{key}' missing from incoming types.")
return False
return f"required parameter '{key}' has no matching column"

return True
return None


def extract_function_schemas(
Expand Down
Loading
Loading