Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
d1e7c0e
test(pipeline): failing regression tests for swallowed store write fa…
Nov 11, 2026
4bbb24c
fix(pipeline): a failed store write fails the run (ITL-878)
Nov 11, 2026
e0a472a
docs: state the store-write failure contract (ITL-878)
Nov 11, 2026
852ca05
fix(pipeline): keep a store write error intact on its way out (ITL-878)
Nov 11, 2026
004afd9
Merge remote-tracking branch 'origin/main' into eywalker/itl-878-pipe…
Nov 11, 2026
2770848
docs(pipeline): docstrings for the store-write-failure tests and Stor…
Nov 11, 2026
0940f99
style(pipeline): unquote two annotations ruff UP037 flags (ITL-878)
Nov 11, 2026
ef78e76
Merge remote-tracking branch 'origin/main' into eywalker/itl-878-pipe…
Nov 11, 2026
66686a9
docs(plan): CodeRabbit round 2 out-of-diff findings (ITL-878)
Nov 11, 2026
9200727
fix(pipeline): note unstored pending writes on a failed run's error (…
Nov 11, 2026
ff4c25c
test(pipeline): an unstorable output is reported like a function fail…
Nov 11, 2026
fb15f96
test(pipeline): an unstorable output is logged like a function failur…
Nov 11, 2026
4addbdf
docs(pipeline): an output that cannot be stored is a row failure (ITL…
Nov 11, 2026
70d618e
Merge remote-tracking branch 'origin/main' into eywalker/itl-878-pipe…
Nov 11, 2026
92f4bef
docs(changelog): StoreWriteError and the store-write failure contract…
Nov 11, 2026
d305c80
style(pipeline): ruff format the row-failure pinning test (ITL-878)
Nov 11, 2026
392ee8e
fix(pipeline): attribute a failed flush to the store, not the row; qu…
Nov 11, 2026
0894556
docs(plan): CodeRabbit round 3 done (ITL-878)
Nov 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,27 @@
# Changelog

## [0.3.0] — unreleased

### Added

- **`orcapod.errors.StoreWriteError`** — raised when a node's results or records cannot be
written to the store. It names the node, the row's tag and the table, and chains the
database's own exception as its cause. (ITL-878 #287)

### Fixed

- A failed store write fails the run. `PipelineJob.run()` used to return normally when a
node's result or record could not be written (the row was skipped like a crashed function
under the default `error_policy="continue"`), so the node read back fewer rows than it
computed. It now raises `StoreWriteError` at the first failed write, whatever
`error_policy` is, on both orchestrators (the async one no longer raises an exception
group). Rows stored before the failure are committed and kept; re-running does not
recompute stored function results. A failed flush of pending writes raises
`StoreWriteError` with `flush=True` and names the store, not a row. A function output that
cannot be turned into a record (a naive `datetime`, a value not matching the declared
type) is still a row failure governed by `error_policy`, like a function that raised. See
*When the store cannot be written* in the function-pods guide. (ITL-878 #287)

## [0.2.0] — 2026-11-10

**Distribution:** Orcapod is MIT licensed starting with `0.2.0` (it was proprietary on
Expand Down
48 changes: 48 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2211,6 +2211,54 @@ value needs a migration assessment and the developer's approval. Not pinned yet:

---

## `src/orcapod/pipeline/job.py`, orchestrators, node write paths — store write failures

### SW1 — A failed store write did not fail `PipelineJob.run()`
**Status:** resolved
**Severity:** critical
**Issue:** ITL-878

A node's write to the store ran inside the per-row handler meant for function failures:
`FunctionNode.execute` (sync) and `_FunctionPodBase.async_execute.process_one` (async) wrap
compute *and* the result-table and pipeline-table writes in one `except Exception`. With the
default `error_policy="continue"` a failed write was skipped like a crashed function: the sync
path logged a WARNING, the async path logged only at DEBUG, `job.run()` returned normally and
the node read back fewer rows (none when every write failed, e.g. a logical-type output on
`SQLiteConnector`, ITL-877). An operator LOG write failure did raise, but before `job.run()`'s
final flush, so the upstream function node's pending pipeline rows were never committed while
its result rows were; async callers got an `ExceptionGroup`.

**Fix:** Contract, for now (developer, November 10, 2026): a failed store write always fails
the run, whatever `error_policy` is. Every node write site (`ResultCache.store`,
`FunctionNode.add_pipeline_record`, `OperatorNode._store_output_stream`, the side-effect
invocation log) raises `orcapod.errors.StoreWriteError`, chained from the backend error and
naming the node, the row's tag and the table. The per-row handlers re-raise it, so the run
stops at the first failed write; `error_policy` still decides only about rows whose function
raised. `AsyncPipelineOrchestrator` unwraps it from the `TaskGroup` exception group, so both
orchestrators raise the same error. `PipelineJob.run()` flushes the writes that succeeded
before re-raising (a flush failure there is logged, not raised over the original), and wraps a
failure of its final flush. Nothing stored is rewritten; no hash, path or schema change. An
opt-in mode that tolerates store failures is a possible later option. A fail-loud post-run
hook or an observer callback that raises while the store error propagates is logged and noted
on it (`orcapod.utils.store_errors`), not raised in its place, so a node function that runs
another cached pod cannot turn that pod's store failure into a skipped row. When concurrent
node tasks fail together, the async orchestrator raises the store error of the node first in
topological order, notes the other failures on it and logs other store errors at WARNING.
A failed flush (the result cache's per-row auto-flush or the job's final flush) raises
`StoreWriteError(flush=True)` naming the store and no row: views of one store share their
pending writes, so the failing batch may be an earlier row's or another table's.
The rule (developer, November 11, 2026): store failures fail the run; a function output that
cannot be stored (a naive `datetime`, a value not matching the declared type, rejected while
`ResultCache.store` prepares the record, before any write) is a row failure governed by
`error_policy` exactly like a function that raised, not a `StoreWriteError`: the value is
specific to that row, the store is healthy, and every re-run would fail at the same row. The
async orchestrator logs row failures at DEBUG and ignores `fail_fast` (ITL-879), unchanged here.
Regressions: `tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async
orchestrator ignores `error_policy` (ITL-879); connector and Delta `flush` drop a batch whose
write fails (ITL-880).

---

## `src/orcapod/core/operators/semijoin.py` — nested columns and string/large_string keys

### SJ1 — `SemiJoin` failed on nested columns and on `string` vs `large_string` keys
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,10 @@ While the following is subject to change based on future development, it represe
│ on success:
│ ctx_observer.on_data_end(node_label, tag, data, result, cached=False)
│ emit (tag_out, result) downstream
│ on exception:
│ on StoreWriteError (a result or record could not be stored):
│ ctx_observer.on_data_crash(node_label, tag, data, exc)
│ raise, whatever error_policy is — the run stops (ITL-878)
│ on any other exception (the function failed):
│ ctx_observer.on_data_crash(node_label, tag, data, exc)
│ if error_policy == "fail_fast": raise
│ otherwise: skip this data, continue
Expand Down
46 changes: 46 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,52 @@ A node from a job that has run also provides:
- **`as_source()`** -- a `DerivedSource` backed by those stored records, usable as the input
of another pipeline

### When the store cannot be written

A failed write to the store always fails the run. If a node's results or records cannot be
stored -- the disk is full, the connection drops, or the database cannot hold one of the
output types -- `job.run()` raises `StoreWriteError` (from `orcapod.errors`) at the first
failed write, on both the synchronous and the asynchronous orchestrator. The error names the
node, the row being stored and the table, and the database's own exception is chained as its
cause. When committing pending writes fails, the error has `flush=True` and names the store
instead of a row and table: every view of a store shares its pending writes, so the write
that failed may belong to an earlier row or another table. A run that returns normally has
stored every result that could be turned into a record; a result rejected before that is a
row failure (see below).

This holds whatever `error_policy` the orchestrator was given. `error_policy="continue"` (the
default) skips rows whose *function* raised and keeps going; it never skips a row because it
could not be stored.

The error you get is always one plain `StoreWriteError`, never an exception group and never
another exception raised in its place. A post-run hook or observer callback that fails while
the store error is on its way out is logged and recorded as a note on the error
(`err.__notes__`). When several nodes fail at once on the asynchronous orchestrator, you get
the store error of the node that comes first in the pipeline, the one the synchronous
orchestrator would have stopped at; the other failures are logged and noted on it.

After the failure, rows stored before it are kept: `job.run()` commits the writes that
succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is
fixed, run the job again: stored function results are not recomputed, only the missing ones
are. An operator with `cache_mode=LOG` recomputes its output on every run, as it always does.
A side effect is different: if its delivery completed but the write to its invocation log
failed, there is no record that it completed, and `track_completion` delivers it again on the
re-run. Before re-running, check which side effects already completed, or make the side
effect safe to repeat.

A value your function returned for one row that cannot be turned into a record at all is a
different case: a naive `datetime`, a `str` from a function declared to return `int`, or an
`int` where a dataclass is declared. Such a value is rejected while the row's record is being
prepared, before anything is written, and it is a *row failure*: it is handled exactly like a
function that raised on that row. The observer gets `on_data_crash`, the failure is logged,
and `error_policy` decides -- `"continue"` skips the row, `"fail_fast"` raises the error
the conversion raised. It is not a `StoreWriteError`, because the store is healthy and every
re-run would fail at the same row. The asynchronous orchestrator currently logs row failures
at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these rows too.

This is the contract for now. An opt-in mode that tolerates store failures may be added
later.

## Multiple input streams

If you pass multiple streams to a function pod, they are automatically joined (using
Expand Down
19 changes: 19 additions & 0 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
from orcapod.core.data_function import CachedDataFunction, PythonDataFunction
from orcapod.core.streams.base import StreamBase
from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER
from orcapod.errors import StoreWriteError
from orcapod.utils.store_errors import keep_store_error, note_callback_failure
from orcapod.protocols.core_protocols import (
ArgumentGroup,
FunctionPodProtocol,
Expand Down Expand Up @@ -327,16 +329,26 @@
def _fire_post_run_hooks(self, payload: PostRunPayload) -> None:
"""Fire all registered hooks with payload in registration order.

When the invocation failed with a ``StoreWriteError``, a fail-loud hook that
raises does not replace it: its failure is logged and noted on the store
error, the remaining hooks are skipped as for any fail-loud failure, and the
caller re-raises the store error.

Args:
payload: The post-run payload to pass to each hook.
"""
error = payload.stats.error
store_error = error if isinstance(error, StoreWriteError) else None
for hook in self._post_run_hooks:
fn = hook.fn if isinstance(hook, HookConfig) else hook
on_error = hook.on_error if isinstance(hook, HookConfig) else "raise"
try:
fn(payload)
except Exception as exc:
if on_error == "raise":
if store_error is not None:
note_callback_failure(store_error, f"Post-run hook {fn!r}", exc)
return
raise
logger.warning(
"Post-run hook %r raised and was suppressed: %s",
Expand Down Expand Up @@ -587,6 +599,13 @@
out_tag, result_data = await self._async_invoke_with_hooks(
tag, data, logger=pkt_logger, run_id=run_id
)
except StoreWriteError as exc:
# A failed store write fails the run; only rows whose
# function raised are skipped.
exc.add_context(tag=tag.as_dict())

Check warning on line 605 in src/orcapod/core/function_pod.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Argument type is partially unknown   Argument corresponds to parameter "tag" in function "add_context"   Argument type is "dict[str, str | int | float | bool | bytes | date | datetime | PathLike[Unknown] | Collection[str | PathLike[Unknown] | None] | Collection[DataValue] | None]" (reportUnknownArgumentType)

Check warning on line 605 in src/orcapod/core/function_pod.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of "as_dict" is partially unknown   Type of "as_dict" is "(*, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False) -> dict[str, str | int | float | bool | bytes | date | datetime | PathLike[Unknown] | Collection[str | PathLike[Unknown] | None] | Collection[DataValue] | None]" (reportUnknownMemberType)
with keep_store_error(exc, "Observer on_data_crash"):
obs.on_data_crash(pod_label, tag, data, exc)
raise
except Exception as exc:
logger.debug(
"Data processing failed, skipping: %s", exc, exc_info=True
Expand Down Expand Up @@ -679,7 +698,7 @@
version: str = "v1.0",
pod_config: PodConfig | None = None,
label: str | None = None,
**kwargs: Any,

Check warning on line 701 in src/orcapod/core/function_pod.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type `Any` is not allowed (reportExplicitAny)

Check warning on line 701 in src/orcapod/core/function_pod.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Type of parameter "kwargs" is Any (reportAny)
) -> "FunctionPod":
"""Construct a ``FunctionPod`` directly from a callable.

Expand Down Expand Up @@ -715,7 +734,7 @@
pod_config=pod_config,
ctx_arg_name=ctx_arg_name,
label=label,
**kwargs,

Check warning on line 737 in src/orcapod/core/function_pod.py

View workflow job for this annotation

GitHub Actions / typecheck / typecheck

Argument type is Any   Argument corresponds to parameter "kwargs" in function "__init__" (reportAny)
)

@property
Expand Down
49 changes: 39 additions & 10 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@
EphemeralResultMissingError,
PipelineJobRequiredError,
SchemaVersionError,
StoreWriteError,
)
from orcapod.utils.store_errors import keep_store_error
from orcapod.protocols.core_protocols import (
FunctionPodProtocol,
DataFunctionExecutorProtocol,
Expand Down Expand Up @@ -1164,21 +1166,28 @@ def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None:
# Internal helpers
# ------------------------------------------------------------------

def _require_pipeline_database(self) -> None:
"""Raise a clear RuntimeError if no pipeline database is attached.
def _require_pipeline_database(self) -> ArrowDatabaseProtocol:
"""Return the attached pipeline database, raising if there is none.

Called at the top of methods that unconditionally access
``self._pipeline_database``. Provides an actionable error message
instead of an opaque ``AttributeError: 'NoneType' object has no
attribute ...`` when a definition-level pipeline is executed without
supplying a database.

Returns:
The attached pipeline database.

Raises:
RuntimeError: If no pipeline database is attached.
"""
if self._pipeline_database is None:
raise RuntimeError(
f"FunctionJobNode '{self.label}' has no pipeline database attached. "
"Either construct the pipeline with a pipeline_database argument, "
"or supply one via Pipeline.load(..., pipeline_database=<db>)."
)
return self._pipeline_database

# ------------------------------------------------------------------
# as_node — return the lightweight FunctionNode equivalent
Expand Down Expand Up @@ -1294,6 +1303,15 @@ def execute(
tag_out, result = self._process_data_internal(
tag, data, logger=pkt_logger, run_id=run_id
)
except StoreWriteError as exc:
# A failed store write fails the run whatever the error policy:
# error_policy only decides about rows whose function raised.
exc.add_context(node_label=node_label, tag=tag.as_dict())
with keep_store_error(exc, "Observer on_data_crash"):
ctx_obs.on_data_crash(node_label, tag, data, exc)
with keep_store_error(exc, "Observer on_node_end"):
ctx_obs.on_node_end(node_label, node_hash)
raise
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
except Exception as exc:
logger.warning(
"Data execution failed in %s: %s",
Expand Down Expand Up @@ -1701,14 +1719,14 @@ def add_pipeline_record(
so that downstream ``EmptyData`` flow-through can locate its cached
result keyed by the same hash (= the downstream's ``INPUT_DATA_HASH_COL``).
"""
self._require_pipeline_database()
pipeline_database = self._require_pipeline_database()
self._ensure_pdb_schema()
base_entry_id = self.compute_base_entry_id(tag, input_data)

# Determine the next recomputation index by querying all existing rows
# for this base_entry_id. No await is used here, so within a single-threaded
# asyncio event loop this read-then-write sequence is uninterrupted.
existing = self._pipeline_database.get_records_with_column_value(
existing = pipeline_database.get_records_with_column_value(
self._versioned_pipeline_path,
{_PIPELINE_BASE_ENTRY_ID_COL: base_entry_id},
)
Expand Down Expand Up @@ -1773,12 +1791,19 @@ def add_pipeline_record(
meta_table,
)

self._pipeline_database.add_record(
self._versioned_pipeline_path,
versioned_entry_id,
combined_record,
skip_duplicates=True,
)
try:
pipeline_database.add_record(
self._versioned_pipeline_path,
versioned_entry_id,
combined_record,
skip_duplicates=True,
)
except Exception as exc:
raise StoreWriteError(
self._versioned_pipeline_path,
node_label=self.label,
tag=tag.as_dict(),
) from exc

# ------------------------------------------------------------------
# Records and sources
Expand Down Expand Up @@ -2517,6 +2542,10 @@ def on_data_crash(
clean = tag.drop_meta_columns(
_TAG_NODE_INPUT_REF, ignore_missing=True
)
if isinstance(error, StoreWriteError):
# The pod re-raises a failed store write after this
# call; name the node it belongs to.
error.add_context(node_label=node_label)
ctx_obs.on_data_crash(node_label, clean, data, error)
# Clean up the dangling input_store entry so it doesn't
# accumulate memory during long-running calls with errors.
Expand Down
24 changes: 17 additions & 7 deletions src/orcapod/core/nodes/operator_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
from orcapod.core.streams.arrow_table_stream import ArrowTableStream
from orcapod.core.streams.base import StreamBase
from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER
from orcapod.errors import PipelineJobRequiredError
from orcapod.errors import PipelineJobRequiredError, StoreWriteError
from orcapod.protocols.core_protocols import (
DataProtocol,
StreamProtocol,
Expand Down Expand Up @@ -798,6 +798,11 @@ def _filter_by_content_hash(self, table: pa.Table) -> pa.Table:

def _store_output_stream(self, stream: StreamProtocol) -> None:
"""Materialize stream and store in the pipeline database with per-row dedup."""
pipeline_database = self._pipeline_database
if pipeline_database is None:
raise RuntimeError(
f"OperatorJobNode {self.label!r} has no pipeline database attached."
)
output_table = stream.as_table(
columns={"source": True, "system_tags": True},
)
Expand Down Expand Up @@ -839,12 +844,17 @@ def _store_output_stream(self, stream: StreamProtocol) -> None:
# the hash), so rows from different runs with identical output will have
# distinct record IDs and both be stored. skip_duplicates=True still
# deduplicates exact re-runs of the same node within a single run.
self._pipeline_database.add_records(
self.node_identity_path,
output_table,
record_id_column=self.HASH_COLUMN_NAME,
skip_duplicates=True,
)
try:
pipeline_database.add_records(
self.node_identity_path,
output_table,
record_id_column=self.HASH_COLUMN_NAME,
skip_duplicates=True,
)
except Exception as exc:
raise StoreWriteError(
self.node_identity_path, node_label=self.label
) from exc

self._cached_output_table = output_table.drop(
[self.HASH_COLUMN_NAME, constants.NODE_CONTENT_HASH_COL]
Expand Down
Loading
Loading