diff --git a/CHANGELOG.md b/CHANGELOG.md index 8053fbb77..ffc35f547 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # Changelog +## [0.3.0] — unreleased + +### Added + +- **`orcapod.errors.StoreWriteError`** — raised when a node's results or records cannot be + written to the store. It names the node, the row's tag and the table, and chains the + database's own exception as its cause. (ITL-878 #287) + +### Fixed + +- A failed store write fails the run. `PipelineJob.run()` used to return normally when a + node's result or record could not be written (the row was skipped like a crashed function + under the default `error_policy="continue"`), so the node read back fewer rows than it + computed. It now raises `StoreWriteError` at the first failed write, whatever + `error_policy` is, on both orchestrators (the async one no longer raises an exception + group). Rows stored before the failure are committed and kept; re-running does not + recompute stored function results. A failed flush of pending writes raises + `StoreWriteError` with `flush=True` and names the store, not a row. A function output that + cannot be turned into a record (a naive `datetime`, a value not matching the declared + type) is still a row failure governed by `error_policy`, like a function that raised. See + *When the store cannot be written* in the function-pods guide. (ITL-878 #287) + ## [0.2.0] — 2026-11-10 **Distribution:** Orcapod is MIT licensed starting with `0.2.0` (it was proprietary on diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 4be0e7e34..ba4c26c34 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2211,6 +2211,54 @@ value needs a migration assessment and the developer's approval. Not pinned yet: --- +## `src/orcapod/pipeline/job.py`, orchestrators, node write paths — store write failures + +### SW1 — A failed store write did not fail `PipelineJob.run()` +**Status:** resolved +**Severity:** critical +**Issue:** ITL-878 + +A node's write to the store ran inside the per-row handler meant for function failures: +`FunctionNode.execute` (sync) and `_FunctionPodBase.async_execute.process_one` (async) wrap +compute *and* the result-table and pipeline-table writes in one `except Exception`. With the +default `error_policy="continue"` a failed write was skipped like a crashed function: the sync +path logged a WARNING, the async path logged only at DEBUG, `job.run()` returned normally and +the node read back fewer rows (none when every write failed, e.g. a logical-type output on +`SQLiteConnector`, ITL-877). An operator LOG write failure did raise, but before `job.run()`'s +final flush, so the upstream function node's pending pipeline rows were never committed while +its result rows were; async callers got an `ExceptionGroup`. + +**Fix:** Contract, for now (developer, November 10, 2026): a failed store write always fails +the run, whatever `error_policy` is. Every node write site (`ResultCache.store`, +`FunctionNode.add_pipeline_record`, `OperatorNode._store_output_stream`, the side-effect +invocation log) raises `orcapod.errors.StoreWriteError`, chained from the backend error and +naming the node, the row's tag and the table. The per-row handlers re-raise it, so the run +stops at the first failed write; `error_policy` still decides only about rows whose function +raised. `AsyncPipelineOrchestrator` unwraps it from the `TaskGroup` exception group, so both +orchestrators raise the same error. `PipelineJob.run()` flushes the writes that succeeded +before re-raising (a flush failure there is logged, not raised over the original), and wraps a +failure of its final flush. Nothing stored is rewritten; no hash, path or schema change. An +opt-in mode that tolerates store failures is a possible later option. A fail-loud post-run +hook or an observer callback that raises while the store error propagates is logged and noted +on it (`orcapod.utils.store_errors`), not raised in its place, so a node function that runs +another cached pod cannot turn that pod's store failure into a skipped row. When concurrent +node tasks fail together, the async orchestrator raises the store error of the node first in +topological order, notes the other failures on it and logs other store errors at WARNING. +A failed flush (the result cache's per-row auto-flush or the job's final flush) raises +`StoreWriteError(flush=True)` naming the store and no row: views of one store share their +pending writes, so the failing batch may be an earlier row's or another table's. +The rule (developer, November 11, 2026): store failures fail the run; a function output that +cannot be stored (a naive `datetime`, a value not matching the declared type, rejected while +`ResultCache.store` prepares the record, before any write) is a row failure governed by +`error_policy` exactly like a function that raised, not a `StoreWriteError`: the value is +specific to that row, the store is healthy, and every re-run would fail at the same row. The +async orchestrator logs row failures at DEBUG and ignores `fail_fast` (ITL-879), unchanged here. +Regressions: `tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async +orchestrator ignores `error_policy` (ITL-879); connector and Delta `flush` drop a batch whose +write fails (ITL-880). + +--- + ## `src/orcapod/core/operators/semijoin.py` — nested columns and string/large_string keys ### SJ1 — `SemiJoin` failed on nested columns and on `string` vs `large_string` keys diff --git a/README.md b/README.md index 32f7190b8..29aeb992f 100644 --- a/README.md +++ b/README.md @@ -238,7 +238,10 @@ While the following is subject to change based on future development, it represe │ on success: │ ctx_observer.on_data_end(node_label, tag, data, result, cached=False) │ emit (tag_out, result) downstream - │ on exception: + │ on StoreWriteError (a result or record could not be stored): + │ ctx_observer.on_data_crash(node_label, tag, data, exc) + │ raise, whatever error_policy is — the run stops (ITL-878) + │ on any other exception (the function failed): │ ctx_observer.on_data_crash(node_label, tag, data, exc) │ if error_policy == "fail_fast": raise │ otherwise: skip this data, continue diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index b8618052c..61ea20fca 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -228,6 +228,52 @@ A node from a job that has run also provides: - **`as_source()`** -- a `DerivedSource` backed by those stored records, usable as the input of another pipeline +### When the store cannot be written + +A failed write to the store always fails the run. If a node's results or records cannot be +stored -- the disk is full, the connection drops, or the database cannot hold one of the +output types -- `job.run()` raises `StoreWriteError` (from `orcapod.errors`) at the first +failed write, on both the synchronous and the asynchronous orchestrator. The error names the +node, the row being stored and the table, and the database's own exception is chained as its +cause. When committing pending writes fails, the error has `flush=True` and names the store +instead of a row and table: every view of a store shares its pending writes, so the write +that failed may belong to an earlier row or another table. A run that returns normally has +stored every result that could be turned into a record; a result rejected before that is a +row failure (see below). + +This holds whatever `error_policy` the orchestrator was given. `error_policy="continue"` (the +default) skips rows whose *function* raised and keeps going; it never skips a row because it +could not be stored. + +The error you get is always one plain `StoreWriteError`, never an exception group and never +another exception raised in its place. A post-run hook or observer callback that fails while +the store error is on its way out is logged and recorded as a note on the error +(`err.__notes__`). When several nodes fail at once on the asynchronous orchestrator, you get +the store error of the node that comes first in the pipeline, the one the synchronous +orchestrator would have stopped at; the other failures are logged and noted on it. + +After the failure, rows stored before it are kept: `job.run()` commits the writes that +succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is +fixed, run the job again: stored function results are not recomputed, only the missing ones +are. An operator with `cache_mode=LOG` recomputes its output on every run, as it always does. +A side effect is different: if its delivery completed but the write to its invocation log +failed, there is no record that it completed, and `track_completion` delivers it again on the +re-run. Before re-running, check which side effects already completed, or make the side +effect safe to repeat. + +A value your function returned for one row that cannot be turned into a record at all is a +different case: a naive `datetime`, a `str` from a function declared to return `int`, or an +`int` where a dataclass is declared. Such a value is rejected while the row's record is being +prepared, before anything is written, and it is a *row failure*: it is handled exactly like a +function that raised on that row. The observer gets `on_data_crash`, the failure is logged, +and `error_policy` decides -- `"continue"` skips the row, `"fail_fast"` raises the error +the conversion raised. It is not a `StoreWriteError`, because the store is healthy and every +re-run would fail at the same row. The asynchronous orchestrator currently logs row failures +at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these rows too. + +This is the contract for now. An opt-in mode that tolerates store failures may be added +later. + ## Multiple input streams If you pass multiple streams to a function pod, they are automatically joined (using diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index ba7e3b1d4..e716f367d 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -14,6 +14,8 @@ from orcapod.core.data_function import CachedDataFunction, PythonDataFunction from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER +from orcapod.errors import StoreWriteError +from orcapod.utils.store_errors import keep_store_error, note_callback_failure from orcapod.protocols.core_protocols import ( ArgumentGroup, FunctionPodProtocol, @@ -327,9 +329,16 @@ def add_post_run_hook(self, hook: PostRunHook) -> None: def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: """Fire all registered hooks with payload in registration order. + When the invocation failed with a ``StoreWriteError``, a fail-loud hook that + raises does not replace it: its failure is logged and noted on the store + error, the remaining hooks are skipped as for any fail-loud failure, and the + caller re-raises the store error. + Args: payload: The post-run payload to pass to each hook. """ + error = payload.stats.error + store_error = error if isinstance(error, StoreWriteError) else None for hook in self._post_run_hooks: fn = hook.fn if isinstance(hook, HookConfig) else hook on_error = hook.on_error if isinstance(hook, HookConfig) else "raise" @@ -337,6 +346,9 @@ def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: fn(payload) except Exception as exc: if on_error == "raise": + if store_error is not None: + note_callback_failure(store_error, f"Post-run hook {fn!r}", exc) + return raise logger.warning( "Post-run hook %r raised and was suppressed: %s", @@ -587,6 +599,13 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: out_tag, result_data = await self._async_invoke_with_hooks( tag, data, logger=pkt_logger, run_id=run_id ) + except StoreWriteError as exc: + # A failed store write fails the run; only rows whose + # function raised are skipped. + exc.add_context(tag=tag.as_dict()) + with keep_store_error(exc, "Observer on_data_crash"): + obs.on_data_crash(pod_label, tag, data, exc) + raise except Exception as exc: logger.debug( "Data processing failed, skipping: %s", exc, exc_info=True diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 7af980049..394807b8a 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -35,7 +35,9 @@ EphemeralResultMissingError, PipelineJobRequiredError, SchemaVersionError, + StoreWriteError, ) +from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.core_protocols import ( FunctionPodProtocol, DataFunctionExecutorProtocol, @@ -1164,14 +1166,20 @@ def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: # Internal helpers # ------------------------------------------------------------------ - def _require_pipeline_database(self) -> None: - """Raise a clear RuntimeError if no pipeline database is attached. + def _require_pipeline_database(self) -> ArrowDatabaseProtocol: + """Return the attached pipeline database, raising if there is none. Called at the top of methods that unconditionally access ``self._pipeline_database``. Provides an actionable error message instead of an opaque ``AttributeError: 'NoneType' object has no attribute ...`` when a definition-level pipeline is executed without supplying a database. + + Returns: + The attached pipeline database. + + Raises: + RuntimeError: If no pipeline database is attached. """ if self._pipeline_database is None: raise RuntimeError( @@ -1179,6 +1187,7 @@ def _require_pipeline_database(self) -> None: "Either construct the pipeline with a pipeline_database argument, " "or supply one via Pipeline.load(..., pipeline_database=)." ) + return self._pipeline_database # ------------------------------------------------------------------ # as_node — return the lightweight FunctionNode equivalent @@ -1294,6 +1303,15 @@ def execute( tag_out, result = self._process_data_internal( tag, data, logger=pkt_logger, run_id=run_id ) + except StoreWriteError as exc: + # A failed store write fails the run whatever the error policy: + # error_policy only decides about rows whose function raised. + exc.add_context(node_label=node_label, tag=tag.as_dict()) + with keep_store_error(exc, "Observer on_data_crash"): + ctx_obs.on_data_crash(node_label, tag, data, exc) + with keep_store_error(exc, "Observer on_node_end"): + ctx_obs.on_node_end(node_label, node_hash) + raise except Exception as exc: logger.warning( "Data execution failed in %s: %s", @@ -1701,14 +1719,14 @@ def add_pipeline_record( so that downstream ``EmptyData`` flow-through can locate its cached result keyed by the same hash (= the downstream's ``INPUT_DATA_HASH_COL``). """ - self._require_pipeline_database() + pipeline_database = self._require_pipeline_database() self._ensure_pdb_schema() base_entry_id = self.compute_base_entry_id(tag, input_data) # Determine the next recomputation index by querying all existing rows # for this base_entry_id. No await is used here, so within a single-threaded # asyncio event loop this read-then-write sequence is uninterrupted. - existing = self._pipeline_database.get_records_with_column_value( + existing = pipeline_database.get_records_with_column_value( self._versioned_pipeline_path, {_PIPELINE_BASE_ENTRY_ID_COL: base_entry_id}, ) @@ -1773,12 +1791,19 @@ def add_pipeline_record( meta_table, ) - self._pipeline_database.add_record( - self._versioned_pipeline_path, - versioned_entry_id, - combined_record, - skip_duplicates=True, - ) + try: + pipeline_database.add_record( + self._versioned_pipeline_path, + versioned_entry_id, + combined_record, + skip_duplicates=True, + ) + except Exception as exc: + raise StoreWriteError( + self._versioned_pipeline_path, + node_label=self.label, + tag=tag.as_dict(), + ) from exc # ------------------------------------------------------------------ # Records and sources @@ -2517,6 +2542,10 @@ def on_data_crash( clean = tag.drop_meta_columns( _TAG_NODE_INPUT_REF, ignore_missing=True ) + if isinstance(error, StoreWriteError): + # The pod re-raises a failed store write after this + # call; name the node it belongs to. + error.add_context(node_label=node_label) ctx_obs.on_data_crash(node_label, clean, data, error) # Clean up the dangling input_store entry so it doesn't # accumulate memory during long-running calls with errors. diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index fa81a4469..f343ba67f 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -29,7 +29,7 @@ from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER -from orcapod.errors import PipelineJobRequiredError +from orcapod.errors import PipelineJobRequiredError, StoreWriteError from orcapod.protocols.core_protocols import ( DataProtocol, StreamProtocol, @@ -798,6 +798,11 @@ def _filter_by_content_hash(self, table: pa.Table) -> pa.Table: def _store_output_stream(self, stream: StreamProtocol) -> None: """Materialize stream and store in the pipeline database with per-row dedup.""" + pipeline_database = self._pipeline_database + if pipeline_database is None: + raise RuntimeError( + f"OperatorJobNode {self.label!r} has no pipeline database attached." + ) output_table = stream.as_table( columns={"source": True, "system_tags": True}, ) @@ -839,12 +844,17 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: # the hash), so rows from different runs with identical output will have # distinct record IDs and both be stored. skip_duplicates=True still # deduplicates exact re-runs of the same node within a single run. - self._pipeline_database.add_records( - self.node_identity_path, - output_table, - record_id_column=self.HASH_COLUMN_NAME, - skip_duplicates=True, - ) + try: + pipeline_database.add_records( + self.node_identity_path, + output_table, + record_id_column=self.HASH_COLUMN_NAME, + skip_duplicates=True, + ) + except Exception as exc: + raise StoreWriteError( + self.node_identity_path, node_label=self.label + ) from exc self._cached_output_table = output_table.drop( [self.HASH_COLUMN_NAME, constants.NODE_CONTENT_HASH_COL] diff --git a/src/orcapod/core/result_cache.py b/src/orcapod/core/result_cache.py index 970a22418..ec9bb788f 100644 --- a/src/orcapod/core/result_cache.py +++ b/src/orcapod/core/result_cache.py @@ -12,7 +12,7 @@ from datetime import datetime, timezone from typing import TYPE_CHECKING, Any -from orcapod.errors import SchemaVersionError +from orcapod.errors import SchemaVersionError, StoreWriteError from orcapod.protocols.core_protocols import DataProtocol from orcapod.protocols.database_protocols import ArrowDatabaseProtocol from orcapod.system_constants import constants, RESULT_DB_SCHEMA_VERSION @@ -287,15 +287,24 @@ def store( pa.array([datetime.now(timezone.utc)], type=pa.timestamp("us", tz="UTC")), ) - self._result_database.add_record( - self._versioned_record_path, - output_data.datagram_uuid.bytes, - data_table, - skip_duplicates=skip_duplicates, - ) - + try: + self._result_database.add_record( + self._versioned_record_path, + output_data.datagram_uuid.bytes, + data_table, + skip_duplicates=skip_duplicates, + ) + except Exception as exc: + raise StoreWriteError(self._versioned_record_path) from exc if self._auto_flush: - self._result_database.flush() + try: + self._result_database.flush() + except Exception as exc: + # Views of one store share their pending writes, so the batch that + # failed may be an earlier row's or another table's. + raise StoreWriteError( + self._result_database.base_path, flush=True + ) from exc def get_all_records( self, include_system_columns: bool = False diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 81c653627..8528c5ba9 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -1,3 +1,7 @@ +from collections.abc import Mapping +from typing import Any + + class InputValidationError(Exception): """ Exception raised when the inputs are not valid. @@ -195,3 +199,113 @@ def __init__( f"node={node_identity_path!r} " f"cached_content_hash={cached_content_hash!r}" ) + + +class StoreWriteError(RuntimeError): + """Raised when a node's results or records could not be written to the store. + + A failed store write always fails the run, whatever the orchestrator's + ``error_policy``: ``error_policy="continue"`` skips rows whose *function* + raised, never rows that could not be stored. The run stops at the first + failed write. Rows stored before the failure are kept, and re-running the + job does not recompute stored function results. + + The backend's own exception is chained as ``__cause__``. + + Attributes: + record_path: Path of the table the write was for, relative to the + database view the writer holds. For a failed flush, the base path of + the database view that was flushed. + node_label: Label of the node whose write failed, or ``None`` when the + failure is not tied to one node (a flush at the end of a run). + tag: Tag values of the row being stored, or ``None`` when the write + covered many rows (an operator's output), no row, or was a flush. + flush: ``True`` when committing pending writes failed. Every view of a + store shares one set of pending writes, so the write that failed may + belong to an earlier row or another table than the one being stored. + """ + + def __init__( + self, + record_path: tuple[str, ...], + node_label: str | None = None, + tag: Mapping[str, Any] | None = None, + *, + flush: bool = False, + ) -> None: + """Create the error for one failed write. + + Args: + record_path: Path of the table the write was for, or of the database + view that was flushed. + node_label: Label of the node whose write failed, if the write site knows it. + tag: Tag values of the row being stored, if the write covered one row. + flush: Whether committing pending writes failed. + """ + self.record_path: tuple[str, ...] = tuple(record_path) + self.node_label: str | None = node_label + self.tag: dict[str, Any] | None = dict(tag) if tag is not None else None + self.flush: bool = flush + super().__init__(self.record_path, node_label, tag) + + @classmethod + def in_group(cls, group: BaseExceptionGroup) -> "list[StoreWriteError]": + """Return every ``StoreWriteError`` in an exception group, nested ones included. + + Args: + group: An exception group, such as the one an ``asyncio.TaskGroup`` + raises. + + Returns: + The ``StoreWriteError`` leaves in the order the group holds them. + """ + found: list[StoreWriteError] = [] + for exc in group.exceptions: + if isinstance(exc, BaseExceptionGroup): + found.extend(cls.in_group(exc)) + elif isinstance(exc, cls): + found.append(exc) + return found + + def add_context( + self, node_label: str | None = None, tag: Mapping[str, Any] | None = None + ) -> None: + """Fill in the node label and row tag if the write site did not know them. + + A failed flush never takes a row tag: the pending write that failed may + belong to another row. + + Args: + node_label: Label of the node that owns the write. + tag: Tag values of the row being stored. + """ + if self.node_label is None: + self.node_label = node_label + if self.tag is None and tag is not None and not self.flush: + self.tag = dict(tag) + + def __str__(self) -> str: + """Describe the failed write: node, row, table and the backend's reason.""" + where = f"node {self.node_label!r}" if self.node_label else "the pipeline" + row = f" for row {self.tag!r}" if self.tag else "" + cause = self.__cause__ + reason = ( + f": {type(cause).__name__}: {str(cause).rstrip('.')}" + if cause is not None + else "" + ) + retry = ( + "The run was stopped. Rows stored before this failure are kept; " + "re-running the job does not recompute stored function results." + ) + if self.flush: + node = f" for node {self.node_label!r}" if self.node_label else "" + return ( + f"Could not commit pending writes to the store{node} " + f"(store {'/'.join(self.record_path)!r}){reason}. The failing write " + "may belong to an earlier row or another table of this store. " + retry + ) + return ( + f"Could not write to the store for {where}{row} " + f"(table {'/'.join(self.record_path)!r}){reason}. " + retry + ) diff --git a/src/orcapod/pipeline/async_orchestrator.py b/src/orcapod/pipeline/async_orchestrator.py index 4a75564c5..8020d3d03 100644 --- a/src/orcapod/pipeline/async_orchestrator.py +++ b/src/orcapod/pipeline/async_orchestrator.py @@ -14,7 +14,9 @@ from typing import TYPE_CHECKING, Any, Literal from orcapod.channels import BroadcastChannel, Channel +from orcapod.errors import StoreWriteError from orcapod.pipeline.result import OrchestratorResult +from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.node_protocols import ( is_function_node, is_operator_node, @@ -137,6 +139,8 @@ async def _run_async( effective_observer = observer if observer is not None else NoOpObserver() effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) + topo_order: list[Any] = [] + store_error: StoreWriteError | None = None try: topo_order = list(graph.topological_sort()) buf = self._buffer_size @@ -250,8 +254,71 @@ async def _run_async( return OrchestratorResult( node_outputs=collectors if materialize_results else {} ) + except ExceptionGroup as group: + # A failed store write stops the run. Raise it on its own, as the sync + # orchestrator does, rather than inside the TaskGroup's exception group. + store_error = _single_store_error(group, topo_order) + if store_error is None: + raise + raise store_error finally: - effective_observer.on_run_end(run_id) + # A failing on_run_end must not replace a failed store write. + with keep_store_error(store_error, "Observer on_run_end"): + effective_observer.on_run_end(run_id) + + +def _single_store_error( + group: ExceptionGroup, topo_order: list[Any] +) -> StoreWriteError | None: + """Pick the one ``StoreWriteError`` an async run raises from its exception group. + + Concurrent node tasks can fail together, so the ``TaskGroup``'s group may hold + several store errors and other exceptions. The caller gets a single + ``StoreWriteError``, as from the sync orchestrator: the one whose node comes + first in topological order (the one the sync orchestrator would meet first), + with ties and node-less errors broken by their order in the group. Every other + exception stays attached for diagnosis: each is noted on the chosen error, + other store errors are also logged at WARNING, and the whole group is the + chosen error's ``__context__``. Its ``__cause__`` stays the backend error. + + Args: + group: The exception group the run's ``TaskGroup`` raised. + topo_order: The run's nodes in topological order. + + Returns: + The store error to raise, or ``None`` when the group holds none. + """ + store_errors = StoreWriteError.in_group(group) + if not store_errors: + return None + rank = {getattr(node, "label", None): i for i, node in enumerate(topo_order)} + chosen = min(store_errors, key=lambda err: rank.get(err.node_label, len(rank))) + for other in _leaves(group): + if other is chosen: + continue + if isinstance(other, StoreWriteError): + logger.warning( + "Another store write failed in the same run: %s", + other, + exc_info=other, + ) + chosen.add_note(f"Another store write failed in the same run: {other}") + else: + chosen.add_note( + f"Another node failed in the same run: {type(other).__name__}: {other}" + ) + return chosen + + +def _leaves(group: BaseExceptionGroup) -> list[BaseException]: + """Return every non-group exception in ``group``, nested groups flattened.""" + found: list[BaseException] = [] + for exc in group.exceptions: + if isinstance(exc, BaseExceptionGroup): + found.extend(_leaves(exc)) + else: + found.append(exc) + return found class _CollectingWriter: diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 8b7f76ffe..7fe706f82 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -12,6 +12,7 @@ from orcapod.core.nodes.source_node import SourceJobNode from orcapod.side_effects import SideEffectJobNode from orcapod.core.tracker import AutoRegisteringContextBasedTracker +from orcapod.errors import StoreWriteError from orcapod.pipeline.base import AbstractPipelineBase from orcapod.pipeline.dag import OrcaDAG from orcapod.protocols import core_protocols as cp @@ -30,6 +31,35 @@ logger = logging.getLogger(__name__) +def _flush_best_effort( + run_error: BaseException, *databases: ArrowDatabaseProtocol +) -> None: + """Flush each database, noting instead of raising if a flush fails. + + Used when a run has already failed: the run's own error is the one to report, + and a second failure while committing the writes that did succeed must not + replace it. A flush failure is logged and added as a note to ``run_error``, + so the caller learns that pending results may not have been stored. + + Args: + run_error: The error the run failed with, which the caller re-raises. + *databases: The databases to flush. + """ + for database in databases: + try: + database.flush() + except Exception as flush_error: + logger.exception( + "Could not flush pending writes after the run failed; " + "the rows they hold were not stored" + ) + run_error.add_note( + "Flushing pending writes after this failure also failed " + + f"({type(flush_error).__name__}: {flush_error}); results that were " + + "still pending may not have been stored." + ) + + class PipelineJob(AbstractPipelineBase[JobNode]): """Pipeline + source bindings + execution context. @@ -736,6 +766,11 @@ def run( Raises: ValueError: If no store is set. RuntimeError: If no pipeline has been compiled. + StoreWriteError: If a node's results or records could not be written + to the store, whatever the orchestrator's ``error_policy``. The run + stops at the first failed write. Writes that succeeded before it + are flushed first, so rows stored before the failure are kept, and + re-running the job does not recompute stored function results. """ import hashlib import uuid @@ -827,20 +862,38 @@ def run( snapshot_hash = hashlib.sha256("\n".join(leaf_hashes).encode()).hexdigest()[:16] pipeline_uri = "/".join(self._name) + "@" + snapshot_hash + pipeline_db = self._store.at(*self._name) + result_db = pipeline_db.at("_result") + effective_orchestrator = orchestrator or SyncPipelineOrchestrator() - effective_orchestrator.run( - exec_dag, - observer=effective_observer, - run_id=run_id, - pipeline_uri=pipeline_uri, - materialize_results=materialize_results, - ) + try: + effective_orchestrator.run( + exec_dag, + observer=effective_observer, + run_id=run_id, + pipeline_uri=pipeline_uri, + materialize_results=materialize_results, + ) + except Exception as run_error: + # Commit the writes that did succeed before the failure, so their rows + # do not look dropped, then report the failure itself. + _flush_best_effort(run_error, pipeline_db, result_db) + raise # Flush databases. - pipeline_db = self._store.at(*self._name) - result_db = pipeline_db.at("_result") - pipeline_db.flush() - result_db.flush() + try: + pipeline_db.flush() + result_db.flush() + except Exception as exc: + flush_error = StoreWriteError(pipeline_db.base_path, flush=True) + # Every view of the store shares one set of pending batches, so a failed + # batch can leave the ones after it unwritten. + flush_error.add_note( + "Writes that were still pending when the flush failed, such as a " + + "cache_mode=LOG operator's output or a node's last records, may not " + + "have been stored." + ) + raise flush_error from exc # Mutate self in place. self._has_run = True diff --git a/src/orcapod/pipeline/sync_orchestrator.py b/src/orcapod/pipeline/sync_orchestrator.py index fcb21a4b4..46c2ab1f3 100644 --- a/src/orcapod/pipeline/sync_orchestrator.py +++ b/src/orcapod/pipeline/sync_orchestrator.py @@ -10,8 +10,10 @@ import uuid from typing import TYPE_CHECKING, Any, Literal +from orcapod.errors import StoreWriteError from orcapod.pipeline.result import OrchestratorResult from orcapod.utils import arrow_utils +from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.node_protocols import ( is_function_node, is_operator_node, @@ -77,6 +79,7 @@ def run( effective_observer = observer if observer is not None else NoOpObserver() effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) + store_error: StoreWriteError | None = None try: topo_order = list(graph.topological_sort()) buffers: dict[Any, list[tuple[TagProtocol, DataProtocol]]] = {} @@ -131,8 +134,13 @@ def run( buffers.clear() return OrchestratorResult(node_outputs=buffers) + except StoreWriteError as exc: + store_error = exc + raise finally: - effective_observer.on_run_end(run_id) + # A failing on_run_end must not replace a failed store write. + with keep_store_error(store_error, "Observer on_run_end"): + effective_observer.on_run_end(run_id) @staticmethod def _gather_upstream( diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index c774d49ab..8b0c0ba16 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -19,6 +19,7 @@ from orcapod.core.nodes.function_node import _build_record_id_preimage from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER +from orcapod.errors import StoreWriteError from orcapod.invocation import ( InvocationContext, InvocationHashConfig, @@ -269,6 +270,10 @@ def _execute_side_effect_row( run_id=run_id, ) return (tag, data) + except StoreWriteError as exc: + # A failed store write fails the run, even under on_error="log". + exc.add_context(node_label=pod_name, tag=tag.as_dict()) + raise except Exception as exc: if pod_config.on_error == "raise": raise @@ -314,12 +319,15 @@ def _write_invocation_row( ), } ) - pipeline_database.add_record( - table_path, - record_id, - record, - skip_duplicates=True, - ) + try: + pipeline_database.add_record( + table_path, + record_id, + record, + skip_duplicates=True, + ) + except Exception as exc: + raise StoreWriteError(table_path) from exc # --------------------------------------------------------------------------- diff --git a/src/orcapod/utils/store_errors.py b/src/orcapod/utils/store_errors.py new file mode 100644 index 000000000..d7cec2df8 --- /dev/null +++ b/src/orcapod/utils/store_errors.py @@ -0,0 +1,67 @@ +"""Helpers that keep a ``StoreWriteError`` intact on its way out of a run. + +A failed store write always fails the run (ITL-878). User callbacks that run while +the error propagates (a post-run hook, an observer callback) may fail too; their +failure is logged and noted on the store error, never raised in its place. +""" + +from __future__ import annotations + +import logging +from collections.abc import Iterator +from contextlib import contextmanager + +from orcapod.errors import StoreWriteError + +logger = logging.getLogger(__name__) + + +def note_callback_failure( + store_error: StoreWriteError, during: str, failure: Exception +) -> None: + """Log a callback failure and record it on the propagating store error. + + Args: + store_error: The store error on its way out of the run. + during: What was running, for the log and the note, such as + ``"Observer on_data_crash"``. + failure: The exception the callback raised. + """ + logger.warning( + "%s raised while a store write failure was propagating, which is raised " + + "instead: %s", + during, + failure, + exc_info=failure, + ) + store_error.add_note( + f"{during} also raised while this error propagated: " + + f"{type(failure).__name__}: {failure}" + ) + + +@contextmanager +def keep_store_error( + store_error: StoreWriteError | None, during: str +) -> Iterator[None]: + """Run a callback so that it cannot replace a propagating store error. + + With ``store_error`` set, an ``Exception`` the block raises is logged and noted on + ``store_error`` (see ``note_callback_failure``) and not re-raised, so the caller's + ``raise`` propagates the store error. With ``store_error=None`` the block runs + unchanged and its exceptions propagate. + + Args: + store_error: The store error on its way out, or ``None`` when none is. + during: What the block runs, for the log and the note. + + Yields: + Nothing; the block runs inside the context. + """ + if store_error is None: + yield + return + try: + yield + except Exception as failure: + note_callback_failure(store_error, during, failure) diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md new file mode 100644 index 000000000..bf329d202 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -0,0 +1,306 @@ +# ITL-878: a failed store write fails the run — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this +> plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** When any node's write to the store fails, `PipelineJob.run()` (and the orchestrators +under it) raises a `StoreWriteError` naming the node and table, on both orchestrators and +independent of `error_policy`, instead of returning normally with the rows silently missing. + +**Status:** Option A proposed to the coordinator on 2026-11-10; the coordinator is taking it to +the developer (new public exception + documented behaviour change). Tasks 1 (failing tests) +may proceed now; Tasks 2+ wait for approval. + +**Architecture:** Every node-owned store write site converts a backend exception into +`orcapod.errors.StoreWriteError` (chained `from` the backend error). The per-row +"skip failed rows" handlers (sync `FunctionNode.execute`, async +`_FunctionPodBase.async_execute.process_one`, side-effect delivery with `on_error="log"`) +re-raise `StoreWriteError` ahead of their generic `except Exception`, so `error_policy` keeps +meaning "skip rows whose *function* failed". The async orchestrator unwraps the +`TaskGroup` `ExceptionGroup` so async callers get the same `StoreWriteError` as sync callers. +`PipelineJob.run()` flushes pending writes best-effort before re-raising any run failure, so +rows that did write are committed and do not look dropped. + +**Tech Stack:** Python 3.11+, PyArrow, asyncio TaskGroup, pytest. + +--- + +## Investigation summary (Step 1, reproduced on main c438d4d1) + +- Sync swallow: `core/nodes/function_node.py` `FunctionNode.execute` per-row + `except Exception` (error_policy "continue") wraps `_process_data_internal`, which does + compute AND both writes (result table via `ResultCache.store`, pipeline table via + `add_pipeline_record`). WARNING log per row, run returns normally. +- Async swallow: `core/function_pod.py` `_FunctionPodBase.async_execute.process_one` catches + every per-row exception, logs at DEBUG (invisible), drops the row. +- Async pipeline-table write (`record_and_forward`) and operator LOG writes already raise, but + async callers get an `ExceptionGroup`. +- Operator-LOG failure aborts the run before `job.run`'s final flush, so pending pipeline rows + of upstream function nodes are never committed while their result rows are: the node reads + back fewer rows than were stored. +- Read failures raise on both orchestrators (not swallowed). Observer status/log writes are + deliberately best-effort and stay so. +- Re-run on a healthy store recovers in every case (missing rows recomputed; orphan results + hit the result cache). + +## Contract (Option A) + +1. A failed write of a node's results or records raises `StoreWriteError` and stops the run at + the first failure, regardless of `error_policy`, on both orchestrators. +2. `str(err)` names the node label and the table path (`"/".join(record_path)`), says rows + stored before the failure are kept and that re-running computes only what is missing. + `err.node_label`, `err.record_path` are attributes; `err.__cause__` is the backend error. +3. Before re-raising, `PipelineJob.run()` flushes pending writes best-effort; a flush failure + during that is logged and does not mask the original error. Nothing already stored is + rewritten; no hash, path or stored-schema change. +4. A failure of `PipelineJob.run()`'s final flush raises `StoreWriteError` (no node label). +5. `error_policy="continue"` still skips rows whose function raised (unchanged). + +## Files + +- Modify: `src/orcapod/errors.py` — add `StoreWriteError(RuntimeError)`. +- Modify: `src/orcapod/core/result_cache.py` — wrap `add_record` + auto-flush in `store`. +- Modify: `src/orcapod/core/nodes/function_node.py` — wrap `add_pipeline_record`'s write; + re-raise `StoreWriteError` in `execute`; tag node label in `async_execute`. +- Modify: `src/orcapod/core/function_pod.py` — `process_one` re-raises `StoreWriteError`. +- Modify: `src/orcapod/core/nodes/operator_node.py` — wrap `_store_output_stream`'s write. +- Modify: `src/orcapod/side_effects.py` — wrap `_write_invocation_row`; re-raise before the + `on_error="log"` handler. +- Modify: `src/orcapod/pipeline/async_orchestrator.py` — unwrap `StoreWriteError` from the + `ExceptionGroup`. +- Modify: `src/orcapod/pipeline/job.py` — best-effort flush on failure; wrap final flush. +- Create: `tests/test_pipeline/test_store_write_failure.py`. +- Modify: `docs/concepts/function-pods.md` (PipelineJob section), `README.md` execution + walkthrough, `DESIGN_ISSUES.md` (one entry appended at the end). + +## Task 1: Failing regression tests (public interface) + +**Files:** Create `tests/test_pipeline/test_store_write_failure.py`. + +Test double `FaultyStore`: wraps an `InMemoryArrowDatabase`; `at()` returns a wrapper sharing +one `Faults` config; fails `add_record`/`add_records` whose full path contains a marker +(`"rdb_v1"` result table, `"pdb_v1"` pipeline table, operator class name), optionally after N +successful matching writes, or fails `flush`. Records the failing relative record path so the +test can assert the message names it. `__getattr__` delegates everything else. + +All tests parametrized over `SyncPipelineOrchestrator` / `AsyncPipelineOrchestrator`. Until +the exception class is approved, assert with `pytest.raises(Exception)` plus +`not isinstance(err, BaseExceptionGroup)`, node label in `str(err)` and failing path in +`str(err)`; tighten to `StoreWriteError` in Task 2. + +- result-table write fails on the first row → run raises, names node + table. +- result-table write fails from the 2nd row → raises; a fresh job over the healthy store + recomputes only the missing rows (function call counter) and returns all rows. +- pipeline-table write fails → raises; fresh healthy re-run makes zero function calls. +- store primed with 2 rows, 3rd (new) row's write fails → raises; the 2 primed rows are still + read back by a fresh job (zero calls for them). +- `error_policy="fail_fast"` and `"continue"` both raise for a store failure. +- operator node with `cache_mode=LOG` whose write fails → raises naming the operator node; + upstream function node's pipeline rows are committed (read through a fresh job's + `nodes[label].get_all_records()` before running it). +- flush failure (`Faults(on_flush=True)`) → raises. +- SQLite trigger: function pod returning `pathlib.Path` on + `ConnectorArrowDatabase(SQLiteConnector(tmp_path / "x.db"))` → raises naming the node. +- Guard (should pass before and after): a function that raises for one row with + `error_policy="continue"` still returns normally and stores the other rows. + +- [ ] Write the tests; run `uv run pytest tests/test_pipeline/test_store_write_failure.py -v`; + confirm the store-failure tests FAIL (run returns normally / ExceptionGroup) and the + guard passes. Commit `test(pipeline): failing regression tests for swallowed store write + failures (ITL-878)`. + +## Task 2: `StoreWriteError` (after approval) + +```python +class StoreWriteError(RuntimeError): + """Raised when a node's results or records could not be written to the store. ...""" + + def __init__(self, record_path: tuple[str, ...], node_label: str | None = None) -> None: + self.record_path = tuple(record_path) + self.node_label = node_label + super().__init__(record_path, node_label) + + def __str__(self) -> str: + where = f"node {self.node_label!r}" if self.node_label else "the pipeline" + cause = self.__cause__ + reason = f": {type(cause).__name__}: {cause}" if cause is not None else "" + return ( + f"Could not write to the store for {where} (table " + f"{'/'.join(self.record_path)!r}){reason}. Rows stored before this failure are " + "kept; re-running computes only the rows that are missing." + ) +``` + +`node_label` is filled in by the node that owns the write when the write site does not know +it (`ResultCache`): `if exc.node_label is None: exc.node_label = self.label`. +Tighten Task 1 assertions to `pytest.raises(StoreWriteError)`. + +## Task 3: Write sites raise `StoreWriteError` + +- `ResultCache.store`: `try: add_record(...); if auto_flush: flush()` / + `except StoreWriteError: raise` / `except Exception as exc: raise + StoreWriteError(self._versioned_record_path) from exc`. +- `FunctionNode.add_pipeline_record`: same around `self._pipeline_database.add_record`, with + `node_label=self.label`, path `self._versioned_pipeline_path`. +- `OperatorNode._store_output_stream`: same around `add_records`, path + `self.node_identity_path`. +- `side_effects._write_invocation_row`: same, path `table_path` (label filled by caller). + +## Task 4: Per-row handlers re-raise + +- `FunctionNode.execute`: before `except Exception`, add + `except StoreWriteError as exc:` → set label, `ctx_obs.on_data_crash(...)`, + `ctx_obs.on_node_end(...)`, `raise`. +- `_FunctionPodBase.async_execute.process_one`: `except StoreWriteError as exc:` + → `obs.on_data_crash(...)`; `raise`. (Only this; no error_policy threading or log-level + change — those are the separately filed async error_policy issue.) +- `FunctionNode.async_execute`: wrap the DB-path `TaskGroup` with `except* StoreWriteError` + setting `node_label` on each leaf whose label is `None`, then re-raise the group. +- `side_effects` delivery: `except StoreWriteError: raise` before the `on_error` handler. + +## Task 5: Orchestrator and job + +- `AsyncPipelineOrchestrator._run_async`: `except* StoreWriteError as eg: raise + _first_leaf(eg)` where `_first_leaf` walks nested groups to the first `StoreWriteError`. +- `PipelineJob.run()`: wrap `effective_orchestrator.run(...)` in `try/except Exception:` + → `_flush_best_effort(pipeline_db, result_db)` (logs flush errors with `logger.exception`) + → `raise`. Final flush: `except Exception as exc: raise + StoreWriteError(pipeline_db.base_path) from exc`. + +- [ ] Run new tests (pass), then `uv run pytest tests/` (full), ruff check/format, + basedpyright exit code. Commit `fix(pipeline): a failed store write fails the run + (ITL-878)`. + +## Task 6: Docs and design log + +- `docs/concepts/function-pods.md` PipelineJob section: subsection "When the store cannot be + written" stating the contract (prose; no executed sample needs infrastructure). +- `README.md` execution walkthrough: note store errors are not skipped by `error_policy`. +- `DESIGN_ISSUES.md`: append one entry (resolved, Fix note, ITL-878). +- [ ] `uv run pytest tests/test_docs/`; commit `docs: state the store-write failure contract + (ITL-878)`. + +## Out of scope (reported, not fixed) + +- Async orchestrator ignores `error_policy`; async per-row function errors logged at DEBUG + (coordinator is filing). This plan touches `process_one` only to re-raise + `StoreWriteError`. +- Connector/Delta `flush` pop the pending batch before writing (coordinator is filing). +- ITL-877 (SQLite logical types). + +--- + +## Follow-up: CodeRabbit round 1 (approved by the coordinator if each finding reproduces) + +**Finding 1 (comment 4534032564): a failing hook or observer callback replaces a +`StoreWriteError`.** A node's own `CachedFunctionPod` never carries hooks, so the reachable +case is a node function that runs another cached pod (`@function_pod(pod_cache_database=..., +post_run_hooks=[...])`) whose store write fails: the ERROR-status hook fires inside +`_invoke_with_hooks`'s `except`, a fail-loud hook raises, its exception replaces the store +error, and the outer node skips the row under `error_policy="continue"`. The same applies to +an observer's `on_data_crash` / `on_node_end` raised on the store-error branch. + +- Test first (public interface): standalone `pod.pod(stream).as_table()` and + `pod.pod.async_execute(...)` with a raising hook raise `StoreWriteError`; a pipeline whose + node function runs such a pod raises `StoreWriteError` on both orchestrators under + `continue`; a pipeline observer whose `on_data_crash` raises does not hide it. +- Fix: `_fire_post_run_hooks(payload, store_error=...)`: when a store error is propagating, + a raising hook is logged at WARNING and recorded with `add_note` on the store error, which + then propagates. Observer callbacks on the store-error branches go through one helper that + does the same. Hook failures without a store error are unchanged. + +**Finding 2 (comment 4534032576): mixed exception groups from the async orchestrator.** +`except* StoreWriteError` re-raises the unmatched rest as an `ExceptionGroup` together with +the store error, and drops further store errors silently. + +- Test first: two sibling nodes failing concurrently (store write + another exception; two + store writes) → the async caller gets one plain `StoreWriteError`. +- Fix: catch `ExceptionGroup`; if it holds any `StoreWriteError`, raise the one whose node + comes first in topological order (ties: group order), the order the sync orchestrator meets + them; log every other `StoreWriteError` at WARNING; add a note per other exception; the + whole group stays reachable as `__context__` (`__cause__` stays the backend error). A + group with no store error propagates unchanged. + +No hash, schema or other behaviour change. + +--- + +## Follow-up: CodeRabbit round 2, out-of-diff findings (review of 0940f995) + +Three findings sat in the review body ("Outside diff range comments"), not in threads. + +**Finding A (job.py final flush, approved if it reproduces): the final-flush `StoreWriteError` +does not warn that pending output may not be stored.** All views of one store share the +pending batches, so `pipeline_db.flush()` commits every table; a batch that fails stops the +flush and the batches after it (a `cache_mode=LOG` operator's output, other nodes' rows) +stay pending, and `result_db.flush()` never runs. + +- Test first: `Faults(on_flush=True)` with the LOG operator; the raised `StoreWriteError` + carries a note saying writes still pending may not have been stored. +- Fix: build the error, `add_note(...)`, `raise err from exc`. Same error class, same + `record_path`, same `__cause__`. + +**Finding B (job.py `_flush_best_effort`, approved if it reproduces): a failed best-effort +flush shows only in the log.** + +- Test first: an operator LOG write fails and the best-effort flush fails too; the raised + `StoreWriteError` is still the operator's, and carries a note that the flush failed and + pending results may not have been stored (upstream rows read back: 0). +- Fix: `_flush_best_effort(run_error, *databases)` keeps logging and adds a note naming the + flush failure to the propagating run error. The raised error is unchanged. + +**Finding C (`ResultCache.store` record preparation): REPRODUCE AND REPORT ONLY.** Establish +on both orchestrators and both error policies which function outputs fail in record +preparation (naive datetime, value not matching the declared type, unsupported type, a +logical type whose `python_to_storage` raises), what the caller sees (row skipped? WARNING +log? observer `on_data_crash`? run returns? anything stored?), and what CodeRabbit's +proposal would change. Ask the coordinator to choose (a) row failure under `error_policy` +reported exactly like a function failure, or (b) `StoreWriteError`. No fix before the reply. + +No hash, schema or stored-data change. + +--- + +## Follow-up: Finding C decided — an unstorable output is a row failure (2026-11-11) + +**Decision (developer, November 11, 2026):** a function output that cannot be turned into a +storable record (naive datetime, `str` for a declared `int`, `int` for a declared dataclass) +is a ROW FAILURE governed by `error_policy`, exactly like a function that raised. It is not a +`StoreWriteError`. No behaviour change. + +Resumed after a machine restart; the previous worker's local commit ff4c25cb (12 pinning +tests) is kept. + +- [x] Pinning tests (`TestUnstorableOutputIsARowFailure`, sync/async x continue/fail_fast x + 3 outputs, each compared with a function raising on the same row) also assert the + per-row log record (level + message) matches the function-failure one, besides the + observer `on_data_crash`, raise-or-return, rows and stored rows. +- [x] Docs (`docs/concepts/function-pods.md`, "When the store cannot be written"): store + failures fail the run; a function output that cannot be stored is a row failure; + async logs row failures at DEBUG and ignores `fail_fast` (ITL-879), unchanged here. +- [x] `DESIGN_ISSUES.md` SW1: the same rule. +- [x] Decline finding (1) with a signed top-level PR comment (issuecomment-6791981712). +- [x] Merged origin/main b197544e (DESIGN_ISSUES conflict: both appended sections kept); + added the `[0.3.0] — unreleased` CHANGELOG section with this PR's entries. +- [x] Merge origin/main (b197544e; CHANGELOG entries under `[0.3.0] — unreleased`), full + suite on 3.11/3.12/3.13, golden unchanged, ruff, format, basedpyright, push, CI, + CodeRabbit round (bodies too). Reconcile with #290 if it merged first. + +--- + +## Follow-up: CodeRabbit round 3 (review of d305c806), approved by the coordinator 2026-11-11 + +- (A) docs: a normal return stores every result that could be turned into a record; point to + the row-failure paragraph. +- (B) docs: a side effect whose delivery completed but whose invocation-log write failed is + delivered again on re-run (track_completion has no record); check or make it idempotent. +- (C) `ResultCache.store`: separate `add_record` from auto-`flush`. A flush failure raises + `StoreWriteError(result_db.base_path, flush=True)`; new keyword-only `flush: bool` + attribute; `add_context` then sets the node label but not the row tag; message says pending + writes could not be committed and the failing write may belong to an earlier row or another + table. `job.py` final flush uses `flush=True`. Test first. +- (D) `StoreWriteError` message: "Rows stored before this failure are kept; re-running the job + does not recompute stored function results." Docs explain LOG operators recompute. +- Keep shape consistent with #290's `StoreReadError`. +- [x] tests, fix (392ee8ed), replies on the 4 threads. +- [ ] CI and CodeRabbit on the final head, worker_done. diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py new file mode 100644 index 000000000..ce25ef1af --- /dev/null +++ b/tests/test_pipeline/test_store_write_failure.py @@ -0,0 +1,822 @@ +"""A failed store write must fail ``PipelineJob.run()`` (ITL-878). + +Before the fix, a node whose write to the store failed was skipped like a row whose +function had raised: ``run()`` returned normally and the node read back fewer rows (often +zero). These tests drive ``PipelineJob`` through its public interface on both orchestrators, +with a test-double store that fails chosen writes, and with the SQLite trigger from the +report (a logical-type output on ``SQLiteConnector``). +""" + +from __future__ import annotations + +import asyncio +import pathlib +from dataclasses import dataclass, field +from datetime import datetime, timezone +from typing import Any + +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.channels import Channel +from orcapod.core.operators import SelectDataColumns +from orcapod.errors import StoreWriteError +from orcapod.databases import ( + ConnectorArrowDatabase, + InMemoryArrowDatabase, + SQLiteConnector, +) +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.pipeline.observer import NoOpObserver +from orcapod.sources import DictSource +from orcapod.types import CacheMode + +JOB_NAME = "itl878" + +# Markers that select a table by its stored path. +RESULT_TABLE = "rdb_v1" # a function node's result table +PIPELINE_TABLE = "pdb_v1" # a function node's pipeline (tag) table +OPERATOR_TABLE = "SelectDataColumns" # the operator node's LOG table + + +# --------------------------------------------------------------------------- +# Test-double store +# --------------------------------------------------------------------------- + + +@dataclass +class Faults: + """Which writes a ``FaultyStore`` fails; shared by every view of one store.""" + + write_marker: str | None = None + succeed_first: int = 0 + on_flush: bool = False + flush_marker: str | None = None + fail_flush_after_write_failure: bool = False + matching_writes: int = 0 + failed_paths: list[str] = field(default_factory=list) + pending_paths: list[str] = field(default_factory=list) + + +class FaultyStore: + """An ``InMemoryArrowDatabase`` that raises on chosen writes or on flush.""" + + def __init__(self, inner: Any, faults: Faults) -> None: + """Wrap ``inner``, failing the writes ``faults`` selects.""" + self._inner = inner + self._faults = faults + + def at(self, *path_components: str) -> FaultyStore: + """Return a view at ``path_components`` that shares this store's faults.""" + return FaultyStore(self._inner.at(*path_components), self._faults) + + def _check_write(self, record_path: tuple[str, ...]) -> None: + """Raise ``OSError`` if the write to ``record_path`` is one to fail.""" + faults = self._faults + full_path = "/".join(self._inner.base_path + tuple(record_path)) + faults.pending_paths.append(full_path) + if faults.write_marker is None or faults.write_marker not in full_path: + return + faults.matching_writes += 1 + if faults.matching_writes > faults.succeed_first: + faults.failed_paths.append("/".join(record_path)) + if faults.fail_flush_after_write_failure: + faults.on_flush = True + raise OSError(f"disk unavailable while writing {full_path}") + + def add_record( + self, record_path, record_id, record, skip_duplicates=False, flush=False + ): + """Write one record unless the faults select this table.""" + self._check_write(record_path) + self._inner.add_record( + record_path, record_id, record, skip_duplicates=skip_duplicates, flush=flush + ) + + def add_records( + self, + record_path, + records, + record_id_column=None, + skip_duplicates=False, + flush=False, + ): + """Write a batch of records unless the faults select this table.""" + self._check_write(record_path) + self._inner.add_records( + record_path, + records, + record_id_column=record_id_column, + skip_duplicates=skip_duplicates, + flush=flush, + ) + + def flush(self) -> None: + """Flush the wrapped store, or raise ``OSError`` if flushes fail. + + With ``flush_marker`` set, a flush fails only while a write to a matching + table is pending, whichever view the flush is called on: every view shares + one set of pending batches. + """ + faults = self._faults + if faults.on_flush: + raise OSError("disk unavailable while flushing") + marker = faults.flush_marker + if marker is not None and any(marker in p for p in faults.pending_paths): + raise OSError(f"disk unavailable while flushing a pending {marker} batch") + self._inner.flush() + faults.pending_paths.clear() + + def __getattr__(self, name: str) -> Any: + """Delegate everything else to the wrapped store.""" + return getattr(self._inner, name) + + +# --------------------------------------------------------------------------- +# Pipeline under test +# --------------------------------------------------------------------------- + +ROWS = [{"k": f"r{i}", "x": i} for i in range(3)] + +calls: list[int] = [] + + +@function_pod(output_keys="double") +def double(x: int) -> int: + """Double ``x``, recording the call.""" + calls.append(x) + return x * 2 + + +@function_pod(output_keys="double") +def double_or_fail(x: int) -> int: + """Double ``x``, raising for ``x == 1``.""" + if x == 1: + raise RuntimeError("function failed for x == 1") + return x * 2 + + +@function_pod(output_keys="where") +def to_path(x: int) -> pathlib.Path: + """Return a ``pathlib.Path`` output, a logical type SQLite cannot store.""" + return pathlib.Path(f"/data/{x}.txt") + + +def build_job( + store: Any, rows: list[dict[str, Any]], *, pod=double, with_operator: bool = False +): + """Build a job with one function node ``dbl`` and optionally an operator ``sel``.""" + job = PipelineJob(name=JOB_NAME, store=store) + with job: + source = DictSource(data=rows, tag_columns=["k"], source_id="src") + out = pod.pod(source, label="dbl") + if with_operator: + SelectDataColumns(["double"])(out, label="sel") + if with_operator: + # OperatorNode.attach_databases is the public way to turn on LOG mode. + job.nodes["sel"].attach_databases( + pipeline_database=store.at(JOB_NAME), cache_mode=CacheMode.LOG + ) + return job + + +@pytest.fixture(autouse=True) +def _reset_calls(): + """Clear the call log before each test.""" + calls.clear() + + +@pytest.fixture(params=["sync", "async"]) +def orchestrator_cls(request): + """Run each test on the sync and the async orchestrator.""" + return {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator}[ + request.param + ] + + +def assert_store_error( + err: BaseException, + faults: Faults, + *, + node_label: str | None, + names_row: bool, + flush: bool = False, +) -> None: + """The run's error is a plain ``StoreWriteError`` naming node, row and table.""" + assert isinstance(err, StoreWriteError), f"{type(err).__name__}: {err}" + message = str(err) + if node_label is not None: + assert err.node_label == node_label + assert f"node {node_label!r}" in message, message + if names_row: + assert err.tag is not None and err.tag["k"] in {row["k"] for row in ROWS} + assert f"for row {err.tag!r}" in message, message + if faults.failed_paths: + assert "/".join(err.record_path) == faults.failed_paths[0] + assert faults.failed_paths[0] in message, message + assert err.flush is flush + if flush: + assert err.tag is None and "for row" not in message, message + assert "pending writes" in message and "earlier row" in message, message + assert isinstance(err.__cause__, OSError) + assert "does not recompute stored function results" in message + + +class FailFlushAtRunEnd(NoOpObserver): + """Make every flush fail once the orchestrator has finished, so the job's own + final flush is the one that fails (the result cache flushes after every row).""" + + def __init__(self, faults: Faults) -> None: + """Arm ``faults`` at the end of the run.""" + self._faults = faults + + def on_run_end(self, run_id) -> None: + """Make the following flushes fail.""" + self._faults.on_flush = True + + +def stored_rows(store: Any, label: str = "dbl") -> int: + """Rows a fresh job over ``store`` reads back for ``label`` without running.""" + records = build_job(store, ROWS).nodes[label].get_all_records() + return 0 if records is None else records.num_rows + + +# --------------------------------------------------------------------------- +# Function node writes +# --------------------------------------------------------------------------- + + +class TestFunctionNodeWriteFailure: + """A function node's failed result or pipeline write fails the run.""" + + def test_result_table_failure_on_first_row_raises(self, orchestrator_cls): + """A result write failing on the first row raises, naming node, row and table.""" + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + + def test_result_table_failure_on_later_row_keeps_earlier_rows( + self, orchestrator_cls + ): + """Rows stored before the failure are kept and are not recomputed on a re-run.""" + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=RESULT_TABLE, succeed_first=1) + job = build_job(FaultyStore(inner, faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + assert stored_rows(inner) == 1 + + # A re-run on a healthy store does not recompute the stored rows. + calls.clear() + rerun = build_job(inner, ROWS) + rerun.run(orchestrator=orchestrator_cls()) + assert rerun.nodes["dbl"].as_table().num_rows == 3 + assert len(calls) == 2 + + def test_pipeline_table_failure_raises(self, orchestrator_cls): + """A pipeline-record write failure raises; the failing row's stored result is reused.""" + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=PIPELINE_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + + # The failing row's result was stored before its pipeline record failed, so a + # healthy re-run reuses it instead of computing it again. (On the async + # orchestrator a row still computing when the run stopped was never stored + # and is computed again, so the total call count is not fixed.) + failed_tag = exc_info.value.tag + assert failed_tag is not None + failed_x = next(row["x"] for row in ROWS if row["k"] == failed_tag["k"]) + calls.clear() + rerun = build_job(inner, ROWS) + rerun.run(orchestrator=orchestrator_cls()) + assert rerun.nodes["dbl"].as_table().num_rows == 3 + assert failed_x not in calls + + def test_failure_on_new_row_keeps_previously_stored_rows(self, orchestrator_cls): + """A failure on a new row keeps the rows a previous run stored.""" + inner = InMemoryArrowDatabase() + build_job(inner, ROWS[:2]).run(orchestrator=orchestrator_cls()) + assert stored_rows(inner) == 2 + + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS) + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + assert stored_rows(inner) == 2 + + @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) + def test_store_failure_raises_under_any_error_policy( + self, orchestrator_cls, error_policy + ): + """A store failure raises under both error policies.""" + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(error_policy=error_policy)) + + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + + def test_auto_flush_failure_raises_naming_the_node(self, orchestrator_cls): + """A failure of the result cache's flush after a row raises, naming the node.""" + faults = Faults(on_flush=True) + store = FaultyStore(InMemoryArrowDatabase(), faults) + job = build_job(store, ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error( + exc_info.value, faults, node_label="dbl", names_row=False, flush=True + ) + + def test_auto_flush_failure_of_an_earlier_write_names_no_row( + self, orchestrator_cls + ): + """A result auto-flush that fails on another table's pending write names + neither the row being stored nor the result table. + + The sync orchestrator meets the failure in the next row's result auto-flush. + The async one may write every pipeline record after the last result flush, + so the job's final flush can be the one that fails; either way it is a flush + error that names no row. + """ + faults = Faults(flush_marker=PIPELINE_TABLE) + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + err = exc_info.value + assert_store_error(err, faults, node_label=None, names_row=False, flush=True) + if orchestrator_cls is SyncPipelineOrchestrator: + assert err.node_label == "dbl" + assert RESULT_TABLE not in "/".join(err.record_path) + assert RESULT_TABLE not in str(err) + + def test_final_flush_failure_raises(self, orchestrator_cls): + """A failure of the job's final flush raises ``StoreWriteError``.""" + faults = Faults() + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(), observer=FailFlushAtRunEnd(faults)) + + assert_store_error( + exc_info.value, faults, node_label=None, names_row=False, flush=True + ) + assert exc_info.value.record_path == (JOB_NAME,) + + def test_function_failure_is_still_skipped_under_continue(self, orchestrator_cls): + """``error_policy="continue"`` keeps skipping rows whose function raised.""" + inner = InMemoryArrowDatabase() + job = build_job(inner, ROWS, pod=double_or_fail) + + job.run(orchestrator=orchestrator_cls(error_policy="continue")) + + assert sorted(job.nodes["dbl"].as_table()["k"].to_pylist()) == ["r0", "r2"] + + +# --------------------------------------------------------------------------- +# Operator node writes (cache_mode=LOG) +# --------------------------------------------------------------------------- + + +class TestOperatorNodeWriteFailure: + """An operator node's failed LOG write fails the run.""" + + def test_operator_log_failure_raises_and_commits_upstream_rows( + self, orchestrator_cls + ): + """The operator's failure raises and the upstream node's rows stay committed.""" + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=OPERATOR_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS, with_operator=True) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, faults, node_label="sel", names_row=False) + # The upstream function node wrote all its rows before the operator failed; + # none of them may look dropped. + assert stored_rows(inner) == 3 + + def test_failed_flush_after_write_failure_is_noted(self, orchestrator_cls): + """A failed best-effort flush is noted on the write error, which stays primary.""" + inner = InMemoryArrowDatabase() + faults = Faults( + write_marker=OPERATOR_TABLE, fail_flush_after_write_failure=True + ) + job = build_job(FaultyStore(inner, faults), ROWS, with_operator=True) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + err = exc_info.value + assert_store_error(err, faults, node_label="sel", names_row=False) + notes = getattr(err, "__notes__", []) + assert any( + "disk unavailable while flushing" in note + and "may not have been stored" in note + for note in notes + ), notes + + def test_final_flush_failure_warns_pending_output(self, orchestrator_cls): + """A failed final flush warns that output still pending may not be stored.""" + faults = Faults() + job = build_job( + FaultyStore(InMemoryArrowDatabase(), faults), ROWS, with_operator=True + ) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(), observer=FailFlushAtRunEnd(faults)) + + err = exc_info.value + assert_store_error(err, faults, node_label=None, names_row=False, flush=True) + notes = getattr(err, "__notes__", []) + assert any( + "pending" in note + and "operator" in note + and "may not have been stored" in note + for note in notes + ), notes + + +# --------------------------------------------------------------------------- +# The reported trigger: a logical-type output on SQLite (ITL-877 is the limit itself) +# --------------------------------------------------------------------------- + + +class TestSQLiteTrigger: + """The reported trigger: a logical-type output on SQLite.""" + + def test_logical_type_output_on_sqlite_raises(self, orchestrator_cls, tmp_path): + """A ``pathlib.Path`` output on SQLite raises, naming the node.""" + store = ConnectorArrowDatabase(SQLiteConnector(str(tmp_path / "store.db"))) + job = build_job(store, ROWS, pod=to_path) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert exc_info.value.node_label == "dbl" + assert "node 'dbl'" in str(exc_info.value) + assert "does not preserve Arrow metadata" in str(exc_info.value) + + +# --------------------------------------------------------------------------- +# A failing hook or observer callback never replaces a store error +# --------------------------------------------------------------------------- + + +def failing_hook(payload: Any) -> None: + """A fail-loud post-run hook that always raises.""" + raise ValueError("post-run hook failed") + + +def make_hooked_triple(store: Any): + """A cached pod with a fail-loud post-run hook, writing its results to ``store``.""" + + @function_pod( + output_keys="tripled", pod_cache_database=store, post_run_hooks=[failing_hook] + ) + def triple(x: int) -> int: + """Triple ``x``.""" + return x * 3 + + return triple + + +def make_outer(inner: Any): + """A function pod whose function runs ``inner`` through its stream.""" + + @function_pod(output_keys="double") + def via_inner(x: int) -> int: + """Run ``inner`` on ``x`` through a one-row stream and return ``3x - x``.""" + stream = DictSource(data=[{"i": "i", "x": x}], tag_columns=["i"]) + return inner.pod(stream).as_table()["tripled"][0].as_py() - x + + return via_inner + + +def assert_hook_failure_noted(err: BaseException) -> None: + """The error is the store error, with the hook failure recorded as a note.""" + assert isinstance(err, StoreWriteError), f"{type(err).__name__}: {err}" + assert isinstance(err.__cause__, OSError) + notes = getattr(err, "__notes__", []) + assert any("post-run hook failed" in note for note in notes), notes + + +class CrashingObserver(NoOpObserver): + """An observer whose crash and end callbacks raise once a row has crashed.""" + + def __init__(self) -> None: + """Start with no crashed row.""" + self.crashed = False + + def on_data_crash(self, node_label, tag, data, error) -> None: + """Record the crash, then fail.""" + self.crashed = True + raise RuntimeError("observer failed in on_data_crash") + + def on_node_end(self, node_label, node_hash) -> None: + """Fail if a row has crashed.""" + if self.crashed: + raise RuntimeError("observer failed in on_node_end") + + def on_run_end(self, run_id) -> None: + """Fail if a row has crashed.""" + if self.crashed: + raise RuntimeError("observer failed in on_run_end") + + +class TestCallbackFailureKeepsStoreError: + """A hook or observer that raises never replaces a store error.""" + + def test_failing_hook_keeps_store_error_standalone(self): + """A standalone cached pod raises its store error, not the hook's.""" + faults = Faults(write_marker="") + triple = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) + stream = DictSource(data=ROWS, tag_columns=["k"]) + + with pytest.raises(StoreWriteError) as exc_info: + triple.pod(stream).as_table() + + assert_hook_failure_noted(exc_info.value) + + def test_failing_hook_keeps_store_error_standalone_async(self): + """The async path raises the store error too, rather than skipping the row.""" + faults = Faults(write_marker="") + triple = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) + stream = DictSource(data=ROWS, tag_columns=["k"]) + input_ch: Channel[Any] = Channel(buffer_size=16) + output_ch: Channel[Any] = Channel(buffer_size=16) + + async def feed() -> None: + """Send the stream's rows into the input channel, then close it.""" + for tag, data in stream.iter_data(): + await input_ch.writer.send((tag, data)) + await input_ch.writer.close() + + async def run() -> None: + """Feed the pod and run it concurrently.""" + await asyncio.gather( + feed(), triple.pod.async_execute([input_ch.reader], output_ch.writer) + ) + + with pytest.raises(ExceptionGroup) as exc_info: + asyncio.run(run()) + + leaves = exc_info.value.exceptions + assert all(isinstance(leaf, StoreWriteError) for leaf in leaves), leaves + for leaf in leaves: + assert_hook_failure_noted(leaf) + + @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) + def test_failing_hook_in_nested_pod_fails_the_run( + self, orchestrator_cls, error_policy + ): + """A node function running a cached pod whose write fails cannot skip the row.""" + faults = Faults(write_marker="") + inner = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) + job = build_job(InMemoryArrowDatabase(), ROWS, pod=make_outer(inner)) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(error_policy=error_policy)) + + assert_hook_failure_noted(exc_info.value) + + def test_failing_observer_keeps_store_error(self, orchestrator_cls): + """Observer failures on the way out are noted on the store error.""" + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(), observer=CrashingObserver()) + + err = exc_info.value + assert_store_error(err, faults, node_label="dbl", names_row=True) + notes = getattr(err, "__notes__", []) + assert any("observer failed in on_data_crash" in note for note in notes), notes + assert any("observer failed in on_run_end" in note for note in notes), notes + + +# --------------------------------------------------------------------------- +# Concurrent failures in the async orchestrator +# --------------------------------------------------------------------------- + +# ``async def`` functions run on the event loop rather than in worker threads, so +# the order in which concurrent node tasks fail is the same on every run. + + +@function_pod(output_keys="double") +async def double_async(x: int) -> int: + """Double ``x`` on the event loop.""" + return x * 2 + + +@function_pod(output_keys="tripled") +async def triple_async(x: int) -> int: + """Triple ``x`` on the event loop.""" + return x * 3 + + +@function_pod(output_keys="tripled") +async def fail_async(x: int) -> int: + """Always raise, on the event loop.""" + raise RuntimeError("function failed") + + +class FailingCrashObserver(NoOpObserver): + """An observer whose ``on_data_crash`` raises for one node, failing that node.""" + + def on_data_crash(self, node_label, tag, data, error) -> None: + """Fail for crashes of node ``other`` only.""" + if node_label == "other": + raise KeyError("observer failed for node 'other'") + + +def build_siblings(store: Any, other_pod: Any): + """Two function nodes reading the same source: ``dbl`` and ``other``.""" + job = PipelineJob(name=JOB_NAME, store=store) + with job: + source = DictSource(data=ROWS, tag_columns=["k"], source_id="src") + double_async.pod(source, label="dbl") + other_pod.pod(source, label="other") + return job + + +class TestConcurrentFailures: + """Concurrent node failures still give one plain ``StoreWriteError``.""" + + def test_store_error_and_other_failure_raise_one_store_error(self): + """A store error mixed with another failure raises the store error alone.""" + faults = Faults(write_marker=RESULT_TABLE) + job = build_siblings(FaultyStore(InMemoryArrowDatabase(), faults), fail_async) + + with pytest.raises(StoreWriteError) as exc_info: + job.run( + orchestrator=AsyncPipelineOrchestrator(), + observer=FailingCrashObserver(), + ) + + err = exc_info.value + assert_store_error(err, faults, node_label="dbl", names_row=True) + notes = getattr(err, "__notes__", []) + assert any("observer failed for node 'other'" in note for note in notes), notes + # The whole group stays reachable for diagnosis. + assert isinstance(err.__context__, ExceptionGroup) + + def test_several_store_errors_raise_the_first_in_node_order( + self, orchestrator_cls, caplog + ): + """Several store errors raise the first node's, the same on both orchestrators.""" + faults = Faults(write_marker=RESULT_TABLE) + job = build_siblings(FaultyStore(InMemoryArrowDatabase(), faults), triple_async) + + with caplog.at_level("WARNING"): + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + # The sync orchestrator meets ``dbl`` first; the async one raises the same. + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + if orchestrator_cls is AsyncPipelineOrchestrator: + others = [ + record + for record in caplog.records + if record.levelname == "WARNING" + and "Another store write failed" in record.getMessage() + ] + assert any("node 'other'" in r.getMessage() for r in others), others + + +# --------------------------------------------------------------------------- +# An output that cannot be turned into a record is a row failure, not a store error +# --------------------------------------------------------------------------- + + +@dataclass +class Pair: + """A dataclass output type.""" + + a: int + b: int + + +@function_pod(output_keys="y") +def raises_for_one(x: int) -> int: + """Raise for ``x == 1``: the reference function failure.""" + if x == 1: + raise RuntimeError("function failed for x == 1") + return x + + +@function_pod(output_keys="y") +def naive_datetime_for_one(x: int) -> datetime: + """Return a naive ``datetime`` for ``x == 1``, rejected by the strict tz policy.""" + return datetime(2020, 1, 1 + x, tzinfo=None if x == 1 else timezone.utc) + + +@function_pod(output_keys="y") +def str_for_int_for_one(x: int) -> int: + """Return a ``str`` for ``x == 1`` although ``int`` is declared.""" + return "abc" if x == 1 else x # pyright: ignore[reportReturnType] # the bad output under test + + +@function_pod(output_keys="y") +def int_for_dataclass_for_one(x: int) -> Pair: + """Return an ``int`` for ``x == 1`` although a dataclass is declared.""" + return 7 if x == 1 else Pair(x, x) # pyright: ignore[reportReturnType] # the bad output under test + + +class CrashRecorder(NoOpObserver): + """Record the node label and row key of every ``on_data_crash``.""" + + def __init__(self) -> None: + """Start with no crashes.""" + self.crashes: list[tuple[str, str]] = [] + + def on_data_crash(self, node_label, tag, data, error) -> None: + """Record the crash.""" + self.crashes.append((node_label, str(tag["k"]))) + + +# The per-row handlers' log messages: sync ``FunctionNode.execute`` and async +# ``process_one`` (DEBUG there, ITL-879). +ROW_FAILURE_LOGS = ( + "Data execution failed in %s: %s", + "Data processing failed, skipping: %s", +) + + +def row_failure_outcome( + pod: Any, orchestrator: Any, caplog: pytest.LogCaptureFixture +) -> dict[str, Any]: + """Run ``pod`` as node ``dbl`` and describe what the caller sees.""" + store = InMemoryArrowDatabase() + observer = CrashRecorder() + job = build_job(store, ROWS, pod=pod) + caplog.clear() + with caplog.at_level("DEBUG", logger="orcapod"): + try: + job.run(orchestrator=orchestrator, observer=observer) + raised = None + rows = sorted(job.nodes["dbl"].as_table()["k"].to_pylist()) + except Exception as exc: + raised = exc + rows = None + logs = [ + (record.levelname, record.msg) + for record in caplog.records + if record.msg in ROW_FAILURE_LOGS + ] + return { + "raised": raised, + "rows": rows, + "crashes": observer.crashes, + "stored": stored_rows(store), + "logs": logs, + } + + +class TestUnstorableOutputIsARowFailure: + """An output that cannot become a record is reported like a function failure.""" + + @pytest.mark.parametrize( + "pod", + [naive_datetime_for_one, str_for_int_for_one, int_for_dataclass_for_one], + ids=["naive_datetime", "str_for_int", "int_for_dataclass"], + ) + @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) + def test_reported_like_a_function_failure( + self, orchestrator_cls, error_policy, pod, caplog + ): + """Same raise-or-return, rows, log, observer report and stored rows as a raise.""" + expected = row_failure_outcome( + raises_for_one, orchestrator_cls(error_policy=error_policy), caplog + ) + actual = row_failure_outcome( + pod, orchestrator_cls(error_policy=error_policy), caplog + ) + + assert not isinstance(actual["raised"], StoreWriteError), actual["raised"] + assert (actual["raised"] is None) == (expected["raised"] is None), actual + assert actual["rows"] == expected["rows"] + assert actual["crashes"] == expected["crashes"] == [("dbl", "r1")] + assert len(expected["logs"]) == 1, expected["logs"] + assert actual["logs"] == expected["logs"] + assert actual["stored"] == expected["stored"]