From d1e7c0efbdfd0e394bd77a710536a4ecb9eb4e01 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 19:00:36 -0800 Subject: [PATCH 01/15] test(pipeline): failing regression tests for swallowed store write failures (ITL-878) Adds the ITL-878 plan and regression tests that drive PipelineJob.run() on both orchestrators with a store that fails chosen writes, and with the SQLite logical-type trigger. They fail today: the run returns normally, or the async orchestrator raises an ExceptionGroup without naming the node. Co-Authored-By: Claude Opus 5.5 --- .../2026-11-10-itl-878-store-write-failure.md | 189 +++++++++++ .../test_pipeline/test_store_write_failure.py | 311 ++++++++++++++++++ 2 files changed, 500 insertions(+) create mode 100644 superpowers/plans/2026-11-10-itl-878-store-write-failure.md create mode 100644 tests/test_pipeline/test_store_write_failure.py diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md new file mode 100644 index 000000000..32f993ec2 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -0,0 +1,189 @@ +# ITL-878: a failed store write fails the run — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this +> plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** When any node's write to the store fails, `PipelineJob.run()` (and the orchestrators +under it) raises a `StoreWriteError` naming the node and table, on both orchestrators and +independent of `error_policy`, instead of returning normally with the rows silently missing. + +**Status:** Option A proposed to the coordinator on 2026-11-10; the coordinator is taking it to +the developer (new public exception + documented behaviour change). Tasks 1 (failing tests) +may proceed now; Tasks 2+ wait for approval. + +**Architecture:** Every node-owned store write site converts a backend exception into +`orcapod.errors.StoreWriteError` (chained `from` the backend error). The per-row +"skip failed rows" handlers (sync `FunctionNode.execute`, async +`_FunctionPodBase.async_execute.process_one`, side-effect delivery with `on_error="log"`) +re-raise `StoreWriteError` ahead of their generic `except Exception`, so `error_policy` keeps +meaning "skip rows whose *function* failed". The async orchestrator unwraps the +`TaskGroup` `ExceptionGroup` so async callers get the same `StoreWriteError` as sync callers. +`PipelineJob.run()` flushes pending writes best-effort before re-raising any run failure, so +rows that did write are committed and do not look dropped. + +**Tech Stack:** Python 3.11+, PyArrow, asyncio TaskGroup, pytest. + +--- + +## Investigation summary (Step 1, reproduced on main c438d4d1) + +- Sync swallow: `core/nodes/function_node.py` `FunctionNode.execute` per-row + `except Exception` (error_policy "continue") wraps `_process_data_internal`, which does + compute AND both writes (result table via `ResultCache.store`, pipeline table via + `add_pipeline_record`). WARNING log per row, run returns normally. +- Async swallow: `core/function_pod.py` `_FunctionPodBase.async_execute.process_one` catches + every per-row exception, logs at DEBUG (invisible), drops the row. +- Async pipeline-table write (`record_and_forward`) and operator LOG writes already raise, but + async callers get an `ExceptionGroup`. +- Operator-LOG failure aborts the run before `job.run`'s final flush, so pending pipeline rows + of upstream function nodes are never committed while their result rows are: the node reads + back fewer rows than were stored. +- Read failures raise on both orchestrators (not swallowed). Observer status/log writes are + deliberately best-effort and stay so. +- Re-run on a healthy store recovers in every case (missing rows recomputed; orphan results + hit the result cache). + +## Contract (Option A) + +1. A failed write of a node's results or records raises `StoreWriteError` and stops the run at + the first failure, regardless of `error_policy`, on both orchestrators. +2. `str(err)` names the node label and the table path (`"/".join(record_path)`), says rows + stored before the failure are kept and that re-running computes only what is missing. + `err.node_label`, `err.record_path` are attributes; `err.__cause__` is the backend error. +3. Before re-raising, `PipelineJob.run()` flushes pending writes best-effort; a flush failure + during that is logged and does not mask the original error. Nothing already stored is + rewritten; no hash, path or stored-schema change. +4. A failure of `PipelineJob.run()`'s final flush raises `StoreWriteError` (no node label). +5. `error_policy="continue"` still skips rows whose function raised (unchanged). + +## Files + +- Modify: `src/orcapod/errors.py` — add `StoreWriteError(RuntimeError)`. +- Modify: `src/orcapod/core/result_cache.py` — wrap `add_record` + auto-flush in `store`. +- Modify: `src/orcapod/core/nodes/function_node.py` — wrap `add_pipeline_record`'s write; + re-raise `StoreWriteError` in `execute`; tag node label in `async_execute`. +- Modify: `src/orcapod/core/function_pod.py` — `process_one` re-raises `StoreWriteError`. +- Modify: `src/orcapod/core/nodes/operator_node.py` — wrap `_store_output_stream`'s write. +- Modify: `src/orcapod/side_effects.py` — wrap `_write_invocation_row`; re-raise before the + `on_error="log"` handler. +- Modify: `src/orcapod/pipeline/async_orchestrator.py` — unwrap `StoreWriteError` from the + `ExceptionGroup`. +- Modify: `src/orcapod/pipeline/job.py` — best-effort flush on failure; wrap final flush. +- Create: `tests/test_pipeline/test_store_write_failure.py`. +- Modify: `docs/concepts/function-pods.md` (PipelineJob section), `README.md` execution + walkthrough, `DESIGN_ISSUES.md` (one entry appended at the end). + +## Task 1: Failing regression tests (public interface) + +**Files:** Create `tests/test_pipeline/test_store_write_failure.py`. + +Test double `FaultyStore`: wraps an `InMemoryArrowDatabase`; `at()` returns a wrapper sharing +one `Faults` config; fails `add_record`/`add_records` whose full path contains a marker +(`"rdb_v1"` result table, `"pdb_v1"` pipeline table, operator class name), optionally after N +successful matching writes, or fails `flush`. Records the failing relative record path so the +test can assert the message names it. `__getattr__` delegates everything else. + +All tests parametrized over `SyncPipelineOrchestrator` / `AsyncPipelineOrchestrator`. Until +the exception class is approved, assert with `pytest.raises(Exception)` plus +`not isinstance(err, BaseExceptionGroup)`, node label in `str(err)` and failing path in +`str(err)`; tighten to `StoreWriteError` in Task 2. + +- result-table write fails on the first row → run raises, names node + table. +- result-table write fails from the 2nd row → raises; a fresh job over the healthy store + recomputes only the missing rows (function call counter) and returns all rows. +- pipeline-table write fails → raises; fresh healthy re-run makes zero function calls. +- store primed with 2 rows, 3rd (new) row's write fails → raises; the 2 primed rows are still + read back by a fresh job (zero calls for them). +- `error_policy="fail_fast"` and `"continue"` both raise for a store failure. +- operator node with `cache_mode=LOG` whose write fails → raises naming the operator node; + upstream function node's pipeline rows are committed (read through a fresh job's + `nodes[label].get_all_records()` before running it). +- flush failure (`Faults(on_flush=True)`) → raises. +- SQLite trigger: function pod returning `pathlib.Path` on + `ConnectorArrowDatabase(SQLiteConnector(tmp_path / "x.db"))` → raises naming the node. +- Guard (should pass before and after): a function that raises for one row with + `error_policy="continue"` still returns normally and stores the other rows. + +- [ ] Write the tests; run `uv run pytest tests/test_pipeline/test_store_write_failure.py -v`; + confirm the store-failure tests FAIL (run returns normally / ExceptionGroup) and the + guard passes. Commit `test(pipeline): failing regression tests for swallowed store write + failures (ITL-878)`. + +## Task 2: `StoreWriteError` (after approval) + +```python +class StoreWriteError(RuntimeError): + """Raised when a node's results or records could not be written to the store. ...""" + + def __init__(self, record_path: tuple[str, ...], node_label: str | None = None) -> None: + self.record_path = tuple(record_path) + self.node_label = node_label + super().__init__(record_path, node_label) + + def __str__(self) -> str: + where = f"node {self.node_label!r}" if self.node_label else "the pipeline" + cause = self.__cause__ + reason = f": {type(cause).__name__}: {cause}" if cause is not None else "" + return ( + f"Could not write to the store for {where} (table " + f"{'/'.join(self.record_path)!r}){reason}. Rows stored before this failure are " + "kept; re-running computes only the rows that are missing." + ) +``` + +`node_label` is filled in by the node that owns the write when the write site does not know +it (`ResultCache`): `if exc.node_label is None: exc.node_label = self.label`. +Tighten Task 1 assertions to `pytest.raises(StoreWriteError)`. + +## Task 3: Write sites raise `StoreWriteError` + +- `ResultCache.store`: `try: add_record(...); if auto_flush: flush()` / + `except StoreWriteError: raise` / `except Exception as exc: raise + StoreWriteError(self._versioned_record_path) from exc`. +- `FunctionNode.add_pipeline_record`: same around `self._pipeline_database.add_record`, with + `node_label=self.label`, path `self._versioned_pipeline_path`. +- `OperatorNode._store_output_stream`: same around `add_records`, path + `self.node_identity_path`. +- `side_effects._write_invocation_row`: same, path `table_path` (label filled by caller). + +## Task 4: Per-row handlers re-raise + +- `FunctionNode.execute`: before `except Exception`, add + `except StoreWriteError as exc:` → set label, `ctx_obs.on_data_crash(...)`, + `ctx_obs.on_node_end(...)`, `raise`. +- `_FunctionPodBase.async_execute.process_one`: `except StoreWriteError as exc:` + → `obs.on_data_crash(...)`; `raise`. (Only this; no error_policy threading or log-level + change — those are the separately filed async error_policy issue.) +- `FunctionNode.async_execute`: wrap the DB-path `TaskGroup` with `except* StoreWriteError` + setting `node_label` on each leaf whose label is `None`, then re-raise the group. +- `side_effects` delivery: `except StoreWriteError: raise` before the `on_error` handler. + +## Task 5: Orchestrator and job + +- `AsyncPipelineOrchestrator._run_async`: `except* StoreWriteError as eg: raise + _first_leaf(eg)` where `_first_leaf` walks nested groups to the first `StoreWriteError`. +- `PipelineJob.run()`: wrap `effective_orchestrator.run(...)` in `try/except Exception:` + → `_flush_best_effort(pipeline_db, result_db)` (logs flush errors with `logger.exception`) + → `raise`. Final flush: `except Exception as exc: raise + StoreWriteError(pipeline_db.base_path) from exc`. + +- [ ] Run new tests (pass), then `uv run pytest tests/` (full), ruff check/format, + basedpyright exit code. Commit `fix(pipeline): a failed store write fails the run + (ITL-878)`. + +## Task 6: Docs and design log + +- `docs/concepts/function-pods.md` PipelineJob section: subsection "When the store cannot be + written" stating the contract (prose; no executed sample needs infrastructure). +- `README.md` execution walkthrough: note store errors are not skipped by `error_policy`. +- `DESIGN_ISSUES.md`: append one entry (resolved, Fix note, ITL-878). +- [ ] `uv run pytest tests/test_docs/`; commit `docs: state the store-write failure contract + (ITL-878)`. + +## Out of scope (reported, not fixed) + +- Async orchestrator ignores `error_policy`; async per-row function errors logged at DEBUG + (coordinator is filing). This plan touches `process_one` only to re-raise + `StoreWriteError`. +- Connector/Delta `flush` pop the pending batch before writing (coordinator is filing). +- ITL-877 (SQLite logical types). diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py new file mode 100644 index 000000000..50b17ea5a --- /dev/null +++ b/tests/test_pipeline/test_store_write_failure.py @@ -0,0 +1,311 @@ +"""A failed store write must fail ``PipelineJob.run()`` (ITL-878). + +Before the fix, a node whose write to the store failed was skipped like a row whose +function had raised: ``run()`` returned normally and the node read back fewer rows (often +zero). These tests drive ``PipelineJob`` through its public interface on both orchestrators, +with a test-double store that fails chosen writes, and with the SQLite trigger from the +report (a logical-type output on ``SQLiteConnector``). +""" + +from __future__ import annotations + +import pathlib +from dataclasses import dataclass, field +from typing import Any + +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.operators import SelectDataColumns +from orcapod.databases import ( + ConnectorArrowDatabase, + InMemoryArrowDatabase, + SQLiteConnector, +) +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.sources import DictSource +from orcapod.types import CacheMode + +JOB_NAME = "itl878" + +# Markers that select a table by its stored path. +RESULT_TABLE = "rdb_v1" # a function node's result table +PIPELINE_TABLE = "pdb_v1" # a function node's pipeline (tag) table +OPERATOR_TABLE = "SelectDataColumns" # the operator node's LOG table + + +# --------------------------------------------------------------------------- +# Test-double store +# --------------------------------------------------------------------------- + + +@dataclass +class Faults: + """Which writes a ``FaultyStore`` fails; shared by every view of one store.""" + + write_marker: str | None = None + succeed_first: int = 0 + on_flush: bool = False + matching_writes: int = 0 + failed_paths: list[str] = field(default_factory=list) + + +class FaultyStore: + """An ``InMemoryArrowDatabase`` that raises on chosen writes or on flush.""" + + def __init__(self, inner: Any, faults: Faults) -> None: + self._inner = inner + self._faults = faults + + def at(self, *path_components: str) -> FaultyStore: + return FaultyStore(self._inner.at(*path_components), self._faults) + + def _check_write(self, record_path: tuple[str, ...]) -> None: + faults = self._faults + full_path = "/".join(self._inner.base_path + tuple(record_path)) + if faults.write_marker is None or faults.write_marker not in full_path: + return + faults.matching_writes += 1 + if faults.matching_writes > faults.succeed_first: + faults.failed_paths.append("/".join(record_path)) + raise OSError(f"disk unavailable while writing {full_path}") + + def add_record( + self, record_path, record_id, record, skip_duplicates=False, flush=False + ): + self._check_write(record_path) + self._inner.add_record( + record_path, record_id, record, skip_duplicates=skip_duplicates, flush=flush + ) + + def add_records( + self, + record_path, + records, + record_id_column=None, + skip_duplicates=False, + flush=False, + ): + self._check_write(record_path) + self._inner.add_records( + record_path, + records, + record_id_column=record_id_column, + skip_duplicates=skip_duplicates, + flush=flush, + ) + + def flush(self) -> None: + if self._faults.on_flush: + raise OSError("disk unavailable while flushing") + self._inner.flush() + + def __getattr__(self, name: str) -> Any: + return getattr(self._inner, name) + + +# --------------------------------------------------------------------------- +# Pipeline under test +# --------------------------------------------------------------------------- + +ROWS = [{"k": f"r{i}", "x": i} for i in range(3)] + +calls: list[int] = [] + + +@function_pod(output_keys="double") +def double(x: int) -> int: + calls.append(x) + return x * 2 + + +@function_pod(output_keys="double") +def double_or_fail(x: int) -> int: + if x == 1: + raise RuntimeError("function failed for x == 1") + return x * 2 + + +@function_pod(output_keys="where") +def to_path(x: int) -> pathlib.Path: + return pathlib.Path(f"/data/{x}.txt") + + +def build_job(store: Any, rows: list[dict], *, pod=double, with_operator: bool = False): + job = PipelineJob(name=JOB_NAME, store=store) + with job: + source = DictSource(data=rows, tag_columns=["k"], source_id="src") + out = pod.pod(source, label="dbl") + if with_operator: + SelectDataColumns(["double"])(out, label="sel") + if with_operator: + # OperatorNode.attach_databases is the public way to turn on LOG mode. + job.nodes["sel"].attach_databases( + pipeline_database=store.at(JOB_NAME), cache_mode=CacheMode.LOG + ) + return job + + +@pytest.fixture(autouse=True) +def _reset_calls(): + calls.clear() + + +@pytest.fixture(params=["sync", "async"]) +def orchestrator_cls(request): + return {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator}[ + request.param + ] + + +def assert_store_error(err: BaseException, node_label: str | None, faults: Faults): + """The run's error is a single error (not a group) naming the node and the table.""" + assert not isinstance(err, BaseExceptionGroup), f"{type(err).__name__}: {err}" + message = str(err) + if node_label is not None: + assert repr(node_label) in message, message + if faults.failed_paths: + assert faults.failed_paths[0] in message, message + + +def stored_rows(store: Any, label: str = "dbl") -> int: + """Rows a fresh job over ``store`` reads back for ``label`` without running.""" + records = build_job(store, ROWS).nodes[label].get_all_records() + return 0 if records is None else records.num_rows + + +# --------------------------------------------------------------------------- +# Function node writes +# --------------------------------------------------------------------------- + + +class TestFunctionNodeWriteFailure: + def test_result_table_failure_on_first_row_raises(self, orchestrator_cls): + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS) + + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, "dbl", faults) + + def test_result_table_failure_on_later_row_keeps_earlier_rows( + self, orchestrator_cls + ): + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=RESULT_TABLE, succeed_first=1) + job = build_job(FaultyStore(inner, faults), ROWS) + + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, "dbl", faults) + assert stored_rows(inner) == 1 + + # A re-run on a healthy store computes only the rows that are missing. + calls.clear() + rerun = build_job(inner, ROWS) + rerun.run(orchestrator=orchestrator_cls()) + assert rerun.nodes["dbl"].as_table().num_rows == 3 + assert len(calls) == 2 + + def test_pipeline_table_failure_raises(self, orchestrator_cls): + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=PIPELINE_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS) + + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, "dbl", faults) + + # The result that was computed is stored; a healthy re-run reuses it. + calls.clear() + rerun = build_job(inner, ROWS) + rerun.run(orchestrator=orchestrator_cls()) + assert rerun.nodes["dbl"].as_table().num_rows == 3 + assert len(calls) == 2 + + def test_failure_on_new_row_keeps_previously_stored_rows(self, orchestrator_cls): + inner = InMemoryArrowDatabase() + build_job(inner, ROWS[:2]).run(orchestrator=orchestrator_cls()) + assert stored_rows(inner) == 2 + + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS) + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, "dbl", faults) + assert stored_rows(inner) == 2 + + @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) + def test_store_failure_raises_under_any_error_policy( + self, orchestrator_cls, error_policy + ): + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls(error_policy=error_policy)) + + assert_store_error(exc_info.value, "dbl", faults) + + def test_flush_failure_raises(self, orchestrator_cls): + faults = Faults(on_flush=True) + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, None, faults) + + def test_function_failure_is_still_skipped_under_continue(self, orchestrator_cls): + """``error_policy="continue"`` keeps skipping rows whose function raised.""" + inner = InMemoryArrowDatabase() + job = build_job(inner, ROWS, pod=double_or_fail) + + job.run(orchestrator=orchestrator_cls(error_policy="continue")) + + assert sorted(job.nodes["dbl"].as_table()["k"].to_pylist()) == ["r0", "r2"] + + +# --------------------------------------------------------------------------- +# Operator node writes (cache_mode=LOG) +# --------------------------------------------------------------------------- + + +class TestOperatorNodeWriteFailure: + def test_operator_log_failure_raises_and_commits_upstream_rows( + self, orchestrator_cls + ): + inner = InMemoryArrowDatabase() + faults = Faults(write_marker=OPERATOR_TABLE) + job = build_job(FaultyStore(inner, faults), ROWS, with_operator=True) + + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error(exc_info.value, "sel", faults) + # The upstream function node wrote all its rows before the operator failed; + # none of them may look dropped. + assert stored_rows(inner) == 3 + + +# --------------------------------------------------------------------------- +# The reported trigger: a logical-type output on SQLite (ITL-877 is the limit itself) +# --------------------------------------------------------------------------- + + +class TestSQLiteTrigger: + def test_logical_type_output_on_sqlite_raises(self, orchestrator_cls, tmp_path): + store = ConnectorArrowDatabase(SQLiteConnector(str(tmp_path / "store.db"))) + job = build_job(store, ROWS, pod=to_path) + + with pytest.raises(Exception) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert not isinstance(exc_info.value, BaseExceptionGroup) + assert "'dbl'" in str(exc_info.value) + assert "does not preserve Arrow metadata" in str(exc_info.value) From 4bbb24c0dfa5f727c992b188c164d048501573d5 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 19:17:15 -0800 Subject: [PATCH 02/15] fix(pipeline): a failed store write fails the run (ITL-878) A failed write of a node's results or records was caught by the per-row error handling meant for function failures: PipelineJob.run() returned normally and the node read back fewer rows (none on the async path, which logged only at DEBUG). Every node write site (result cache, function-node pipeline records, operator LOG output, side-effect invocation log) now raises the new public orcapod.errors.StoreWriteError, chained from the backend error and naming the node, the row's tag and the table. The per-row handlers re-raise it whatever error_policy is, so the run stops at the first failed write; the async orchestrator unwraps it from the TaskGroup exception group. PipelineJob.run() flushes the writes that succeeded before re-raising, and wraps a failure of its final flush. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/function_pod.py | 7 ++ src/orcapod/core/nodes/function_node.py | 46 +++++++--- src/orcapod/core/nodes/operator_node.py | 24 ++++-- src/orcapod/core/result_cache.py | 22 ++--- src/orcapod/errors.py | 85 +++++++++++++++++++ src/orcapod/pipeline/async_orchestrator.py | 5 ++ src/orcapod/pipeline/job.py | 58 ++++++++++--- src/orcapod/side_effects.py | 20 +++-- .../test_pipeline/test_store_write_failure.py | 66 ++++++++------ 9 files changed, 264 insertions(+), 69 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index ba7e3b1d4..aa0d15634 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -14,6 +14,7 @@ from orcapod.core.data_function import CachedDataFunction, PythonDataFunction from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER +from orcapod.errors import StoreWriteError from orcapod.protocols.core_protocols import ( ArgumentGroup, FunctionPodProtocol, @@ -587,6 +588,12 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: out_tag, result_data = await self._async_invoke_with_hooks( tag, data, logger=pkt_logger, run_id=run_id ) + except StoreWriteError as exc: + # A failed store write fails the run; only rows whose + # function raised are skipped. + exc.add_context(tag=tag.as_dict()) + obs.on_data_crash(pod_label, tag, data, exc) + raise except Exception as exc: logger.debug( "Data processing failed, skipping: %s", exc, exc_info=True diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 7af980049..8ed336d8b 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -35,6 +35,7 @@ EphemeralResultMissingError, PipelineJobRequiredError, SchemaVersionError, + StoreWriteError, ) from orcapod.protocols.core_protocols import ( FunctionPodProtocol, @@ -1164,14 +1165,20 @@ def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: # Internal helpers # ------------------------------------------------------------------ - def _require_pipeline_database(self) -> None: - """Raise a clear RuntimeError if no pipeline database is attached. + def _require_pipeline_database(self) -> ArrowDatabaseProtocol: + """Return the attached pipeline database, raising if there is none. Called at the top of methods that unconditionally access ``self._pipeline_database``. Provides an actionable error message instead of an opaque ``AttributeError: 'NoneType' object has no attribute ...`` when a definition-level pipeline is executed without supplying a database. + + Returns: + The attached pipeline database. + + Raises: + RuntimeError: If no pipeline database is attached. """ if self._pipeline_database is None: raise RuntimeError( @@ -1179,6 +1186,7 @@ def _require_pipeline_database(self) -> None: "Either construct the pipeline with a pipeline_database argument, " "or supply one via Pipeline.load(..., pipeline_database=)." ) + return self._pipeline_database # ------------------------------------------------------------------ # as_node — return the lightweight FunctionNode equivalent @@ -1294,6 +1302,13 @@ def execute( tag_out, result = self._process_data_internal( tag, data, logger=pkt_logger, run_id=run_id ) + except StoreWriteError as exc: + # A failed store write fails the run whatever the error policy: + # error_policy only decides about rows whose function raised. + exc.add_context(node_label=node_label, tag=tag.as_dict()) + ctx_obs.on_data_crash(node_label, tag, data, exc) + ctx_obs.on_node_end(node_label, node_hash) + raise except Exception as exc: logger.warning( "Data execution failed in %s: %s", @@ -1701,14 +1716,14 @@ def add_pipeline_record( so that downstream ``EmptyData`` flow-through can locate its cached result keyed by the same hash (= the downstream's ``INPUT_DATA_HASH_COL``). """ - self._require_pipeline_database() + pipeline_database = self._require_pipeline_database() self._ensure_pdb_schema() base_entry_id = self.compute_base_entry_id(tag, input_data) # Determine the next recomputation index by querying all existing rows # for this base_entry_id. No await is used here, so within a single-threaded # asyncio event loop this read-then-write sequence is uninterrupted. - existing = self._pipeline_database.get_records_with_column_value( + existing = pipeline_database.get_records_with_column_value( self._versioned_pipeline_path, {_PIPELINE_BASE_ENTRY_ID_COL: base_entry_id}, ) @@ -1773,12 +1788,19 @@ def add_pipeline_record( meta_table, ) - self._pipeline_database.add_record( - self._versioned_pipeline_path, - versioned_entry_id, - combined_record, - skip_duplicates=True, - ) + try: + pipeline_database.add_record( + self._versioned_pipeline_path, + versioned_entry_id, + combined_record, + skip_duplicates=True, + ) + except Exception as exc: + raise StoreWriteError( + self._versioned_pipeline_path, + node_label=self.label, + tag=tag.as_dict(), + ) from exc # ------------------------------------------------------------------ # Records and sources @@ -2517,6 +2539,10 @@ def on_data_crash( clean = tag.drop_meta_columns( _TAG_NODE_INPUT_REF, ignore_missing=True ) + if isinstance(error, StoreWriteError): + # The pod re-raises a failed store write after this + # call; name the node it belongs to. + error.add_context(node_label=node_label) ctx_obs.on_data_crash(node_label, clean, data, error) # Clean up the dangling input_store entry so it doesn't # accumulate memory during long-running calls with errors. diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index fa81a4469..f343ba67f 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -29,7 +29,7 @@ from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER -from orcapod.errors import PipelineJobRequiredError +from orcapod.errors import PipelineJobRequiredError, StoreWriteError from orcapod.protocols.core_protocols import ( DataProtocol, StreamProtocol, @@ -798,6 +798,11 @@ def _filter_by_content_hash(self, table: pa.Table) -> pa.Table: def _store_output_stream(self, stream: StreamProtocol) -> None: """Materialize stream and store in the pipeline database with per-row dedup.""" + pipeline_database = self._pipeline_database + if pipeline_database is None: + raise RuntimeError( + f"OperatorJobNode {self.label!r} has no pipeline database attached." + ) output_table = stream.as_table( columns={"source": True, "system_tags": True}, ) @@ -839,12 +844,17 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: # the hash), so rows from different runs with identical output will have # distinct record IDs and both be stored. skip_duplicates=True still # deduplicates exact re-runs of the same node within a single run. - self._pipeline_database.add_records( - self.node_identity_path, - output_table, - record_id_column=self.HASH_COLUMN_NAME, - skip_duplicates=True, - ) + try: + pipeline_database.add_records( + self.node_identity_path, + output_table, + record_id_column=self.HASH_COLUMN_NAME, + skip_duplicates=True, + ) + except Exception as exc: + raise StoreWriteError( + self.node_identity_path, node_label=self.label + ) from exc self._cached_output_table = output_table.drop( [self.HASH_COLUMN_NAME, constants.NODE_CONTENT_HASH_COL] diff --git a/src/orcapod/core/result_cache.py b/src/orcapod/core/result_cache.py index 970a22418..e4dab8a96 100644 --- a/src/orcapod/core/result_cache.py +++ b/src/orcapod/core/result_cache.py @@ -12,7 +12,7 @@ from datetime import datetime, timezone from typing import TYPE_CHECKING, Any -from orcapod.errors import SchemaVersionError +from orcapod.errors import SchemaVersionError, StoreWriteError from orcapod.protocols.core_protocols import DataProtocol from orcapod.protocols.database_protocols import ArrowDatabaseProtocol from orcapod.system_constants import constants, RESULT_DB_SCHEMA_VERSION @@ -287,15 +287,17 @@ def store( pa.array([datetime.now(timezone.utc)], type=pa.timestamp("us", tz="UTC")), ) - self._result_database.add_record( - self._versioned_record_path, - output_data.datagram_uuid.bytes, - data_table, - skip_duplicates=skip_duplicates, - ) - - if self._auto_flush: - self._result_database.flush() + try: + self._result_database.add_record( + self._versioned_record_path, + output_data.datagram_uuid.bytes, + data_table, + skip_duplicates=skip_duplicates, + ) + if self._auto_flush: + self._result_database.flush() + except Exception as exc: + raise StoreWriteError(self._versioned_record_path) from exc def get_all_records( self, include_system_columns: bool = False diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 81c653627..e50d2582a 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -1,3 +1,7 @@ +from collections.abc import Mapping +from typing import Any + + class InputValidationError(Exception): """ Exception raised when the inputs are not valid. @@ -195,3 +199,84 @@ def __init__( f"node={node_identity_path!r} " f"cached_content_hash={cached_content_hash!r}" ) + + +class StoreWriteError(RuntimeError): + """Raised when a node's results or records could not be written to the store. + + A failed store write always fails the run, whatever the orchestrator's + ``error_policy``: ``error_policy="continue"`` skips rows whose *function* + raised, never rows that could not be stored. The run stops at the first + failed write. Rows stored before the failure are kept, and re-running the + job computes only the rows that are missing. + + The backend's own exception is chained as ``__cause__``. + + Attributes: + record_path: Path of the table the write was for, relative to the + database view the writer holds. + node_label: Label of the node whose write failed, or ``None`` when the + failure is not tied to one node (a flush at the end of a run). + tag: Tag values of the row being stored, or ``None`` when the write + covered many rows (an operator's output) or no row. + """ + + def __init__( + self, + record_path: tuple[str, ...], + node_label: str | None = None, + tag: Mapping[str, Any] | None = None, + ) -> None: + self.record_path = tuple(record_path) + self.node_label = node_label + self.tag = dict(tag) if tag is not None else None + super().__init__(self.record_path, node_label, tag) + + @classmethod + def in_group(cls, group: BaseExceptionGroup) -> "list[StoreWriteError]": + """Return every ``StoreWriteError`` in an exception group, nested ones included. + + Args: + group: An exception group, such as the one an ``asyncio.TaskGroup`` + raises. + + Returns: + The ``StoreWriteError`` leaves in the order the group holds them. + """ + found: "list[StoreWriteError]" = [] + for exc in group.exceptions: + if isinstance(exc, BaseExceptionGroup): + found.extend(cls.in_group(exc)) + elif isinstance(exc, cls): + found.append(exc) + return found + + def add_context( + self, node_label: str | None = None, tag: Mapping[str, Any] | None = None + ) -> None: + """Fill in the node label and row tag if the write site did not know them. + + Args: + node_label: Label of the node that owns the write. + tag: Tag values of the row being stored. + """ + if self.node_label is None: + self.node_label = node_label + if self.tag is None and tag is not None: + self.tag = dict(tag) + + def __str__(self) -> str: + where = f"node {self.node_label!r}" if self.node_label else "the pipeline" + row = f" for row {self.tag!r}" if self.tag else "" + cause = self.__cause__ + reason = ( + f": {type(cause).__name__}: {str(cause).rstrip('.')}" + if cause is not None + else "" + ) + return ( + f"Could not write to the store for {where}{row} " + f"(table {'/'.join(self.record_path)!r}){reason}. The run was stopped. " + "Rows stored before this failure are kept; re-running the job computes " + "only the rows that are missing." + ) diff --git a/src/orcapod/pipeline/async_orchestrator.py b/src/orcapod/pipeline/async_orchestrator.py index 4a75564c5..03205c5eb 100644 --- a/src/orcapod/pipeline/async_orchestrator.py +++ b/src/orcapod/pipeline/async_orchestrator.py @@ -14,6 +14,7 @@ from typing import TYPE_CHECKING, Any, Literal from orcapod.channels import BroadcastChannel, Channel +from orcapod.errors import StoreWriteError from orcapod.pipeline.result import OrchestratorResult from orcapod.protocols.node_protocols import ( is_function_node, @@ -250,6 +251,10 @@ async def _run_async( return OrchestratorResult( node_outputs=collectors if materialize_results else {} ) + except* StoreWriteError as group: + # A failed store write stops the run. Raise it on its own, as the sync + # orchestrator does, rather than inside the TaskGroup's exception group. + raise StoreWriteError.in_group(group)[0] finally: effective_observer.on_run_end(run_id) diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 8b7f76ffe..dd884dfe6 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -12,6 +12,7 @@ from orcapod.core.nodes.source_node import SourceJobNode from orcapod.side_effects import SideEffectJobNode from orcapod.core.tracker import AutoRegisteringContextBasedTracker +from orcapod.errors import StoreWriteError from orcapod.pipeline.base import AbstractPipelineBase from orcapod.pipeline.dag import OrcaDAG from orcapod.protocols import core_protocols as cp @@ -30,6 +31,26 @@ logger = logging.getLogger(__name__) +def _flush_best_effort(*databases: "ArrowDatabaseProtocol") -> None: + """Flush each database, logging instead of raising if a flush fails. + + Used when a run has already failed: the run's own error is the one to report, + and a second failure while committing the writes that did succeed must not + replace it. + + Args: + *databases: The databases to flush. + """ + for database in databases: + try: + database.flush() + except Exception: + logger.exception( + "Could not flush pending writes after the run failed; " + "the rows they hold were not stored" + ) + + class PipelineJob(AbstractPipelineBase[JobNode]): """Pipeline + source bindings + execution context. @@ -736,6 +757,11 @@ def run( Raises: ValueError: If no store is set. RuntimeError: If no pipeline has been compiled. + StoreWriteError: If a node's results or records could not be written + to the store, whatever the orchestrator's ``error_policy``. The run + stops at the first failed write. Writes that succeeded before it + are flushed first, so rows stored before the failure are kept, and + re-running the job computes only the rows that are missing. """ import hashlib import uuid @@ -827,20 +853,30 @@ def run( snapshot_hash = hashlib.sha256("\n".join(leaf_hashes).encode()).hexdigest()[:16] pipeline_uri = "/".join(self._name) + "@" + snapshot_hash + pipeline_db = self._store.at(*self._name) + result_db = pipeline_db.at("_result") + effective_orchestrator = orchestrator or SyncPipelineOrchestrator() - effective_orchestrator.run( - exec_dag, - observer=effective_observer, - run_id=run_id, - pipeline_uri=pipeline_uri, - materialize_results=materialize_results, - ) + try: + effective_orchestrator.run( + exec_dag, + observer=effective_observer, + run_id=run_id, + pipeline_uri=pipeline_uri, + materialize_results=materialize_results, + ) + except Exception: + # Commit the writes that did succeed before the failure, so their rows + # do not look dropped, then report the failure itself. + _flush_best_effort(pipeline_db, result_db) + raise # Flush databases. - pipeline_db = self._store.at(*self._name) - result_db = pipeline_db.at("_result") - pipeline_db.flush() - result_db.flush() + try: + pipeline_db.flush() + result_db.flush() + except Exception as exc: + raise StoreWriteError(pipeline_db.base_path) from exc # Mutate self in place. self._has_run = True diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index c774d49ab..8b0c0ba16 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -19,6 +19,7 @@ from orcapod.core.nodes.function_node import _build_record_id_preimage from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER +from orcapod.errors import StoreWriteError from orcapod.invocation import ( InvocationContext, InvocationHashConfig, @@ -269,6 +270,10 @@ def _execute_side_effect_row( run_id=run_id, ) return (tag, data) + except StoreWriteError as exc: + # A failed store write fails the run, even under on_error="log". + exc.add_context(node_label=pod_name, tag=tag.as_dict()) + raise except Exception as exc: if pod_config.on_error == "raise": raise @@ -314,12 +319,15 @@ def _write_invocation_row( ), } ) - pipeline_database.add_record( - table_path, - record_id, - record, - skip_duplicates=True, - ) + try: + pipeline_database.add_record( + table_path, + record_id, + record, + skip_duplicates=True, + ) + except Exception as exc: + raise StoreWriteError(table_path) from exc # --------------------------------------------------------------------------- diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index 50b17ea5a..7820eec20 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -17,6 +17,7 @@ from orcapod import PipelineJob, function_pod from orcapod.core.operators import SelectDataColumns +from orcapod.errors import StoreWriteError from orcapod.databases import ( ConnectorArrowDatabase, InMemoryArrowDatabase, @@ -131,7 +132,9 @@ def to_path(x: int) -> pathlib.Path: return pathlib.Path(f"/data/{x}.txt") -def build_job(store: Any, rows: list[dict], *, pod=double, with_operator: bool = False): +def build_job( + store: Any, rows: list[dict[str, Any]], *, pod=double, with_operator: bool = False +): job = PipelineJob(name=JOB_NAME, store=store) with job: source = DictSource(data=rows, tag_columns=["k"], source_id="src") @@ -158,14 +161,27 @@ def orchestrator_cls(request): ] -def assert_store_error(err: BaseException, node_label: str | None, faults: Faults): - """The run's error is a single error (not a group) naming the node and the table.""" - assert not isinstance(err, BaseExceptionGroup), f"{type(err).__name__}: {err}" +def assert_store_error( + err: BaseException, + faults: Faults, + *, + node_label: str | None, + names_row: bool, +) -> None: + """The run's error is a plain ``StoreWriteError`` naming node, row and table.""" + assert isinstance(err, StoreWriteError), f"{type(err).__name__}: {err}" message = str(err) if node_label is not None: - assert repr(node_label) in message, message + assert err.node_label == node_label + assert f"node {node_label!r}" in message, message + if names_row: + assert err.tag is not None and err.tag["k"] in {row["k"] for row in ROWS} + assert f"for row {err.tag!r}" in message, message if faults.failed_paths: + assert "/".join(err.record_path) == faults.failed_paths[0] assert faults.failed_paths[0] in message, message + assert isinstance(err.__cause__, OSError) + assert "rows that are missing" in message def stored_rows(store: Any, label: str = "dbl") -> int: @@ -185,10 +201,10 @@ def test_result_table_failure_on_first_row_raises(self, orchestrator_cls): faults = Faults(write_marker=RESULT_TABLE) job = build_job(FaultyStore(inner, faults), ROWS) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert_store_error(exc_info.value, "dbl", faults) + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) def test_result_table_failure_on_later_row_keeps_earlier_rows( self, orchestrator_cls @@ -197,10 +213,10 @@ def test_result_table_failure_on_later_row_keeps_earlier_rows( faults = Faults(write_marker=RESULT_TABLE, succeed_first=1) job = build_job(FaultyStore(inner, faults), ROWS) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert_store_error(exc_info.value, "dbl", faults) + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) assert stored_rows(inner) == 1 # A re-run on a healthy store computes only the rows that are missing. @@ -215,17 +231,17 @@ def test_pipeline_table_failure_raises(self, orchestrator_cls): faults = Faults(write_marker=PIPELINE_TABLE) job = build_job(FaultyStore(inner, faults), ROWS) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert_store_error(exc_info.value, "dbl", faults) + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) - # The result that was computed is stored; a healthy re-run reuses it. - calls.clear() + # Every result computed before the failure was stored, so a healthy re-run + # reuses it: across both runs each row is computed exactly once. rerun = build_job(inner, ROWS) rerun.run(orchestrator=orchestrator_cls()) assert rerun.nodes["dbl"].as_table().num_rows == 3 - assert len(calls) == 2 + assert sorted(calls) == [0, 1, 2] def test_failure_on_new_row_keeps_previously_stored_rows(self, orchestrator_cls): inner = InMemoryArrowDatabase() @@ -234,10 +250,10 @@ def test_failure_on_new_row_keeps_previously_stored_rows(self, orchestrator_cls) faults = Faults(write_marker=RESULT_TABLE) job = build_job(FaultyStore(inner, faults), ROWS) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert_store_error(exc_info.value, "dbl", faults) + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) assert stored_rows(inner) == 2 @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) @@ -247,19 +263,19 @@ def test_store_failure_raises_under_any_error_policy( faults = Faults(write_marker=RESULT_TABLE) job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls(error_policy=error_policy)) - assert_store_error(exc_info.value, "dbl", faults) + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) def test_flush_failure_raises(self, orchestrator_cls): faults = Faults(on_flush=True) job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert_store_error(exc_info.value, None, faults) + assert_store_error(exc_info.value, faults, node_label=None, names_row=False) def test_function_failure_is_still_skipped_under_continue(self, orchestrator_cls): """``error_policy="continue"`` keeps skipping rows whose function raised.""" @@ -284,10 +300,10 @@ def test_operator_log_failure_raises_and_commits_upstream_rows( faults = Faults(write_marker=OPERATOR_TABLE) job = build_job(FaultyStore(inner, faults), ROWS, with_operator=True) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert_store_error(exc_info.value, "sel", faults) + assert_store_error(exc_info.value, faults, node_label="sel", names_row=False) # The upstream function node wrote all its rows before the operator failed; # none of them may look dropped. assert stored_rows(inner) == 3 @@ -303,9 +319,9 @@ def test_logical_type_output_on_sqlite_raises(self, orchestrator_cls, tmp_path): store = ConnectorArrowDatabase(SQLiteConnector(str(tmp_path / "store.db"))) job = build_job(store, ROWS, pod=to_path) - with pytest.raises(Exception) as exc_info: + with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert not isinstance(exc_info.value, BaseExceptionGroup) - assert "'dbl'" in str(exc_info.value) + assert exc_info.value.node_label == "dbl" + assert "node 'dbl'" in str(exc_info.value) assert "does not preserve Arrow metadata" in str(exc_info.value) From e0a472a10d1aef24dade06139265ba7c85b91b26 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 19:17:49 -0800 Subject: [PATCH 03/15] docs: state the store-write failure contract (ITL-878) Co-Authored-By: Claude Opus 5.5 --- DESIGN_ISSUES.md | 34 ++++++++++++++++++++++++++++++++++ README.md | 5 ++++- docs/concepts/function-pods.md | 20 ++++++++++++++++++++ 3 files changed, 58 insertions(+), 1 deletion(-) diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index a6f9b59ed..5b0186798 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2171,3 +2171,37 @@ equals the materialized one. Hashes of pipelines that already worked are unchang Regressions: `tests/test_core/operators/test_chained_aggregation.py`. Open: on the async barrier path, per-row source info is still row 0's (ITL-845); that assertion is a strict xfail. + +--- + +## `src/orcapod/pipeline/job.py`, orchestrators, node write paths — store write failures + +### SW1 — A failed store write did not fail `PipelineJob.run()` +**Status:** resolved +**Severity:** critical +**Issue:** ITL-878 + +A node's write to the store ran inside the per-row handler meant for function failures: +`FunctionNode.execute` (sync) and `_FunctionPodBase.async_execute.process_one` (async) wrap +compute *and* the result-table and pipeline-table writes in one `except Exception`. With the +default `error_policy="continue"` a failed write was skipped like a crashed function: the sync +path logged a WARNING, the async path logged only at DEBUG, `job.run()` returned normally and +the node read back fewer rows (none when every write failed, e.g. a logical-type output on +`SQLiteConnector`, ITL-877). An operator LOG write failure did raise, but before `job.run()`'s +final flush, so the upstream function node's pending pipeline rows were never committed while +its result rows were; async callers got an `ExceptionGroup`. + +**Fix:** Contract, for now (developer, November 10, 2026): a failed store write always fails +the run, whatever `error_policy` is. Every node write site (`ResultCache.store`, +`FunctionNode.add_pipeline_record`, `OperatorNode._store_output_stream`, the side-effect +invocation log) raises `orcapod.errors.StoreWriteError`, chained from the backend error and +naming the node, the row's tag and the table. The per-row handlers re-raise it, so the run +stops at the first failed write; `error_policy` still decides only about rows whose function +raised. `AsyncPipelineOrchestrator` unwraps it from the `TaskGroup` exception group, so both +orchestrators raise the same error. `PipelineJob.run()` flushes the writes that succeeded +before re-raising (a flush failure there is logged, not raised over the original), and wraps a +failure of its final flush. Nothing stored is rewritten; no hash, path or schema change. An +opt-in mode that tolerates store failures is a possible later option. Regressions: +`tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async +orchestrator ignores `error_policy` (ITL-879); connector and Delta `flush` drop a batch whose +write fails (ITL-880). diff --git a/README.md b/README.md index 3a48c48c7..7e440f148 100644 --- a/README.md +++ b/README.md @@ -198,7 +198,10 @@ While the following is subject to change based on future development, it represe │ on success: │ ctx_observer.on_data_end(node_label, tag, data, result, cached=False) │ emit (tag_out, result) downstream - │ on exception: + │ on StoreWriteError (a result or record could not be stored): + │ ctx_observer.on_data_crash(node_label, tag, data, exc) + │ raise, whatever error_policy is — the run stops (ITL-878) + │ on any other exception (the function failed): │ ctx_observer.on_data_crash(node_label, tag, data, exc) │ if error_policy == "fail_fast": raise │ otherwise: skip this data, continue diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index b8618052c..f92bcefac 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -228,6 +228,26 @@ A node from a job that has run also provides: - **`as_source()`** -- a `DerivedSource` backed by those stored records, usable as the input of another pipeline +### When the store cannot be written + +A failed write to the store always fails the run. If a node's results or records cannot be +stored -- the disk is full, the connection drops, or the database cannot hold one of the +output types -- `job.run()` raises `StoreWriteError` (from `orcapod.errors`) at the first +failed write, on both the synchronous and the asynchronous orchestrator. The error names the +node, the row being stored and the table, and the database's own exception is chained as its +cause. A run that returns normally has stored every result it computed. + +This holds whatever `error_policy` the orchestrator was given. `error_policy="continue"` (the +default) skips rows whose *function* raised and keeps going; it never skips a row because it +could not be stored. + +After the failure, rows stored before it are kept: `job.run()` commits the writes that +succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is +fixed, run the job again; only the rows that are missing are computed. + +This is the contract for now. An opt-in mode that tolerates store failures may be added +later. + ## Multiple input streams If you pass multiple streams to a function pod, they are automatically joined (using From 852ca05a4887b76ee43d647a40556a5e782b626c Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:05:07 -0800 Subject: [PATCH 04/15] fix(pipeline): keep a store write error intact on its way out (ITL-878) Addresses CodeRabbit's review of #287: - A fail-loud post-run hook or an observer callback that raises while a StoreWriteError propagates is logged at WARNING and noted on the store error (orcapod.utils.store_errors), not raised in its place. A node function that runs another cached pod can no longer turn that pod's store failure into a row skipped under error_policy="continue". - The async orchestrator raises a single StoreWriteError from a mixed exception group: the store error of the node first in topological order (the one the sync orchestrator meets first). Other failures are noted on it, other store errors are logged at WARNING, and the group stays its __context__. - test_pipeline_table_failure_raises[async] asserted an exact call count that a row still computing when the run stopped can exceed (13/40 failures on 3.13 before this change); it now asserts the failing row's stored result is reused. Co-Authored-By: Claude Opus 5.5 --- DESIGN_ISSUES.md | 9 +- docs/concepts/function-pods.md | 7 + src/orcapod/core/function_pod.py | 14 +- src/orcapod/core/nodes/function_node.py | 7 +- src/orcapod/pipeline/async_orchestrator.py | 68 +++++- src/orcapod/pipeline/sync_orchestrator.py | 10 +- src/orcapod/utils/store_errors.py | 67 ++++++ .../2026-11-10-itl-878-store-write-failure.md | 35 +++ .../test_pipeline/test_store_write_failure.py | 218 +++++++++++++++++- 9 files changed, 423 insertions(+), 12 deletions(-) create mode 100644 src/orcapod/utils/store_errors.py diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 5b0186798..ee8075498 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2201,7 +2201,12 @@ raised. `AsyncPipelineOrchestrator` unwraps it from the `TaskGroup` exception gr orchestrators raise the same error. `PipelineJob.run()` flushes the writes that succeeded before re-raising (a flush failure there is logged, not raised over the original), and wraps a failure of its final flush. Nothing stored is rewritten; no hash, path or schema change. An -opt-in mode that tolerates store failures is a possible later option. Regressions: -`tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async +opt-in mode that tolerates store failures is a possible later option. A fail-loud post-run +hook or an observer callback that raises while the store error propagates is logged and noted +on it (`orcapod.utils.store_errors`), not raised in its place, so a node function that runs +another cached pod cannot turn that pod's store failure into a skipped row. When concurrent +node tasks fail together, the async orchestrator raises the store error of the node first in +topological order, notes the other failures on it and logs other store errors at WARNING. +Regressions: `tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async orchestrator ignores `error_policy` (ITL-879); connector and Delta `flush` drop a batch whose write fails (ITL-880). diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index f92bcefac..e681ba565 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -241,6 +241,13 @@ This holds whatever `error_policy` the orchestrator was given. `error_policy="co default) skips rows whose *function* raised and keeps going; it never skips a row because it could not be stored. +The error you get is always one plain `StoreWriteError`, never an exception group and never +another exception raised in its place. A post-run hook or observer callback that fails while +the store error is on its way out is logged and recorded as a note on the error +(`err.__notes__`). When several nodes fail at once on the asynchronous orchestrator, you get +the store error of the node that comes first in the pipeline, the one the synchronous +orchestrator would have stopped at; the other failures are logged and noted on it. + After the failure, rows stored before it are kept: `job.run()` commits the writes that succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is fixed, run the job again; only the rows that are missing are computed. diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index aa0d15634..e716f367d 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -15,6 +15,7 @@ from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import StoreWriteError +from orcapod.utils.store_errors import keep_store_error, note_callback_failure from orcapod.protocols.core_protocols import ( ArgumentGroup, FunctionPodProtocol, @@ -328,9 +329,16 @@ def add_post_run_hook(self, hook: PostRunHook) -> None: def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: """Fire all registered hooks with payload in registration order. + When the invocation failed with a ``StoreWriteError``, a fail-loud hook that + raises does not replace it: its failure is logged and noted on the store + error, the remaining hooks are skipped as for any fail-loud failure, and the + caller re-raises the store error. + Args: payload: The post-run payload to pass to each hook. """ + error = payload.stats.error + store_error = error if isinstance(error, StoreWriteError) else None for hook in self._post_run_hooks: fn = hook.fn if isinstance(hook, HookConfig) else hook on_error = hook.on_error if isinstance(hook, HookConfig) else "raise" @@ -338,6 +346,9 @@ def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: fn(payload) except Exception as exc: if on_error == "raise": + if store_error is not None: + note_callback_failure(store_error, f"Post-run hook {fn!r}", exc) + return raise logger.warning( "Post-run hook %r raised and was suppressed: %s", @@ -592,7 +603,8 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: # A failed store write fails the run; only rows whose # function raised are skipped. exc.add_context(tag=tag.as_dict()) - obs.on_data_crash(pod_label, tag, data, exc) + with keep_store_error(exc, "Observer on_data_crash"): + obs.on_data_crash(pod_label, tag, data, exc) raise except Exception as exc: logger.debug( diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 8ed336d8b..394807b8a 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -37,6 +37,7 @@ SchemaVersionError, StoreWriteError, ) +from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.core_protocols import ( FunctionPodProtocol, DataFunctionExecutorProtocol, @@ -1306,8 +1307,10 @@ def execute( # A failed store write fails the run whatever the error policy: # error_policy only decides about rows whose function raised. exc.add_context(node_label=node_label, tag=tag.as_dict()) - ctx_obs.on_data_crash(node_label, tag, data, exc) - ctx_obs.on_node_end(node_label, node_hash) + with keep_store_error(exc, "Observer on_data_crash"): + ctx_obs.on_data_crash(node_label, tag, data, exc) + with keep_store_error(exc, "Observer on_node_end"): + ctx_obs.on_node_end(node_label, node_hash) raise except Exception as exc: logger.warning( diff --git a/src/orcapod/pipeline/async_orchestrator.py b/src/orcapod/pipeline/async_orchestrator.py index 03205c5eb..8020d3d03 100644 --- a/src/orcapod/pipeline/async_orchestrator.py +++ b/src/orcapod/pipeline/async_orchestrator.py @@ -16,6 +16,7 @@ from orcapod.channels import BroadcastChannel, Channel from orcapod.errors import StoreWriteError from orcapod.pipeline.result import OrchestratorResult +from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.node_protocols import ( is_function_node, is_operator_node, @@ -138,6 +139,8 @@ async def _run_async( effective_observer = observer if observer is not None else NoOpObserver() effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) + topo_order: list[Any] = [] + store_error: StoreWriteError | None = None try: topo_order = list(graph.topological_sort()) buf = self._buffer_size @@ -251,12 +254,71 @@ async def _run_async( return OrchestratorResult( node_outputs=collectors if materialize_results else {} ) - except* StoreWriteError as group: + except ExceptionGroup as group: # A failed store write stops the run. Raise it on its own, as the sync # orchestrator does, rather than inside the TaskGroup's exception group. - raise StoreWriteError.in_group(group)[0] + store_error = _single_store_error(group, topo_order) + if store_error is None: + raise + raise store_error finally: - effective_observer.on_run_end(run_id) + # A failing on_run_end must not replace a failed store write. + with keep_store_error(store_error, "Observer on_run_end"): + effective_observer.on_run_end(run_id) + + +def _single_store_error( + group: ExceptionGroup, topo_order: list[Any] +) -> StoreWriteError | None: + """Pick the one ``StoreWriteError`` an async run raises from its exception group. + + Concurrent node tasks can fail together, so the ``TaskGroup``'s group may hold + several store errors and other exceptions. The caller gets a single + ``StoreWriteError``, as from the sync orchestrator: the one whose node comes + first in topological order (the one the sync orchestrator would meet first), + with ties and node-less errors broken by their order in the group. Every other + exception stays attached for diagnosis: each is noted on the chosen error, + other store errors are also logged at WARNING, and the whole group is the + chosen error's ``__context__``. Its ``__cause__`` stays the backend error. + + Args: + group: The exception group the run's ``TaskGroup`` raised. + topo_order: The run's nodes in topological order. + + Returns: + The store error to raise, or ``None`` when the group holds none. + """ + store_errors = StoreWriteError.in_group(group) + if not store_errors: + return None + rank = {getattr(node, "label", None): i for i, node in enumerate(topo_order)} + chosen = min(store_errors, key=lambda err: rank.get(err.node_label, len(rank))) + for other in _leaves(group): + if other is chosen: + continue + if isinstance(other, StoreWriteError): + logger.warning( + "Another store write failed in the same run: %s", + other, + exc_info=other, + ) + chosen.add_note(f"Another store write failed in the same run: {other}") + else: + chosen.add_note( + f"Another node failed in the same run: {type(other).__name__}: {other}" + ) + return chosen + + +def _leaves(group: BaseExceptionGroup) -> list[BaseException]: + """Return every non-group exception in ``group``, nested groups flattened.""" + found: list[BaseException] = [] + for exc in group.exceptions: + if isinstance(exc, BaseExceptionGroup): + found.extend(_leaves(exc)) + else: + found.append(exc) + return found class _CollectingWriter: diff --git a/src/orcapod/pipeline/sync_orchestrator.py b/src/orcapod/pipeline/sync_orchestrator.py index fcb21a4b4..46c2ab1f3 100644 --- a/src/orcapod/pipeline/sync_orchestrator.py +++ b/src/orcapod/pipeline/sync_orchestrator.py @@ -10,8 +10,10 @@ import uuid from typing import TYPE_CHECKING, Any, Literal +from orcapod.errors import StoreWriteError from orcapod.pipeline.result import OrchestratorResult from orcapod.utils import arrow_utils +from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.node_protocols import ( is_function_node, is_operator_node, @@ -77,6 +79,7 @@ def run( effective_observer = observer if observer is not None else NoOpObserver() effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) + store_error: StoreWriteError | None = None try: topo_order = list(graph.topological_sort()) buffers: dict[Any, list[tuple[TagProtocol, DataProtocol]]] = {} @@ -131,8 +134,13 @@ def run( buffers.clear() return OrchestratorResult(node_outputs=buffers) + except StoreWriteError as exc: + store_error = exc + raise finally: - effective_observer.on_run_end(run_id) + # A failing on_run_end must not replace a failed store write. + with keep_store_error(store_error, "Observer on_run_end"): + effective_observer.on_run_end(run_id) @staticmethod def _gather_upstream( diff --git a/src/orcapod/utils/store_errors.py b/src/orcapod/utils/store_errors.py new file mode 100644 index 000000000..d7cec2df8 --- /dev/null +++ b/src/orcapod/utils/store_errors.py @@ -0,0 +1,67 @@ +"""Helpers that keep a ``StoreWriteError`` intact on its way out of a run. + +A failed store write always fails the run (ITL-878). User callbacks that run while +the error propagates (a post-run hook, an observer callback) may fail too; their +failure is logged and noted on the store error, never raised in its place. +""" + +from __future__ import annotations + +import logging +from collections.abc import Iterator +from contextlib import contextmanager + +from orcapod.errors import StoreWriteError + +logger = logging.getLogger(__name__) + + +def note_callback_failure( + store_error: StoreWriteError, during: str, failure: Exception +) -> None: + """Log a callback failure and record it on the propagating store error. + + Args: + store_error: The store error on its way out of the run. + during: What was running, for the log and the note, such as + ``"Observer on_data_crash"``. + failure: The exception the callback raised. + """ + logger.warning( + "%s raised while a store write failure was propagating, which is raised " + + "instead: %s", + during, + failure, + exc_info=failure, + ) + store_error.add_note( + f"{during} also raised while this error propagated: " + + f"{type(failure).__name__}: {failure}" + ) + + +@contextmanager +def keep_store_error( + store_error: StoreWriteError | None, during: str +) -> Iterator[None]: + """Run a callback so that it cannot replace a propagating store error. + + With ``store_error`` set, an ``Exception`` the block raises is logged and noted on + ``store_error`` (see ``note_callback_failure``) and not re-raised, so the caller's + ``raise`` propagates the store error. With ``store_error=None`` the block runs + unchanged and its exceptions propagate. + + Args: + store_error: The store error on its way out, or ``None`` when none is. + during: What the block runs, for the log and the note. + + Yields: + Nothing; the block runs inside the context. + """ + if store_error is None: + yield + return + try: + yield + except Exception as failure: + note_callback_failure(store_error, during, failure) diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md index 32f993ec2..e5dcbf9e4 100644 --- a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -187,3 +187,38 @@ Tighten Task 1 assertions to `pytest.raises(StoreWriteError)`. `StoreWriteError`. - Connector/Delta `flush` pop the pending batch before writing (coordinator is filing). - ITL-877 (SQLite logical types). + +--- + +## Follow-up: CodeRabbit round 1 (approved by the coordinator if each finding reproduces) + +**Finding 1 (comment 4534032564): a failing hook or observer callback replaces a +`StoreWriteError`.** A node's own `CachedFunctionPod` never carries hooks, so the reachable +case is a node function that runs another cached pod (`@function_pod(pod_cache_database=..., +post_run_hooks=[...])`) whose store write fails: the ERROR-status hook fires inside +`_invoke_with_hooks`'s `except`, a fail-loud hook raises, its exception replaces the store +error, and the outer node skips the row under `error_policy="continue"`. The same applies to +an observer's `on_data_crash` / `on_node_end` raised on the store-error branch. + +- Test first (public interface): standalone `pod.pod(stream).as_table()` and + `pod.pod.async_execute(...)` with a raising hook raise `StoreWriteError`; a pipeline whose + node function runs such a pod raises `StoreWriteError` on both orchestrators under + `continue`; a pipeline observer whose `on_data_crash` raises does not hide it. +- Fix: `_fire_post_run_hooks(payload, store_error=...)`: when a store error is propagating, + a raising hook is logged at WARNING and recorded with `add_note` on the store error, which + then propagates. Observer callbacks on the store-error branches go through one helper that + does the same. Hook failures without a store error are unchanged. + +**Finding 2 (comment 4534032576): mixed exception groups from the async orchestrator.** +`except* StoreWriteError` re-raises the unmatched rest as an `ExceptionGroup` together with +the store error, and drops further store errors silently. + +- Test first: two sibling nodes failing concurrently (store write + another exception; two + store writes) → the async caller gets one plain `StoreWriteError`. +- Fix: catch `ExceptionGroup`; if it holds any `StoreWriteError`, raise the one whose node + comes first in topological order (ties: group order), the order the sync orchestrator meets + them; log every other `StoreWriteError` at WARNING; add a note per other exception; the + whole group stays reachable as `__context__` (`__cause__` stays the backend error). A + group with no store error propagates unchanged. + +No hash, schema or other behaviour change. diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index 7820eec20..901a52585 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -9,6 +9,7 @@ from __future__ import annotations +import asyncio import pathlib from dataclasses import dataclass, field from typing import Any @@ -16,6 +17,7 @@ import pytest from orcapod import PipelineJob, function_pod +from orcapod.channels import Channel from orcapod.core.operators import SelectDataColumns from orcapod.errors import StoreWriteError from orcapod.databases import ( @@ -24,6 +26,7 @@ SQLiteConnector, ) from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.pipeline.observer import NoOpObserver from orcapod.sources import DictSource from orcapod.types import CacheMode @@ -236,12 +239,18 @@ def test_pipeline_table_failure_raises(self, orchestrator_cls): assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) - # Every result computed before the failure was stored, so a healthy re-run - # reuses it: across both runs each row is computed exactly once. + # The failing row's result was stored before its pipeline record failed, so a + # healthy re-run reuses it instead of computing it again. (On the async + # orchestrator a row still computing when the run stopped was never stored + # and is computed again, so the total call count is not fixed.) + failed_tag = exc_info.value.tag + assert failed_tag is not None + failed_x = next(row["x"] for row in ROWS if row["k"] == failed_tag["k"]) + calls.clear() rerun = build_job(inner, ROWS) rerun.run(orchestrator=orchestrator_cls()) assert rerun.nodes["dbl"].as_table().num_rows == 3 - assert sorted(calls) == [0, 1, 2] + assert failed_x not in calls def test_failure_on_new_row_keeps_previously_stored_rows(self, orchestrator_cls): inner = InMemoryArrowDatabase() @@ -325,3 +334,206 @@ def test_logical_type_output_on_sqlite_raises(self, orchestrator_cls, tmp_path): assert exc_info.value.node_label == "dbl" assert "node 'dbl'" in str(exc_info.value) assert "does not preserve Arrow metadata" in str(exc_info.value) + + +# --------------------------------------------------------------------------- +# A failing hook or observer callback never replaces a store error +# --------------------------------------------------------------------------- + + +def failing_hook(payload: Any) -> None: + raise ValueError("post-run hook failed") + + +def make_hooked_triple(store: Any): + """A cached pod with a fail-loud post-run hook, writing its results to ``store``.""" + + @function_pod( + output_keys="tripled", pod_cache_database=store, post_run_hooks=[failing_hook] + ) + def triple(x: int) -> int: + return x * 3 + + return triple + + +def make_outer(inner: Any): + """A function pod whose function runs ``inner`` through its stream.""" + + @function_pod(output_keys="double") + def via_inner(x: int) -> int: + stream = DictSource(data=[{"i": "i", "x": x}], tag_columns=["i"]) + return inner.pod(stream).as_table()["tripled"][0].as_py() - x + + return via_inner + + +def assert_hook_failure_noted(err: BaseException) -> None: + assert isinstance(err, StoreWriteError), f"{type(err).__name__}: {err}" + assert isinstance(err.__cause__, OSError) + notes = getattr(err, "__notes__", []) + assert any("post-run hook failed" in note for note in notes), notes + + +class CrashingObserver(NoOpObserver): + """An observer whose crash and end callbacks raise once a row has crashed.""" + + def __init__(self) -> None: + self.crashed = False + + def on_data_crash(self, node_label, tag, data, error) -> None: + self.crashed = True + raise RuntimeError("observer failed in on_data_crash") + + def on_node_end(self, node_label, node_hash) -> None: + if self.crashed: + raise RuntimeError("observer failed in on_node_end") + + def on_run_end(self, run_id) -> None: + if self.crashed: + raise RuntimeError("observer failed in on_run_end") + + +class TestCallbackFailureKeepsStoreError: + def test_failing_hook_keeps_store_error_standalone(self): + faults = Faults(write_marker="") + triple = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) + stream = DictSource(data=ROWS, tag_columns=["k"]) + + with pytest.raises(StoreWriteError) as exc_info: + triple.pod(stream).as_table() + + assert_hook_failure_noted(exc_info.value) + + def test_failing_hook_keeps_store_error_standalone_async(self): + faults = Faults(write_marker="") + triple = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) + stream = DictSource(data=ROWS, tag_columns=["k"]) + input_ch: Channel[Any] = Channel(buffer_size=16) + output_ch: Channel[Any] = Channel(buffer_size=16) + + async def feed() -> None: + for tag, data in stream.iter_data(): + await input_ch.writer.send((tag, data)) + await input_ch.writer.close() + + async def run() -> None: + await asyncio.gather( + feed(), triple.pod.async_execute([input_ch.reader], output_ch.writer) + ) + + with pytest.raises(ExceptionGroup) as exc_info: + asyncio.run(run()) + + leaves = exc_info.value.exceptions + assert all(isinstance(leaf, StoreWriteError) for leaf in leaves), leaves + for leaf in leaves: + assert_hook_failure_noted(leaf) + + @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) + def test_failing_hook_in_nested_pod_fails_the_run( + self, orchestrator_cls, error_policy + ): + """A node function running a cached pod whose write fails cannot skip the row.""" + faults = Faults(write_marker="") + inner = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) + job = build_job(InMemoryArrowDatabase(), ROWS, pod=make_outer(inner)) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(error_policy=error_policy)) + + assert_hook_failure_noted(exc_info.value) + + def test_failing_observer_keeps_store_error(self, orchestrator_cls): + faults = Faults(write_marker=RESULT_TABLE) + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(), observer=CrashingObserver()) + + err = exc_info.value + assert_store_error(err, faults, node_label="dbl", names_row=True) + notes = getattr(err, "__notes__", []) + assert any("observer failed in on_data_crash" in note for note in notes), notes + assert any("observer failed in on_run_end" in note for note in notes), notes + + +# --------------------------------------------------------------------------- +# Concurrent failures in the async orchestrator +# --------------------------------------------------------------------------- + +# ``async def`` functions run on the event loop rather than in worker threads, so +# the order in which concurrent node tasks fail is the same on every run. + + +@function_pod(output_keys="double") +async def double_async(x: int) -> int: + return x * 2 + + +@function_pod(output_keys="tripled") +async def triple_async(x: int) -> int: + return x * 3 + + +@function_pod(output_keys="tripled") +async def fail_async(x: int) -> int: + raise RuntimeError("function failed") + + +class FailingCrashObserver(NoOpObserver): + """An observer whose ``on_data_crash`` raises for one node, failing that node.""" + + def on_data_crash(self, node_label, tag, data, error) -> None: + if node_label == "other": + raise KeyError("observer failed for node 'other'") + + +def build_siblings(store: Any, other_pod: Any): + """Two function nodes reading the same source: ``dbl`` and ``other``.""" + job = PipelineJob(name=JOB_NAME, store=store) + with job: + source = DictSource(data=ROWS, tag_columns=["k"], source_id="src") + double_async.pod(source, label="dbl") + other_pod.pod(source, label="other") + return job + + +class TestConcurrentFailures: + def test_store_error_and_other_failure_raise_one_store_error(self): + faults = Faults(write_marker=RESULT_TABLE) + job = build_siblings(FaultyStore(InMemoryArrowDatabase(), faults), fail_async) + + with pytest.raises(StoreWriteError) as exc_info: + job.run( + orchestrator=AsyncPipelineOrchestrator(), + observer=FailingCrashObserver(), + ) + + err = exc_info.value + assert_store_error(err, faults, node_label="dbl", names_row=True) + notes = getattr(err, "__notes__", []) + assert any("observer failed for node 'other'" in note for note in notes), notes + # The whole group stays reachable for diagnosis. + assert isinstance(err.__context__, ExceptionGroup) + + def test_several_store_errors_raise_the_first_in_node_order( + self, orchestrator_cls, caplog + ): + faults = Faults(write_marker=RESULT_TABLE) + job = build_siblings(FaultyStore(InMemoryArrowDatabase(), faults), triple_async) + + with caplog.at_level("WARNING"): + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + # The sync orchestrator meets ``dbl`` first; the async one raises the same. + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + if orchestrator_cls is AsyncPipelineOrchestrator: + others = [ + record + for record in caplog.records + if record.levelname == "WARNING" + and "Another store write failed" in record.getMessage() + ] + assert any("node 'other'" in r.getMessage() for r in others), others From 2770848ee39cd796b80fcfa3254b595d70506aa9 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:17:29 -0800 Subject: [PATCH 05/15] docs(pipeline): docstrings for the store-write-failure tests and StoreWriteError (ITL-878) Addresses CodeRabbit's docstring-coverage pre-merge check on #287. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/errors.py | 8 +++ .../test_pipeline/test_store_write_failure.py | 50 +++++++++++++++++++ 2 files changed, 58 insertions(+) diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index e50d2582a..8e8f7b00a 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -227,6 +227,13 @@ def __init__( node_label: str | None = None, tag: Mapping[str, Any] | None = None, ) -> None: + """Create the error for one failed write. + + Args: + record_path: Path of the table the write was for. + node_label: Label of the node whose write failed, if the write site knows it. + tag: Tag values of the row being stored, if the write covered one row. + """ self.record_path = tuple(record_path) self.node_label = node_label self.tag = dict(tag) if tag is not None else None @@ -266,6 +273,7 @@ def add_context( self.tag = dict(tag) def __str__(self) -> str: + """Describe the failed write: node, row, table and the backend's reason.""" where = f"node {self.node_label!r}" if self.node_label else "the pipeline" row = f" for row {self.tag!r}" if self.tag else "" cause = self.__cause__ diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index 901a52585..499f159d6 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -58,13 +58,16 @@ class FaultyStore: """An ``InMemoryArrowDatabase`` that raises on chosen writes or on flush.""" def __init__(self, inner: Any, faults: Faults) -> None: + """Wrap ``inner``, failing the writes ``faults`` selects.""" self._inner = inner self._faults = faults def at(self, *path_components: str) -> FaultyStore: + """Return a view at ``path_components`` that shares this store's faults.""" return FaultyStore(self._inner.at(*path_components), self._faults) def _check_write(self, record_path: tuple[str, ...]) -> None: + """Raise ``OSError`` if the write to ``record_path`` is one to fail.""" faults = self._faults full_path = "/".join(self._inner.base_path + tuple(record_path)) if faults.write_marker is None or faults.write_marker not in full_path: @@ -77,6 +80,7 @@ def _check_write(self, record_path: tuple[str, ...]) -> None: def add_record( self, record_path, record_id, record, skip_duplicates=False, flush=False ): + """Write one record unless the faults select this table.""" self._check_write(record_path) self._inner.add_record( record_path, record_id, record, skip_duplicates=skip_duplicates, flush=flush @@ -90,6 +94,7 @@ def add_records( skip_duplicates=False, flush=False, ): + """Write a batch of records unless the faults select this table.""" self._check_write(record_path) self._inner.add_records( record_path, @@ -100,11 +105,13 @@ def add_records( ) def flush(self) -> None: + """Flush the wrapped store, or raise ``OSError`` if flushes fail.""" if self._faults.on_flush: raise OSError("disk unavailable while flushing") self._inner.flush() def __getattr__(self, name: str) -> Any: + """Delegate everything else to the wrapped store.""" return getattr(self._inner, name) @@ -119,12 +126,14 @@ def __getattr__(self, name: str) -> Any: @function_pod(output_keys="double") def double(x: int) -> int: + """Double ``x``, recording the call.""" calls.append(x) return x * 2 @function_pod(output_keys="double") def double_or_fail(x: int) -> int: + """Double ``x``, raising for ``x == 1``.""" if x == 1: raise RuntimeError("function failed for x == 1") return x * 2 @@ -132,12 +141,14 @@ def double_or_fail(x: int) -> int: @function_pod(output_keys="where") def to_path(x: int) -> pathlib.Path: + """Return a ``pathlib.Path`` output, a logical type SQLite cannot store.""" return pathlib.Path(f"/data/{x}.txt") def build_job( store: Any, rows: list[dict[str, Any]], *, pod=double, with_operator: bool = False ): + """Build a job with one function node ``dbl`` and optionally an operator ``sel``.""" job = PipelineJob(name=JOB_NAME, store=store) with job: source = DictSource(data=rows, tag_columns=["k"], source_id="src") @@ -154,11 +165,13 @@ def build_job( @pytest.fixture(autouse=True) def _reset_calls(): + """Clear the call log before each test.""" calls.clear() @pytest.fixture(params=["sync", "async"]) def orchestrator_cls(request): + """Run each test on the sync and the async orchestrator.""" return {"sync": SyncPipelineOrchestrator, "async": AsyncPipelineOrchestrator}[ request.param ] @@ -199,7 +212,10 @@ def stored_rows(store: Any, label: str = "dbl") -> int: class TestFunctionNodeWriteFailure: + """A function node's failed result or pipeline write fails the run.""" + def test_result_table_failure_on_first_row_raises(self, orchestrator_cls): + """A result write failing on the first row raises, naming node, row and table.""" inner = InMemoryArrowDatabase() faults = Faults(write_marker=RESULT_TABLE) job = build_job(FaultyStore(inner, faults), ROWS) @@ -212,6 +228,7 @@ def test_result_table_failure_on_first_row_raises(self, orchestrator_cls): def test_result_table_failure_on_later_row_keeps_earlier_rows( self, orchestrator_cls ): + """Rows stored before the failure are kept and are not recomputed on a re-run.""" inner = InMemoryArrowDatabase() faults = Faults(write_marker=RESULT_TABLE, succeed_first=1) job = build_job(FaultyStore(inner, faults), ROWS) @@ -230,6 +247,7 @@ def test_result_table_failure_on_later_row_keeps_earlier_rows( assert len(calls) == 2 def test_pipeline_table_failure_raises(self, orchestrator_cls): + """A pipeline-record write failure raises; the failing row's stored result is reused.""" inner = InMemoryArrowDatabase() faults = Faults(write_marker=PIPELINE_TABLE) job = build_job(FaultyStore(inner, faults), ROWS) @@ -253,6 +271,7 @@ def test_pipeline_table_failure_raises(self, orchestrator_cls): assert failed_x not in calls def test_failure_on_new_row_keeps_previously_stored_rows(self, orchestrator_cls): + """A failure on a new row keeps the rows a previous run stored.""" inner = InMemoryArrowDatabase() build_job(inner, ROWS[:2]).run(orchestrator=orchestrator_cls()) assert stored_rows(inner) == 2 @@ -269,6 +288,7 @@ def test_failure_on_new_row_keeps_previously_stored_rows(self, orchestrator_cls) def test_store_failure_raises_under_any_error_policy( self, orchestrator_cls, error_policy ): + """A store failure raises under both error policies.""" faults = Faults(write_marker=RESULT_TABLE) job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) @@ -278,6 +298,7 @@ def test_store_failure_raises_under_any_error_policy( assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) def test_flush_failure_raises(self, orchestrator_cls): + """A failure of the job's final flush raises ``StoreWriteError``.""" faults = Faults(on_flush=True) job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) @@ -302,9 +323,12 @@ def test_function_failure_is_still_skipped_under_continue(self, orchestrator_cls class TestOperatorNodeWriteFailure: + """An operator node's failed LOG write fails the run.""" + def test_operator_log_failure_raises_and_commits_upstream_rows( self, orchestrator_cls ): + """The operator's failure raises and the upstream node's rows stay committed.""" inner = InMemoryArrowDatabase() faults = Faults(write_marker=OPERATOR_TABLE) job = build_job(FaultyStore(inner, faults), ROWS, with_operator=True) @@ -324,7 +348,10 @@ def test_operator_log_failure_raises_and_commits_upstream_rows( class TestSQLiteTrigger: + """The reported trigger: a logical-type output on SQLite.""" + def test_logical_type_output_on_sqlite_raises(self, orchestrator_cls, tmp_path): + """A ``pathlib.Path`` output on SQLite raises, naming the node.""" store = ConnectorArrowDatabase(SQLiteConnector(str(tmp_path / "store.db"))) job = build_job(store, ROWS, pod=to_path) @@ -342,6 +369,7 @@ def test_logical_type_output_on_sqlite_raises(self, orchestrator_cls, tmp_path): def failing_hook(payload: Any) -> None: + """A fail-loud post-run hook that always raises.""" raise ValueError("post-run hook failed") @@ -352,6 +380,7 @@ def make_hooked_triple(store: Any): output_keys="tripled", pod_cache_database=store, post_run_hooks=[failing_hook] ) def triple(x: int) -> int: + """Triple ``x``.""" return x * 3 return triple @@ -362,6 +391,7 @@ def make_outer(inner: Any): @function_pod(output_keys="double") def via_inner(x: int) -> int: + """Run ``inner`` on ``x`` through a one-row stream and return ``3x - x``.""" stream = DictSource(data=[{"i": "i", "x": x}], tag_columns=["i"]) return inner.pod(stream).as_table()["tripled"][0].as_py() - x @@ -369,6 +399,7 @@ def via_inner(x: int) -> int: def assert_hook_failure_noted(err: BaseException) -> None: + """The error is the store error, with the hook failure recorded as a note.""" assert isinstance(err, StoreWriteError), f"{type(err).__name__}: {err}" assert isinstance(err.__cause__, OSError) notes = getattr(err, "__notes__", []) @@ -379,23 +410,30 @@ class CrashingObserver(NoOpObserver): """An observer whose crash and end callbacks raise once a row has crashed.""" def __init__(self) -> None: + """Start with no crashed row.""" self.crashed = False def on_data_crash(self, node_label, tag, data, error) -> None: + """Record the crash, then fail.""" self.crashed = True raise RuntimeError("observer failed in on_data_crash") def on_node_end(self, node_label, node_hash) -> None: + """Fail if a row has crashed.""" if self.crashed: raise RuntimeError("observer failed in on_node_end") def on_run_end(self, run_id) -> None: + """Fail if a row has crashed.""" if self.crashed: raise RuntimeError("observer failed in on_run_end") class TestCallbackFailureKeepsStoreError: + """A hook or observer that raises never replaces a store error.""" + def test_failing_hook_keeps_store_error_standalone(self): + """A standalone cached pod raises its store error, not the hook's.""" faults = Faults(write_marker="") triple = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) stream = DictSource(data=ROWS, tag_columns=["k"]) @@ -406,6 +444,7 @@ def test_failing_hook_keeps_store_error_standalone(self): assert_hook_failure_noted(exc_info.value) def test_failing_hook_keeps_store_error_standalone_async(self): + """The async path raises the store error too, rather than skipping the row.""" faults = Faults(write_marker="") triple = make_hooked_triple(FaultyStore(InMemoryArrowDatabase(), faults)) stream = DictSource(data=ROWS, tag_columns=["k"]) @@ -413,11 +452,13 @@ def test_failing_hook_keeps_store_error_standalone_async(self): output_ch: Channel[Any] = Channel(buffer_size=16) async def feed() -> None: + """Send the stream's rows into the input channel, then close it.""" for tag, data in stream.iter_data(): await input_ch.writer.send((tag, data)) await input_ch.writer.close() async def run() -> None: + """Feed the pod and run it concurrently.""" await asyncio.gather( feed(), triple.pod.async_execute([input_ch.reader], output_ch.writer) ) @@ -445,6 +486,7 @@ def test_failing_hook_in_nested_pod_fails_the_run( assert_hook_failure_noted(exc_info.value) def test_failing_observer_keeps_store_error(self, orchestrator_cls): + """Observer failures on the way out are noted on the store error.""" faults = Faults(write_marker=RESULT_TABLE) job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) @@ -468,16 +510,19 @@ def test_failing_observer_keeps_store_error(self, orchestrator_cls): @function_pod(output_keys="double") async def double_async(x: int) -> int: + """Double ``x`` on the event loop.""" return x * 2 @function_pod(output_keys="tripled") async def triple_async(x: int) -> int: + """Triple ``x`` on the event loop.""" return x * 3 @function_pod(output_keys="tripled") async def fail_async(x: int) -> int: + """Always raise, on the event loop.""" raise RuntimeError("function failed") @@ -485,6 +530,7 @@ class FailingCrashObserver(NoOpObserver): """An observer whose ``on_data_crash`` raises for one node, failing that node.""" def on_data_crash(self, node_label, tag, data, error) -> None: + """Fail for crashes of node ``other`` only.""" if node_label == "other": raise KeyError("observer failed for node 'other'") @@ -500,7 +546,10 @@ def build_siblings(store: Any, other_pod: Any): class TestConcurrentFailures: + """Concurrent node failures still give one plain ``StoreWriteError``.""" + def test_store_error_and_other_failure_raise_one_store_error(self): + """A store error mixed with another failure raises the store error alone.""" faults = Faults(write_marker=RESULT_TABLE) job = build_siblings(FaultyStore(InMemoryArrowDatabase(), faults), fail_async) @@ -520,6 +569,7 @@ def test_store_error_and_other_failure_raise_one_store_error(self): def test_several_store_errors_raise_the_first_in_node_order( self, orchestrator_cls, caplog ): + """Several store errors raise the first node's, the same on both orchestrators.""" faults = Faults(write_marker=RESULT_TABLE) job = build_siblings(FaultyStore(InMemoryArrowDatabase(), faults), triple_async) From 0940f9957ad7a8c72d1279bca8cbf9c168e51abf Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:24:06 -0800 Subject: [PATCH 06/15] style(pipeline): unquote two annotations ruff UP037 flags (ITL-878) Addresses CodeRabbit's review of #287. The return annotation of StoreWriteError.in_group stays quoted: errors.py has no postponed annotations, so the class name is not defined when the method is. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/errors.py | 2 +- src/orcapod/pipeline/job.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 8e8f7b00a..3c2acdbee 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -250,7 +250,7 @@ def in_group(cls, group: BaseExceptionGroup) -> "list[StoreWriteError]": Returns: The ``StoreWriteError`` leaves in the order the group holds them. """ - found: "list[StoreWriteError]" = [] + found: list[StoreWriteError] = [] for exc in group.exceptions: if isinstance(exc, BaseExceptionGroup): found.extend(cls.in_group(exc)) diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index dd884dfe6..4b24d029f 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -31,7 +31,7 @@ logger = logging.getLogger(__name__) -def _flush_best_effort(*databases: "ArrowDatabaseProtocol") -> None: +def _flush_best_effort(*databases: ArrowDatabaseProtocol) -> None: """Flush each database, logging instead of raising if a flush fails. Used when a run has already failed: the run's own error is the one to report, From 66686a97faab09fff5645f9cc24d4dd4b932b5bb Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 23:42:35 -0800 Subject: [PATCH 07/15] docs(plan): CodeRabbit round 2 out-of-diff findings (ITL-878) Co-Authored-By: Claude Opus 5.5 --- .../2026-11-10-itl-878-store-write-failure.md | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md index e5dcbf9e4..51a9a8567 100644 --- a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -222,3 +222,39 @@ the store error, and drops further store errors silently. group with no store error propagates unchanged. No hash, schema or other behaviour change. + +--- + +## Follow-up: CodeRabbit round 2, out-of-diff findings (review of 0940f995) + +Three findings sat in the review body ("Outside diff range comments"), not in threads. + +**Finding A (job.py final flush, approved if it reproduces): the final-flush `StoreWriteError` +does not warn that pending output may not be stored.** All views of one store share the +pending batches, so `pipeline_db.flush()` commits every table; a batch that fails stops the +flush and the batches after it (a `cache_mode=LOG` operator's output, other nodes' rows) +stay pending, and `result_db.flush()` never runs. + +- Test first: `Faults(on_flush=True)` with the LOG operator; the raised `StoreWriteError` + carries a note saying writes still pending may not have been stored. +- Fix: build the error, `add_note(...)`, `raise err from exc`. Same error class, same + `record_path`, same `__cause__`. + +**Finding B (job.py `_flush_best_effort`, approved if it reproduces): a failed best-effort +flush shows only in the log.** + +- Test first: an operator LOG write fails and the best-effort flush fails too; the raised + `StoreWriteError` is still the operator's, and carries a note that the flush failed and + pending results may not have been stored (upstream rows read back: 0). +- Fix: `_flush_best_effort(run_error, *databases)` keeps logging and adds a note naming the + flush failure to the propagating run error. The raised error is unchanged. + +**Finding C (`ResultCache.store` record preparation): REPRODUCE AND REPORT ONLY.** Establish +on both orchestrators and both error policies which function outputs fail in record +preparation (naive datetime, value not matching the declared type, unsupported type, a +logical type whose `python_to_storage` raises), what the caller sees (row skipped? WARNING +log? observer `on_data_crash`? run returns? anything stored?), and what CodeRabbit's +proposal would change. Ask the coordinator to choose (a) row failure under `error_policy` +reported exactly like a function failure, or (b) `StoreWriteError`. No fix before the reply. + +No hash, schema or stored-data change. From 9200727ae74949b8d6e65351b8d109a146294950 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 23:43:51 -0800 Subject: [PATCH 08/15] fix(pipeline): note unstored pending writes on a failed run's error (ITL-878) A failed final flush now warns that writes still pending, such as a cache_mode=LOG operator's output, may not have been stored. A best-effort flush that fails after a run failure is noted on the run's error instead of only being logged. The raised error is unchanged in both cases. The flush-failure test now arms the fault at run end so it reaches the job's final flush; with the fault armed from the start the result cache's per-row flush failed first. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/pipeline/job.py | 31 ++++++-- .../test_pipeline/test_store_write_failure.py | 71 ++++++++++++++++++- 2 files changed, 93 insertions(+), 9 deletions(-) diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 4b24d029f..8309d4eca 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -31,24 +31,33 @@ logger = logging.getLogger(__name__) -def _flush_best_effort(*databases: ArrowDatabaseProtocol) -> None: - """Flush each database, logging instead of raising if a flush fails. +def _flush_best_effort( + run_error: BaseException, *databases: ArrowDatabaseProtocol +) -> None: + """Flush each database, noting instead of raising if a flush fails. Used when a run has already failed: the run's own error is the one to report, and a second failure while committing the writes that did succeed must not - replace it. + replace it. A flush failure is logged and added as a note to ``run_error``, + so the caller learns that pending results may not have been stored. Args: + run_error: The error the run failed with, which the caller re-raises. *databases: The databases to flush. """ for database in databases: try: database.flush() - except Exception: + except Exception as flush_error: logger.exception( "Could not flush pending writes after the run failed; " "the rows they hold were not stored" ) + run_error.add_note( + "Flushing pending writes after this failure also failed " + + f"({type(flush_error).__name__}: {flush_error}); results that were " + + "still pending may not have been stored." + ) class PipelineJob(AbstractPipelineBase[JobNode]): @@ -865,10 +874,10 @@ def run( pipeline_uri=pipeline_uri, materialize_results=materialize_results, ) - except Exception: + except Exception as run_error: # Commit the writes that did succeed before the failure, so their rows # do not look dropped, then report the failure itself. - _flush_best_effort(pipeline_db, result_db) + _flush_best_effort(run_error, pipeline_db, result_db) raise # Flush databases. @@ -876,7 +885,15 @@ def run( pipeline_db.flush() result_db.flush() except Exception as exc: - raise StoreWriteError(pipeline_db.base_path) from exc + flush_error = StoreWriteError(pipeline_db.base_path) + # Every view of the store shares one set of pending batches, so a failed + # batch can leave the ones after it unwritten. + flush_error.add_note( + "Writes that were still pending when the flush failed, such as a " + + "cache_mode=LOG operator's output or a node's last records, may not " + + "have been stored." + ) + raise flush_error from exc # Mutate self in place. self._has_run = True diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index 499f159d6..63abe61c2 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -50,6 +50,7 @@ class Faults: write_marker: str | None = None succeed_first: int = 0 on_flush: bool = False + fail_flush_after_write_failure: bool = False matching_writes: int = 0 failed_paths: list[str] = field(default_factory=list) @@ -75,6 +76,8 @@ def _check_write(self, record_path: tuple[str, ...]) -> None: faults.matching_writes += 1 if faults.matching_writes > faults.succeed_first: faults.failed_paths.append("/".join(record_path)) + if faults.fail_flush_after_write_failure: + faults.on_flush = True raise OSError(f"disk unavailable while writing {full_path}") def add_record( @@ -200,6 +203,19 @@ def assert_store_error( assert "rows that are missing" in message +class FailFlushAtRunEnd(NoOpObserver): + """Make every flush fail once the orchestrator has finished, so the job's own + final flush is the one that fails (the result cache flushes after every row).""" + + def __init__(self, faults: Faults) -> None: + """Arm ``faults`` at the end of the run.""" + self._faults = faults + + def on_run_end(self, run_id) -> None: + """Make the following flushes fail.""" + self._faults.on_flush = True + + def stored_rows(store: Any, label: str = "dbl") -> int: """Rows a fresh job over ``store`` reads back for ``label`` without running.""" records = build_job(store, ROWS).nodes[label].get_all_records() @@ -297,15 +313,26 @@ def test_store_failure_raises_under_any_error_policy( assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) - def test_flush_failure_raises(self, orchestrator_cls): - """A failure of the job's final flush raises ``StoreWriteError``.""" + def test_auto_flush_failure_raises_naming_the_node(self, orchestrator_cls): + """A failure of the result cache's flush after a row raises, naming the node.""" faults = Faults(on_flush=True) job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) + assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + + def test_final_flush_failure_raises(self, orchestrator_cls): + """A failure of the job's final flush raises ``StoreWriteError``.""" + faults = Faults() + job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(), observer=FailFlushAtRunEnd(faults)) + assert_store_error(exc_info.value, faults, node_label=None, names_row=False) + assert exc_info.value.record_path == (JOB_NAME,) def test_function_failure_is_still_skipped_under_continue(self, orchestrator_cls): """``error_policy="continue"`` keeps skipping rows whose function raised.""" @@ -341,6 +368,46 @@ def test_operator_log_failure_raises_and_commits_upstream_rows( # none of them may look dropped. assert stored_rows(inner) == 3 + def test_failed_flush_after_write_failure_is_noted(self, orchestrator_cls): + """A failed best-effort flush is noted on the write error, which stays primary.""" + inner = InMemoryArrowDatabase() + faults = Faults( + write_marker=OPERATOR_TABLE, fail_flush_after_write_failure=True + ) + job = build_job(FaultyStore(inner, faults), ROWS, with_operator=True) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + err = exc_info.value + assert_store_error(err, faults, node_label="sel", names_row=False) + notes = getattr(err, "__notes__", []) + assert any( + "disk unavailable while flushing" in note + and "may not have been stored" in note + for note in notes + ), notes + + def test_final_flush_failure_warns_pending_output(self, orchestrator_cls): + """A failed final flush warns that output still pending may not be stored.""" + faults = Faults() + job = build_job( + FaultyStore(InMemoryArrowDatabase(), faults), ROWS, with_operator=True + ) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls(), observer=FailFlushAtRunEnd(faults)) + + err = exc_info.value + assert_store_error(err, faults, node_label=None, names_row=False) + notes = getattr(err, "__notes__", []) + assert any( + "pending" in note + and "operator" in note + and "may not have been stored" in note + for note in notes + ), notes + # --------------------------------------------------------------------------- # The reported trigger: a logical-type output on SQLite (ITL-877 is the limit itself) From ff4c25cbaf1449184fec21ed0d410c6ce08e97d0 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 23:49:20 -0800 Subject: [PATCH 09/15] test(pipeline): an unstorable output is reported like a function failure (ITL-878) Pins option (a) for the record-preparation finding. Local only, pending the developer's decision. Co-Authored-By: Claude Opus 5.5 --- .../test_pipeline/test_store_write_failure.py | 97 +++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index 63abe61c2..744fbad93 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -12,6 +12,7 @@ import asyncio import pathlib from dataclasses import dataclass, field +from datetime import datetime, timezone from typing import Any import pytest @@ -654,3 +655,99 @@ def test_several_store_errors_raise_the_first_in_node_order( and "Another store write failed" in record.getMessage() ] assert any("node 'other'" in r.getMessage() for r in others), others + + +# --------------------------------------------------------------------------- +# An output that cannot be turned into a record is a row failure, not a store error +# --------------------------------------------------------------------------- + + +@dataclass +class Pair: + """A dataclass output type.""" + + a: int + b: int + + +@function_pod(output_keys="y") +def raises_for_one(x: int) -> int: + """Raise for ``x == 1``: the reference function failure.""" + if x == 1: + raise RuntimeError("function failed for x == 1") + return x + + +@function_pod(output_keys="y") +def naive_datetime_for_one(x: int) -> datetime: + """Return a naive ``datetime`` for ``x == 1``, rejected by the strict tz policy.""" + return datetime(2020, 1, 1 + x, tzinfo=None if x == 1 else timezone.utc) + + +@function_pod(output_keys="y") +def str_for_int_for_one(x: int) -> int: + """Return a ``str`` for ``x == 1`` although ``int`` is declared.""" + return "abc" if x == 1 else x # pyright: ignore[reportReturnType] # the bad output under test + + +@function_pod(output_keys="y") +def int_for_dataclass_for_one(x: int) -> Pair: + """Return an ``int`` for ``x == 1`` although a dataclass is declared.""" + return 7 if x == 1 else Pair(x, x) # pyright: ignore[reportReturnType] # the bad output under test + + +class CrashRecorder(NoOpObserver): + """Record the node label and row key of every ``on_data_crash``.""" + + def __init__(self) -> None: + """Start with no crashes.""" + self.crashes: list[tuple[str, str]] = [] + + def on_data_crash(self, node_label, tag, data, error) -> None: + """Record the crash.""" + self.crashes.append((node_label, str(tag["k"]))) + + +def row_failure_outcome(pod: Any, orchestrator: Any) -> dict[str, Any]: + """Run ``pod`` as node ``dbl`` and describe what the caller sees.""" + store = InMemoryArrowDatabase() + observer = CrashRecorder() + job = build_job(store, ROWS, pod=pod) + try: + job.run(orchestrator=orchestrator, observer=observer) + raised = None + rows = sorted(job.nodes["dbl"].as_table()["k"].to_pylist()) + except Exception as exc: + raised = exc + rows = None + return { + "raised": raised, + "rows": rows, + "crashes": observer.crashes, + "stored": stored_rows(store), + } + + +class TestUnstorableOutputIsARowFailure: + """An output that cannot become a record is reported like a function failure.""" + + @pytest.mark.parametrize( + "pod", + [naive_datetime_for_one, str_for_int_for_one, int_for_dataclass_for_one], + ids=["naive_datetime", "str_for_int", "int_for_dataclass"], + ) + @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) + def test_reported_like_a_function_failure( + self, orchestrator_cls, error_policy, pod + ): + """Same raise-or-return, rows, observer report and stored rows as a raise.""" + expected = row_failure_outcome( + raises_for_one, orchestrator_cls(error_policy=error_policy) + ) + actual = row_failure_outcome(pod, orchestrator_cls(error_policy=error_policy)) + + assert not isinstance(actual["raised"], StoreWriteError), actual["raised"] + assert (actual["raised"] is None) == (expected["raised"] is None), actual + assert actual["rows"] == expected["rows"] + assert actual["crashes"] == expected["crashes"] == [("dbl", "r1")] + assert actual["stored"] == expected["stored"] From fb15f9637e6227a379204e79ca745f32d21fc666 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 01:15:54 -0800 Subject: [PATCH 10/15] test(pipeline): an unstorable output is logged like a function failure (ITL-878) Co-Authored-By: Claude Opus 5.5 --- .../2026-11-10-itl-878-store-write-failure.md | 25 +++++++++++ .../test_pipeline/test_store_write_failure.py | 43 +++++++++++++------ 2 files changed, 56 insertions(+), 12 deletions(-) diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md index 51a9a8567..246e33bb2 100644 --- a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -258,3 +258,28 @@ proposal would change. Ask the coordinator to choose (a) row failure under `erro reported exactly like a function failure, or (b) `StoreWriteError`. No fix before the reply. No hash, schema or stored-data change. + +--- + +## Follow-up: Finding C decided — an unstorable output is a row failure (2026-11-11) + +**Decision (developer, November 11, 2026):** a function output that cannot be turned into a +storable record (naive datetime, `str` for a declared `int`, `int` for a declared dataclass) +is a ROW FAILURE governed by `error_policy`, exactly like a function that raised. It is not a +`StoreWriteError`. No behaviour change. + +Resumed after a machine restart; the previous worker's local commit ff4c25cb (12 pinning +tests) is kept. + +- [x] Pinning tests (`TestUnstorableOutputIsARowFailure`, sync/async x continue/fail_fast x + 3 outputs, each compared with a function raising on the same row) also assert the + per-row log record (level + message) matches the function-failure one, besides the + observer `on_data_crash`, raise-or-return, rows and stored rows. +- [ ] Docs (`docs/concepts/function-pods.md`, "When the store cannot be written"): store + failures fail the run; a function output that cannot be stored is a row failure; + async logs row failures at DEBUG and ignores `fail_fast` (ITL-879), unchanged here. +- [ ] `DESIGN_ISSUES.md` SW1: the same rule. +- [ ] Decline finding (1) with a signed top-level PR comment. +- [ ] Merge origin/main (b197544e; CHANGELOG entries under `[0.3.0] — unreleased`), full + suite on 3.11/3.12/3.13, golden unchanged, ruff, format, basedpyright, push, CI, + CodeRabbit round (bodies too). Reconcile with #290 if it merged first. diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index 744fbad93..ef187ce11 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -708,23 +708,38 @@ def on_data_crash(self, node_label, tag, data, error) -> None: self.crashes.append((node_label, str(tag["k"]))) -def row_failure_outcome(pod: Any, orchestrator: Any) -> dict[str, Any]: +# The per-row handlers' log messages: sync ``FunctionNode.execute`` and async +# ``process_one`` (DEBUG there, ITL-879). +ROW_FAILURE_LOGS = ("Data execution failed in %s: %s", "Data processing failed, skipping: %s") + + +def row_failure_outcome( + pod: Any, orchestrator: Any, caplog: pytest.LogCaptureFixture +) -> dict[str, Any]: """Run ``pod`` as node ``dbl`` and describe what the caller sees.""" store = InMemoryArrowDatabase() observer = CrashRecorder() job = build_job(store, ROWS, pod=pod) - try: - job.run(orchestrator=orchestrator, observer=observer) - raised = None - rows = sorted(job.nodes["dbl"].as_table()["k"].to_pylist()) - except Exception as exc: - raised = exc - rows = None + caplog.clear() + with caplog.at_level("DEBUG", logger="orcapod"): + try: + job.run(orchestrator=orchestrator, observer=observer) + raised = None + rows = sorted(job.nodes["dbl"].as_table()["k"].to_pylist()) + except Exception as exc: + raised = exc + rows = None + logs = [ + (record.levelname, record.msg) + for record in caplog.records + if record.msg in ROW_FAILURE_LOGS + ] return { "raised": raised, "rows": rows, "crashes": observer.crashes, "stored": stored_rows(store), + "logs": logs, } @@ -738,16 +753,20 @@ class TestUnstorableOutputIsARowFailure: ) @pytest.mark.parametrize("error_policy", ["continue", "fail_fast"]) def test_reported_like_a_function_failure( - self, orchestrator_cls, error_policy, pod + self, orchestrator_cls, error_policy, pod, caplog ): - """Same raise-or-return, rows, observer report and stored rows as a raise.""" + """Same raise-or-return, rows, log, observer report and stored rows as a raise.""" expected = row_failure_outcome( - raises_for_one, orchestrator_cls(error_policy=error_policy) + raises_for_one, orchestrator_cls(error_policy=error_policy), caplog + ) + actual = row_failure_outcome( + pod, orchestrator_cls(error_policy=error_policy), caplog ) - actual = row_failure_outcome(pod, orchestrator_cls(error_policy=error_policy)) assert not isinstance(actual["raised"], StoreWriteError), actual["raised"] assert (actual["raised"] is None) == (expected["raised"] is None), actual assert actual["rows"] == expected["rows"] assert actual["crashes"] == expected["crashes"] == [("dbl", "r1")] + assert len(expected["logs"]) == 1, expected["logs"] + assert actual["logs"] == expected["logs"] assert actual["stored"] == expected["stored"] From 4addbdf6d7feecc4e13e29410093be847f25b59b Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 01:16:42 -0800 Subject: [PATCH 11/15] docs(pipeline): an output that cannot be stored is a row failure (ITL-878) Co-Authored-By: Claude Opus 5.5 --- DESIGN_ISSUES.md | 6 ++++++ docs/concepts/function-pods.md | 10 ++++++++++ 2 files changed, 16 insertions(+) diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index a0968fff4..05e788d6b 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2244,6 +2244,12 @@ on it (`orcapod.utils.store_errors`), not raised in its place, so a node functio another cached pod cannot turn that pod's store failure into a skipped row. When concurrent node tasks fail together, the async orchestrator raises the store error of the node first in topological order, notes the other failures on it and logs other store errors at WARNING. +The rule (developer, November 11, 2026): store failures fail the run; a function output that +cannot be stored (a naive `datetime`, a value not matching the declared type, rejected while +`ResultCache.store` prepares the record, before any write) is a row failure governed by +`error_policy` exactly like a function that raised, not a `StoreWriteError`: the value is +specific to that row, the store is healthy, and every re-run would fail at the same row. The +async orchestrator logs row failures at DEBUG and ignores `fail_fast` (ITL-879), unchanged here. Regressions: `tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async orchestrator ignores `error_policy` (ITL-879); connector and Delta `flush` drop a batch whose write fails (ITL-880). diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index e681ba565..74c0f1b9c 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -252,6 +252,16 @@ After the failure, rows stored before it are kept: `job.run()` commits the write succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is fixed, run the job again; only the rows that are missing are computed. +A value your function returned for one row that cannot be turned into a record at all is a +different case: a naive `datetime`, a `str` from a function declared to return `int`, or an +`int` where a dataclass is declared. Such a value is rejected while the row's record is being +prepared, before anything is written, and it is a *row failure*: it is handled exactly like a +function that raised on that row. The observer gets `on_data_crash`, the failure is logged, +and `error_policy` decides -- `"continue"` skips the row, `"fail_fast"` raises the error +the conversion raised. It is not a `StoreWriteError`, because the store is healthy and every +re-run would fail at the same row. The asynchronous orchestrator currently logs row failures +at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these rows too. + This is the contract for now. An opt-in mode that tolerates store failures may be added later. From 92f4befbfd24a9291a3ad3f3f8468dc8f4236e83 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 01:17:16 -0800 Subject: [PATCH 12/15] docs(changelog): StoreWriteError and the store-write failure contract (ITL-878) Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8053fbb77..462badcb1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,26 @@ # Changelog +## [0.3.0] — unreleased + +### Added + +- **`orcapod.errors.StoreWriteError`** — raised when a node's results or records cannot be + written to the store. It names the node, the row's tag and the table, and chains the + database's own exception as its cause. (ITL-878 #287) + +### Fixed + +- A failed store write fails the run. `PipelineJob.run()` used to return normally when a + node's result or record could not be written (the row was skipped like a crashed function + under the default `error_policy="continue"`), so the node read back fewer rows than it + computed. It now raises `StoreWriteError` at the first failed write, whatever + `error_policy` is, on both orchestrators (the async one no longer raises an exception + group). Rows stored before the failure are committed and kept; re-running computes only + the missing rows. A function output that cannot be turned into a record (a naive + `datetime`, a value not matching the declared type) is still a row failure governed by + `error_policy`, like a function that raised. See *When the store cannot be written* in + the function-pods guide. (ITL-878 #287) + ## [0.2.0] — 2026-11-10 **Distribution:** Orcapod is MIT licensed starting with `0.2.0` (it was proprietary on From d305c806f61c6761a5c5ab7b34fa56375a659e3b Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 01:18:27 -0800 Subject: [PATCH 13/15] style(pipeline): ruff format the row-failure pinning test (ITL-878) Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-11-10-itl-878-store-write-failure.md | 8 +++++--- tests/test_pipeline/test_store_write_failure.py | 5 ++++- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md index 246e33bb2..83d92642d 100644 --- a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -275,11 +275,13 @@ tests) is kept. 3 outputs, each compared with a function raising on the same row) also assert the per-row log record (level + message) matches the function-failure one, besides the observer `on_data_crash`, raise-or-return, rows and stored rows. -- [ ] Docs (`docs/concepts/function-pods.md`, "When the store cannot be written"): store +- [x] Docs (`docs/concepts/function-pods.md`, "When the store cannot be written"): store failures fail the run; a function output that cannot be stored is a row failure; async logs row failures at DEBUG and ignores `fail_fast` (ITL-879), unchanged here. -- [ ] `DESIGN_ISSUES.md` SW1: the same rule. -- [ ] Decline finding (1) with a signed top-level PR comment. +- [x] `DESIGN_ISSUES.md` SW1: the same rule. +- [x] Decline finding (1) with a signed top-level PR comment (issuecomment-6791981712). +- [x] Merged origin/main b197544e (DESIGN_ISSUES conflict: both appended sections kept); + added the `[0.3.0] — unreleased` CHANGELOG section with this PR's entries. - [ ] Merge origin/main (b197544e; CHANGELOG entries under `[0.3.0] — unreleased`), full suite on 3.11/3.12/3.13, golden unchanged, ruff, format, basedpyright, push, CI, CodeRabbit round (bodies too). Reconcile with #290 if it merged first. diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index ef187ce11..f7e3e6021 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -710,7 +710,10 @@ def on_data_crash(self, node_label, tag, data, error) -> None: # The per-row handlers' log messages: sync ``FunctionNode.execute`` and async # ``process_one`` (DEBUG there, ITL-879). -ROW_FAILURE_LOGS = ("Data execution failed in %s: %s", "Data processing failed, skipping: %s") +ROW_FAILURE_LOGS = ( + "Data execution failed in %s: %s", + "Data processing failed, skipping: %s", +) def row_failure_outcome( From 392ee8eddcf1bea9d66ec73f3c727d0f4f49b2ba Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 01:36:00 -0800 Subject: [PATCH 14/15] fix(pipeline): attribute a failed flush to the store, not the row; qualify retry advice (ITL-878) CodeRabbit round 3: - ResultCache.store separates add_record from its auto-flush. A failed flush raises StoreWriteError(base_path, flush=True): views share pending writes, so the failing batch may be an earlier row's or another table's. add_context then sets no row tag. The job's final flush uses flush=True too. - The StoreWriteError message says re-running does not recompute stored function results (LOG operators always recompute). - Docs: a normal return stores every result that could become a record; a side effect whose invocation-log write failed is delivered again on re-run. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 11 ++-- DESIGN_ISSUES.md | 3 + docs/concepts/function-pods.md | 13 +++- src/orcapod/core/result_cache.py | 11 +++- src/orcapod/errors.py | 43 +++++++++---- src/orcapod/pipeline/job.py | 4 +- .../2026-11-10-itl-878-store-write-failure.md | 18 ++++++ .../test_pipeline/test_store_write_failure.py | 61 ++++++++++++++++--- 8 files changed, 135 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 462badcb1..ffc35f547 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,11 +15,12 @@ under the default `error_policy="continue"`), so the node read back fewer rows than it computed. It now raises `StoreWriteError` at the first failed write, whatever `error_policy` is, on both orchestrators (the async one no longer raises an exception - group). Rows stored before the failure are committed and kept; re-running computes only - the missing rows. A function output that cannot be turned into a record (a naive - `datetime`, a value not matching the declared type) is still a row failure governed by - `error_policy`, like a function that raised. See *When the store cannot be written* in - the function-pods guide. (ITL-878 #287) + group). Rows stored before the failure are committed and kept; re-running does not + recompute stored function results. A failed flush of pending writes raises + `StoreWriteError` with `flush=True` and names the store, not a row. A function output that + cannot be turned into a record (a naive `datetime`, a value not matching the declared + type) is still a row failure governed by `error_policy`, like a function that raised. See + *When the store cannot be written* in the function-pods guide. (ITL-878 #287) ## [0.2.0] — 2026-11-10 diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 31aa2751e..ba4c26c34 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2244,6 +2244,9 @@ on it (`orcapod.utils.store_errors`), not raised in its place, so a node functio another cached pod cannot turn that pod's store failure into a skipped row. When concurrent node tasks fail together, the async orchestrator raises the store error of the node first in topological order, notes the other failures on it and logs other store errors at WARNING. +A failed flush (the result cache's per-row auto-flush or the job's final flush) raises +`StoreWriteError(flush=True)` naming the store and no row: views of one store share their +pending writes, so the failing batch may be an earlier row's or another table's. The rule (developer, November 11, 2026): store failures fail the run; a function output that cannot be stored (a naive `datetime`, a value not matching the declared type, rejected while `ResultCache.store` prepares the record, before any write) is a row failure governed by diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 74c0f1b9c..61ea20fca 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -235,7 +235,11 @@ stored -- the disk is full, the connection drops, or the database cannot hold on output types -- `job.run()` raises `StoreWriteError` (from `orcapod.errors`) at the first failed write, on both the synchronous and the asynchronous orchestrator. The error names the node, the row being stored and the table, and the database's own exception is chained as its -cause. A run that returns normally has stored every result it computed. +cause. When committing pending writes fails, the error has `flush=True` and names the store +instead of a row and table: every view of a store shares its pending writes, so the write +that failed may belong to an earlier row or another table. A run that returns normally has +stored every result that could be turned into a record; a result rejected before that is a +row failure (see below). This holds whatever `error_policy` the orchestrator was given. `error_policy="continue"` (the default) skips rows whose *function* raised and keeps going; it never skips a row because it @@ -250,7 +254,12 @@ orchestrator would have stopped at; the other failures are logged and noted on i After the failure, rows stored before it are kept: `job.run()` commits the writes that succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is -fixed, run the job again; only the rows that are missing are computed. +fixed, run the job again: stored function results are not recomputed, only the missing ones +are. An operator with `cache_mode=LOG` recomputes its output on every run, as it always does. +A side effect is different: if its delivery completed but the write to its invocation log +failed, there is no record that it completed, and `track_completion` delivers it again on the +re-run. Before re-running, check which side effects already completed, or make the side +effect safe to repeat. A value your function returned for one row that cannot be turned into a record at all is a different case: a naive `datetime`, a `str` from a function declared to return `int`, or an diff --git a/src/orcapod/core/result_cache.py b/src/orcapod/core/result_cache.py index e4dab8a96..ec9bb788f 100644 --- a/src/orcapod/core/result_cache.py +++ b/src/orcapod/core/result_cache.py @@ -294,10 +294,17 @@ def store( data_table, skip_duplicates=skip_duplicates, ) - if self._auto_flush: - self._result_database.flush() except Exception as exc: raise StoreWriteError(self._versioned_record_path) from exc + if self._auto_flush: + try: + self._result_database.flush() + except Exception as exc: + # Views of one store share their pending writes, so the batch that + # failed may be an earlier row's or another table's. + raise StoreWriteError( + self._result_database.base_path, flush=True + ) from exc def get_all_records( self, include_system_columns: bool = False diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 3c2acdbee..8528c5ba9 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -208,17 +208,21 @@ class StoreWriteError(RuntimeError): ``error_policy``: ``error_policy="continue"`` skips rows whose *function* raised, never rows that could not be stored. The run stops at the first failed write. Rows stored before the failure are kept, and re-running the - job computes only the rows that are missing. + job does not recompute stored function results. The backend's own exception is chained as ``__cause__``. Attributes: record_path: Path of the table the write was for, relative to the - database view the writer holds. + database view the writer holds. For a failed flush, the base path of + the database view that was flushed. node_label: Label of the node whose write failed, or ``None`` when the failure is not tied to one node (a flush at the end of a run). tag: Tag values of the row being stored, or ``None`` when the write - covered many rows (an operator's output) or no row. + covered many rows (an operator's output), no row, or was a flush. + flush: ``True`` when committing pending writes failed. Every view of a + store shares one set of pending writes, so the write that failed may + belong to an earlier row or another table than the one being stored. """ def __init__( @@ -226,17 +230,22 @@ def __init__( record_path: tuple[str, ...], node_label: str | None = None, tag: Mapping[str, Any] | None = None, + *, + flush: bool = False, ) -> None: """Create the error for one failed write. Args: - record_path: Path of the table the write was for. + record_path: Path of the table the write was for, or of the database + view that was flushed. node_label: Label of the node whose write failed, if the write site knows it. tag: Tag values of the row being stored, if the write covered one row. + flush: Whether committing pending writes failed. """ - self.record_path = tuple(record_path) - self.node_label = node_label - self.tag = dict(tag) if tag is not None else None + self.record_path: tuple[str, ...] = tuple(record_path) + self.node_label: str | None = node_label + self.tag: dict[str, Any] | None = dict(tag) if tag is not None else None + self.flush: bool = flush super().__init__(self.record_path, node_label, tag) @classmethod @@ -263,13 +272,16 @@ def add_context( ) -> None: """Fill in the node label and row tag if the write site did not know them. + A failed flush never takes a row tag: the pending write that failed may + belong to another row. + Args: node_label: Label of the node that owns the write. tag: Tag values of the row being stored. """ if self.node_label is None: self.node_label = node_label - if self.tag is None and tag is not None: + if self.tag is None and tag is not None and not self.flush: self.tag = dict(tag) def __str__(self) -> str: @@ -282,9 +294,18 @@ def __str__(self) -> str: if cause is not None else "" ) + retry = ( + "The run was stopped. Rows stored before this failure are kept; " + "re-running the job does not recompute stored function results." + ) + if self.flush: + node = f" for node {self.node_label!r}" if self.node_label else "" + return ( + f"Could not commit pending writes to the store{node} " + f"(store {'/'.join(self.record_path)!r}){reason}. The failing write " + "may belong to an earlier row or another table of this store. " + retry + ) return ( f"Could not write to the store for {where}{row} " - f"(table {'/'.join(self.record_path)!r}){reason}. The run was stopped. " - "Rows stored before this failure are kept; re-running the job computes " - "only the rows that are missing." + f"(table {'/'.join(self.record_path)!r}){reason}. " + retry ) diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 8309d4eca..7fe706f82 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -770,7 +770,7 @@ def run( to the store, whatever the orchestrator's ``error_policy``. The run stops at the first failed write. Writes that succeeded before it are flushed first, so rows stored before the failure are kept, and - re-running the job computes only the rows that are missing. + re-running the job does not recompute stored function results. """ import hashlib import uuid @@ -885,7 +885,7 @@ def run( pipeline_db.flush() result_db.flush() except Exception as exc: - flush_error = StoreWriteError(pipeline_db.base_path) + flush_error = StoreWriteError(pipeline_db.base_path, flush=True) # Every view of the store shares one set of pending batches, so a failed # batch can leave the ones after it unwritten. flush_error.add_note( diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md index 83d92642d..bb8f3dc20 100644 --- a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -285,3 +285,21 @@ tests) is kept. - [ ] Merge origin/main (b197544e; CHANGELOG entries under `[0.3.0] — unreleased`), full suite on 3.11/3.12/3.13, golden unchanged, ruff, format, basedpyright, push, CI, CodeRabbit round (bodies too). Reconcile with #290 if it merged first. + +--- + +## Follow-up: CodeRabbit round 3 (review of d305c806), approved by the coordinator 2026-11-11 + +- (A) docs: a normal return stores every result that could be turned into a record; point to + the row-failure paragraph. +- (B) docs: a side effect whose delivery completed but whose invocation-log write failed is + delivered again on re-run (track_completion has no record); check or make it idempotent. +- (C) `ResultCache.store`: separate `add_record` from auto-`flush`. A flush failure raises + `StoreWriteError(result_db.base_path, flush=True)`; new keyword-only `flush: bool` + attribute; `add_context` then sets the node label but not the row tag; message says pending + writes could not be committed and the failing write may belong to an earlier row or another + table. `job.py` final flush uses `flush=True`. Test first. +- (D) `StoreWriteError` message: "Rows stored before this failure are kept; re-running the job + does not recompute stored function results." Docs explain LOG operators recompute. +- Keep shape consistent with #290's `StoreReadError`. +- [ ] tests, fix, one commit, replies on 4 threads, CI, CodeRabbit round, worker_done. diff --git a/tests/test_pipeline/test_store_write_failure.py b/tests/test_pipeline/test_store_write_failure.py index f7e3e6021..ce25ef1af 100644 --- a/tests/test_pipeline/test_store_write_failure.py +++ b/tests/test_pipeline/test_store_write_failure.py @@ -51,9 +51,11 @@ class Faults: write_marker: str | None = None succeed_first: int = 0 on_flush: bool = False + flush_marker: str | None = None fail_flush_after_write_failure: bool = False matching_writes: int = 0 failed_paths: list[str] = field(default_factory=list) + pending_paths: list[str] = field(default_factory=list) class FaultyStore: @@ -72,6 +74,7 @@ def _check_write(self, record_path: tuple[str, ...]) -> None: """Raise ``OSError`` if the write to ``record_path`` is one to fail.""" faults = self._faults full_path = "/".join(self._inner.base_path + tuple(record_path)) + faults.pending_paths.append(full_path) if faults.write_marker is None or faults.write_marker not in full_path: return faults.matching_writes += 1 @@ -109,10 +112,20 @@ def add_records( ) def flush(self) -> None: - """Flush the wrapped store, or raise ``OSError`` if flushes fail.""" - if self._faults.on_flush: + """Flush the wrapped store, or raise ``OSError`` if flushes fail. + + With ``flush_marker`` set, a flush fails only while a write to a matching + table is pending, whichever view the flush is called on: every view shares + one set of pending batches. + """ + faults = self._faults + if faults.on_flush: raise OSError("disk unavailable while flushing") + marker = faults.flush_marker + if marker is not None and any(marker in p for p in faults.pending_paths): + raise OSError(f"disk unavailable while flushing a pending {marker} batch") self._inner.flush() + faults.pending_paths.clear() def __getattr__(self, name: str) -> Any: """Delegate everything else to the wrapped store.""" @@ -187,6 +200,7 @@ def assert_store_error( *, node_label: str | None, names_row: bool, + flush: bool = False, ) -> None: """The run's error is a plain ``StoreWriteError`` naming node, row and table.""" assert isinstance(err, StoreWriteError), f"{type(err).__name__}: {err}" @@ -200,8 +214,12 @@ def assert_store_error( if faults.failed_paths: assert "/".join(err.record_path) == faults.failed_paths[0] assert faults.failed_paths[0] in message, message + assert err.flush is flush + if flush: + assert err.tag is None and "for row" not in message, message + assert "pending writes" in message and "earlier row" in message, message assert isinstance(err.__cause__, OSError) - assert "rows that are missing" in message + assert "does not recompute stored function results" in message class FailFlushAtRunEnd(NoOpObserver): @@ -256,7 +274,7 @@ def test_result_table_failure_on_later_row_keeps_earlier_rows( assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) assert stored_rows(inner) == 1 - # A re-run on a healthy store computes only the rows that are missing. + # A re-run on a healthy store does not recompute the stored rows. calls.clear() rerun = build_job(inner, ROWS) rerun.run(orchestrator=orchestrator_cls()) @@ -317,12 +335,39 @@ def test_store_failure_raises_under_any_error_policy( def test_auto_flush_failure_raises_naming_the_node(self, orchestrator_cls): """A failure of the result cache's flush after a row raises, naming the node.""" faults = Faults(on_flush=True) + store = FaultyStore(InMemoryArrowDatabase(), faults) + job = build_job(store, ROWS) + + with pytest.raises(StoreWriteError) as exc_info: + job.run(orchestrator=orchestrator_cls()) + + assert_store_error( + exc_info.value, faults, node_label="dbl", names_row=False, flush=True + ) + + def test_auto_flush_failure_of_an_earlier_write_names_no_row( + self, orchestrator_cls + ): + """A result auto-flush that fails on another table's pending write names + neither the row being stored nor the result table. + + The sync orchestrator meets the failure in the next row's result auto-flush. + The async one may write every pipeline record after the last result flush, + so the job's final flush can be the one that fails; either way it is a flush + error that names no row. + """ + faults = Faults(flush_marker=PIPELINE_TABLE) job = build_job(FaultyStore(InMemoryArrowDatabase(), faults), ROWS) with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls()) - assert_store_error(exc_info.value, faults, node_label="dbl", names_row=True) + err = exc_info.value + assert_store_error(err, faults, node_label=None, names_row=False, flush=True) + if orchestrator_cls is SyncPipelineOrchestrator: + assert err.node_label == "dbl" + assert RESULT_TABLE not in "/".join(err.record_path) + assert RESULT_TABLE not in str(err) def test_final_flush_failure_raises(self, orchestrator_cls): """A failure of the job's final flush raises ``StoreWriteError``.""" @@ -332,7 +377,9 @@ def test_final_flush_failure_raises(self, orchestrator_cls): with pytest.raises(StoreWriteError) as exc_info: job.run(orchestrator=orchestrator_cls(), observer=FailFlushAtRunEnd(faults)) - assert_store_error(exc_info.value, faults, node_label=None, names_row=False) + assert_store_error( + exc_info.value, faults, node_label=None, names_row=False, flush=True + ) assert exc_info.value.record_path == (JOB_NAME,) def test_function_failure_is_still_skipped_under_continue(self, orchestrator_cls): @@ -400,7 +447,7 @@ def test_final_flush_failure_warns_pending_output(self, orchestrator_cls): job.run(orchestrator=orchestrator_cls(), observer=FailFlushAtRunEnd(faults)) err = exc_info.value - assert_store_error(err, faults, node_label=None, names_row=False) + assert_store_error(err, faults, node_label=None, names_row=False, flush=True) notes = getattr(err, "__notes__", []) assert any( "pending" in note From 0894556dcad55e60739332acbcd154a8eec6d5b2 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 01:42:03 -0800 Subject: [PATCH 15/15] docs(plan): CodeRabbit round 3 done (ITL-878) Co-Authored-By: Claude Opus 5.5 --- superpowers/plans/2026-11-10-itl-878-store-write-failure.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md index bb8f3dc20..bf329d202 100644 --- a/superpowers/plans/2026-11-10-itl-878-store-write-failure.md +++ b/superpowers/plans/2026-11-10-itl-878-store-write-failure.md @@ -282,7 +282,7 @@ tests) is kept. - [x] Decline finding (1) with a signed top-level PR comment (issuecomment-6791981712). - [x] Merged origin/main b197544e (DESIGN_ISSUES conflict: both appended sections kept); added the `[0.3.0] — unreleased` CHANGELOG section with this PR's entries. -- [ ] Merge origin/main (b197544e; CHANGELOG entries under `[0.3.0] — unreleased`), full +- [x] Merge origin/main (b197544e; CHANGELOG entries under `[0.3.0] — unreleased`), full suite on 3.11/3.12/3.13, golden unchanged, ruff, format, basedpyright, push, CI, CodeRabbit round (bodies too). Reconcile with #290 if it merged first. @@ -302,4 +302,5 @@ tests) is kept. - (D) `StoreWriteError` message: "Rows stored before this failure are kept; re-running the job does not recompute stored function results." Docs explain LOG operators recompute. - Keep shape consistent with #290's `StoreReadError`. -- [ ] tests, fix, one commit, replies on 4 threads, CI, CodeRabbit round, worker_done. +- [x] tests, fix (392ee8ed), replies on the 4 threads. +- [ ] CI and CodeRabbit on the final head, worker_done.