diff --git a/.zed/rules b/.zed/rules index f6e613504..0c226b8ac 100644 --- a/.zed/rules +++ b/.zed/rules @@ -46,6 +46,15 @@ Just change the code and update all references directly. Never manipulate sys.modules directly (e.g. sys.modules.setdefault). If a subpackage import path doesn't work, create a proper re-export package with an __init__.py instead. +## No message-only parameters + +Never pass a value into a function only so that the function can put it in an error or log +message (developer, November 11, 2026). If the function does not use the value to do its job, +it must not receive it: the function raises what it knows, and the caller, which has the +context, catches and wraps or annotates the error (path, row, column, node label). A helper +whose job is to report or record (log a suppressed failure, attach a note) legitimately takes +the label and context it records. + ## Docstrings Use Google style (https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) diff --git a/CHANGELOG.md b/CHANGELOG.md index 36ae8fd95..2044dd015 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,25 +2,80 @@ ## [0.3.0] — unreleased +### Breaking changes and migration + +#### A changed column type is refused when appending to a Delta table (ITL-884) + +Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows' +column types with the stored ones. If a column's type differs, the flush raises +`InputValidationError` naming the table, the column and both types, and writes nothing to that +table (other tables' pending rows in the same flush may still be committed). Before, +Delta's merge silently cast the new rows to the stored type (`int32` or `float64` rows written +into an `int64` column, `large_binary` into `large_string`), or, when an extension (logical) type +was involved, accepted rows the table could then no longer read. The comparison treats types the +way Delta stores them: `string` and `large_string` (and the other large and non-large layouts) +are the same type, types the Delta writer converts on the way in are compared as converted +(unsigned integers as signed, every timestamp in microseconds), an extension type is compared by +its storage type, and nullability is not compared. Struct fields are compared by name at any depth, including inside lists and maps. +New and missing columns, and new and missing struct fields, pass the check; whether Delta's +merge adds a new one follows the database's `allow_schema_evolution` setting, as before. + +`DeltaTableDatabase.flush()` (and `add_records(..., flush=True)`) now raises a +`StoreReadError` or this `InputValidationError` as itself. Before, it wrapped every failure in a +`RuntimeError`. When several tables fail in one flush, the `StoreReadError` is raised, or else the +first `InputValidationError`, with each other failure attached as a note and logged. Failed rows +stay queued for the next flush. A flush that fails only for other reasons still raises +`RuntimeError`, with the first failure as its cause. + ### Added - **`orcapod.errors.StoreWriteError`** — raised when a node's results or records cannot be written to the store. It names the node, the row's tag and the table, and chains the database's own exception as its cause. (ITL-878 #287) +- **`orcapod.errors.StoreReadError`** — raised when a stored table exists but cannot be read. + It names the table, and the node and row being processed when they are known, and chains + the backend's exception as its cause. (ITL-884 #290) +- **`orcapod.errors.StoreError`** — the base of `StoreReadError` and `StoreWriteError`; + `except StoreError` handles both. (ITL-884 #290) ### Fixed -- A failed store write fails the run. `PipelineJob.run()` used to return normally when a - node's result or record could not be written (the row was skipped like a crashed function - under the default `error_policy="continue"`), so the node read back fewer rows than it - computed. It now raises `StoreWriteError` at the first failed write, whatever - `error_policy` is, on both orchestrators (the async one no longer raises an exception - group). Rows stored before the failure are committed and kept; re-running does not - recompute stored function results. A failed flush of pending writes raises - `StoreWriteError` with `flush=True` and names the store, not a row. A function output that - cannot be turned into a record (a naive `datetime`, a value not matching the declared - type) is still a row failure governed by `error_policy`, like a function that raised. See - *When the store cannot be written* in the function-pods guide. (ITL-878 #287) +#### A Delta table that cannot be read is no longer reported as having no rows (ITL-884) + +When a `DeltaTableDatabase` table existed but could not be read (a missing or damaged data +file, a log the backend cannot load, data files whose column type no longer matches the +table), `get_all_records` and `get_records_by_ids` returned `None`, the same answer as for a +table that does not exist. A `PipelineJob` then warned that the stored results "may have been +deleted externally", swallowed the per-row read errors under `error_policy="continue"`, finished +without raising, and its nodes showed no records. Every read now raises +`orcapod.errors.StoreReadError`, which names the table and chains the backend's error, and a +store read failure fails the run under both orchestrators whatever the `error_policy`, also +when it happens while a row's result is being stored. A table that does not exist still reads as +empty. + +A flush that could not load an existing table no longer treats it as absent. Before this fix a +`DeltaProtocolError` while loading the table made the flush write the batch with +`mode="overwrite"`, replacing the stored rows. The flush now raises and writes nothing. + +A write that fails because the stored table cannot be read raises `StoreReadError`, not +`StoreWriteError`. When read and write failures happen in the same run, both orchestrators +raise the error of the node that comes first in the pipeline (ITL-884 #290). + +#### A failed store write fails the run (ITL-878) + +A failed store write fails the run. `PipelineJob.run()` used to return normally when a node's +result or record could not be written (the row was skipped like a crashed function under the +default `error_policy="continue"`), so the node read back fewer rows than it computed. It now +raises `StoreWriteError` at the first failed write, whatever `error_policy` is, on both +orchestrators (the async one no longer raises an exception group). Rows stored before the +failure are committed and kept; re-running does not recompute stored function results. A failed +flush of pending writes raises `StoreWriteError` with `flush=True` and names the store, not a +row. A function output that cannot be turned into a record (a naive `datetime`, a value not +matching the declared type) is still a row failure governed by `error_policy`, like a function +that raised. See *When the store cannot be written* in the function-pods guide. (ITL-878 #287) + +#### Newcomer papercuts (ITL-821) + - `orcapod.enable_file_hash_caching` is exported at the top level (ITL-821). - A source built from data with no rows and no schema (e.g. `DictSource(data=[])`) raises `ValueError: Table is empty` instead of `tag_columns not found in table` (ITL-821). diff --git a/CLAUDE.md b/CLAUDE.md index bc810d382..1908ec144 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,6 +50,15 @@ Just change the code and update all references directly. Never manipulate `sys.modules` directly (e.g. `sys.modules.setdefault`). If a subpackage import path doesn't work, create a proper re-export package with an `__init__.py` instead. +## No message-only parameters + +Never pass a value into a function only so that the function can put it in an error or log +message (developer, November 11, 2026). If the function does not use the value to do its job, +it must not receive it: the function raises what it knows, and the caller, which has the +context, catches and wraps or annotates the error (path, row, column, node label). A helper +whose job is to report or record (log a suppressed failure, attach a note) legitimately takes +the label and context it records. + ## Docstrings Use [Google style](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 4e8d63d3b..24f977a51 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2296,6 +2296,57 @@ yields an empty result on the async path instead of an error. `Join`'s async key --- +## `src/orcapod/databases/delta_lake_databases.py` — read failures + +### DR1 — A Delta table that could not be read was reported as having no rows +**Status:** resolved +**Severity:** critical +**Issue:** ITL-884 + +`DeltaTableDatabase.get_all_records` logged a read error at WARNING and returned `None`, and +`get_records_by_ids` returned `None` on any error. Both conflated "the table does not exist" +with "the table exists but cannot be read". Callers treat `None` as no rows. A function node +recomputed with a "deleted externally" warning, its per-row lookups failed inside the +`error_policy="continue"` handler, and the run finished without raising, so stored results +looked dropped. `_flush_by_key` also treated a `DeltaProtocolError` on load as "no table" and +wrote the batch with `mode="overwrite"`. The case that exposed it: an append whose column type +differs from the stored table, which Delta's merge either casts silently to the stored type or, +with an extension type involved, accepts and then cannot read. Other backends were not +affected: the in-memory and connector databases (SQLite, PostgreSQL, SpiralDB) return `None` +only for a table that does not exist and let read errors propagate. + +**Fix:** only `TableNotFoundError` means absent. Every other load or read error raises +`orcapod.errors.StoreReadError`, which names the table and chains the cause. `FunctionNode.execute`, the +async `FunctionPod` row handler and the async orchestrator let it fail the run whatever the +`error_policy`. Before merging into an existing table, `_flush_by_key` compares the batch with +the stored Delta schema through Delta's own type mapping (extension types by storage, +large-normalized, nullability ignored). On a difference it raises `InputValidationError` naming the +table, column and both types, and nothing is written; struct fields are compared by name at +every depth, so a new nested field is schema evolution, not a difference. The batch's types +are converted with the Delta writer's own conversion (`deltalake.writer._conversion`, the one +`write_deltalake` and `merge` use) before the comparison, and the stored schema is read without +field metadata, so a type the writer stores differently (timestamps in microseconds, unsigned as +signed) or a logical type whose Delta storage differs is never refused; the differential guard +`tests/test_databases/test_delta_append_types.py` appends an identical schema for every Arrow and +logical type Delta can store. `flush()` raises a `StoreReadError` or +that `InputValidationError` as itself, not wrapped in `RuntimeError`, with every other failed +batch noted on it, so a read failure in a per-row result-cache flush still fails the run. The +async orchestrator selects the `StoreReadError` from the `TaskGroup`'s exception group with a +plain `except`, noting the other failures, so a concurrent unrelated failure cannot turn it into +an `ExceptionGroup`. With ITL-878 (SW1) both errors derive from `orcapod.errors.StoreError` +and share one path: the per-row handlers, `orcapod.utils.store_errors` and both orchestrators +handle `StoreError`. A write that fails because the stored table cannot be read raises +`StoreReadError`, not `StoreWriteError`; a refused append is a `StoreWriteError` caused by the +`InputValidationError`. The orchestrators label a store error with the node it escaped from, +and the async orchestrator ranks read and write errors alike by node order. Regressions: +`tests/test_databases/test_delta_read_failures.py`, +`tests/test_pipeline/test_store_read_failure.py` and +`tests/test_pipeline/test_store_errors_together.py`. Open: whether a nullability-only change +should be refused (Delta's merge relaxes a non-null nested field to nullable) belongs to the +nullability audit, ITL-842. + +--- + ## Newcomer papercuts found by the docs audit ### NP1 — Missing export, misleading errors, inconsistent defaults, broken example, write noise diff --git a/README.md b/README.md index 29aeb992f..c2790ef5e 100644 --- a/README.md +++ b/README.md @@ -238,9 +238,9 @@ While the following is subject to change based on future development, it represe │ on success: │ ctx_observer.on_data_end(node_label, tag, data, result, cached=False) │ emit (tag_out, result) downstream - │ on StoreWriteError (a result or record could not be stored): + │ on StoreError (the store could not be read or written): │ ctx_observer.on_data_crash(node_label, tag, data, exc) - │ raise, whatever error_policy is — the run stops (ITL-878) + │ raise, whatever error_policy is — the run stops (ITL-878, ITL-884) │ on any other exception (the function failed): │ ctx_observer.on_data_crash(node_label, tag, data, exc) │ if error_policy == "fail_fast": raise diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 6363d0196..4a6a5aa77 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -219,6 +219,35 @@ print("function calls:", len(calls)) version, e.g. `@function_pod(output_keys="bmi", version="v1.0")` (the default is `"v0.0"`; a minor bump such as `"v0.1"` is not enough), or rename the function. +!!! note "When the store cannot be read" + A table that does not exist yet reads as empty, and its rows are computed. In a + `DeltaTableDatabase`, a table that exists but cannot be read (for example, a data file was + deleted or damaged) is never treated as empty: the run stops with + `orcapod.errors.StoreReadError`, which names the + table, the node and row being processed when they are known, and the underlying error, + whatever the orchestrator's `error_policy`. Its stored rows are not recomputed and the + table is not overwritten, so you can repair the table or restore the file and run again. + A write that fails because the stored table cannot be read raises `StoreReadError` too, + not `StoreWriteError` (see *When the store cannot be written* below). + + In a `DeltaTableDatabase`, rows whose column types differ from the stored table's, as + Delta stores them (`float64` values for a column stored as `int64`, say), are refused with + `InputValidationError`, which names the table, the column and both types, and nothing is + written to that table. They are not converted to the stored type. `string` and + `large_string`, and the other large and non-large layouts of one type, count as the same + type, and so do types the Delta writer converts on the way in (`uint64` is stored as + `int64`, every timestamp in microseconds). New columns and new struct fields pass this + check; whether they are added follows + the database's `allow_schema_evolution` setting. In a job run, a refused append fails the + run as a `StoreWriteError` whose cause is the `InputValidationError`. + + When you use the database directly, both errors reach you as themselves, also from + `flush()` and from `add_records(..., flush=True)`. When several pending tables fail in one flush, the + `StoreReadError` (or else the first `InputValidationError`) is raised, and each other + failure is attached to it as a note and logged. Failed rows stay queued, so a later + flush tries them again. A flush that fails for any other reason raises `RuntimeError`, + with the first failure as its cause. + A node from a job that has run also provides: - **`get_all_records()`** -- every record this step of the job has stored, as a PyArrow @@ -246,12 +275,15 @@ This holds whatever `error_policy` the orchestrator was given. `error_policy="co default) skips rows whose *function* raised and keeps going; it never skips a row because it could not be stored. -The error you get is always one plain `StoreWriteError`, never an exception group and never -another exception raised in its place. A post-run hook or observer callback that fails while -the store error is on its way out is logged and recorded as a note on the error -(`err.__notes__`). When several nodes fail at once on the asynchronous orchestrator, you get -the store error of the node that comes first in the pipeline, the one the synchronous -orchestrator would have stopped at; the other failures are logged and noted on it. +`StoreWriteError` and `StoreReadError` both derive from `orcapod.errors.StoreError`, so +`except StoreError` handles either. The error you get is always one plain store error, never +an exception group and never another exception raised in its place. A post-run hook or +observer callback that fails while the store error is on its way out is logged and recorded +as a note on the error (`err.__notes__`). When several nodes fail at once on the asynchronous +orchestrator, read and write failures alike, you get the store error of the node that comes +first in the pipeline, the one the synchronous orchestrator would have stopped at; the other +failures are logged and noted on it. A node whose failure the asynchronous orchestrator +cancelled before it happened is not reported. After the failure, rows stored before it are kept: `job.run()` commits the writes that succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 29d8b7699..3bd2c8e74 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -14,7 +14,7 @@ from orcapod.core.data_function import CachedDataFunction, PythonDataFunction from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER -from orcapod.errors import StoreWriteError +from orcapod.errors import StoreError from orcapod.utils.store_errors import keep_store_error, note_callback_failure from orcapod.protocols.core_protocols import ( ArgumentGroup, @@ -329,7 +329,7 @@ def add_post_run_hook(self, hook: PostRunHook) -> None: def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: """Fire all registered hooks with payload in registration order. - When the invocation failed with a ``StoreWriteError``, a fail-loud hook that + When the invocation failed with a ``StoreError``, a fail-loud hook that raises does not replace it: its failure is logged and noted on the store error, the remaining hooks are skipped as for any fail-loud failure, and the caller re-raises the store error. @@ -338,7 +338,7 @@ def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: payload: The post-run payload to pass to each hook. """ error = payload.stats.error - store_error = error if isinstance(error, StoreWriteError) else None + store_error = error if isinstance(error, StoreError) else None for hook in self._post_run_hooks: fn = hook.fn if isinstance(hook, HookConfig) else hook on_error = hook.on_error if isinstance(hook, HookConfig) else "raise" @@ -599,8 +599,8 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: out_tag, result_data = await self._async_invoke_with_hooks( tag, data, logger=pkt_logger, run_id=run_id ) - except StoreWriteError as exc: - # A failed store write fails the run; only rows whose + except StoreError as exc: + # A failed store read or write fails the run; only rows whose # function raised are skipped. exc.add_context(tag=tag.as_dict()) with keep_store_error(exc, "Observer on_data_crash"): diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 8586cc5f3..3454c8856 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -35,6 +35,8 @@ EphemeralResultMissingError, PipelineJobRequiredError, SchemaVersionError, + StoreError, + StoreReadError, StoreWriteError, ) from orcapod.utils.store_errors import keep_store_error @@ -1305,9 +1307,10 @@ def execute( tag_out, result = self._process_data_internal( tag, data, logger=pkt_logger, run_id=run_id ) - except StoreWriteError as exc: - # A failed store write fails the run whatever the error policy: - # error_policy only decides about rows whose function raised. + except StoreError as exc: + # A failed store read or write fails the run whatever the error + # policy: error_policy only decides about rows whose function + # raised. exc.add_context(node_label=node_label, tag=tag.as_dict()) with keep_store_error(exc, "Observer on_data_crash"): ctx_obs.on_data_crash(node_label, tag, data, exc) @@ -1800,6 +1803,9 @@ def add_pipeline_record( combined_record, skip_duplicates=True, ) + except StoreReadError: + # The stored table could not be read: report the read failure itself. + raise except Exception as exc: raise StoreWriteError( self._versioned_pipeline_path, @@ -2447,14 +2453,22 @@ async def record_and_forward() -> None: execution_pod.RESULT_COMPUTED_FLAG, False ) ) - self.add_pipeline_record( - original_tag, - input_data, - data_record_id=output_data.datagram_uuid, - computed=result_computed, - is_ephemeral=is_ephemeral, - output_data_hash=output_data.content_hash(), - ) + try: + self.add_pipeline_record( + original_tag, + input_data, + data_record_id=output_data.datagram_uuid, + computed=result_computed, + is_ephemeral=is_ephemeral, + output_data_hash=output_data.content_hash(), + ) + except StoreError as exc: + # This runs outside the pod's per-row handler: name the + # node and row here. + exc.add_context( + node_label=node_label, tag=original_tag.as_dict() + ) + raise # Update in-memory cache so iter_data() sees the result. base_entry_id = self.compute_base_entry_id( original_tag, input_data @@ -2544,9 +2558,9 @@ def on_data_crash( clean = tag.drop_meta_columns( _TAG_NODE_INPUT_REF, ignore_missing=True ) - if isinstance(error, StoreWriteError): - # The pod re-raises a failed store write after this - # call; name the node it belongs to. + if isinstance(error, StoreError): + # The pod re-raises a failed store read or write after + # this call; name the node it belongs to. error.add_context(node_label=node_label) ctx_obs.on_data_crash(node_label, clean, data, error) # Clean up the dangling input_store entry so it doesn't diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index f343ba67f..0e6c4c188 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -29,7 +29,11 @@ from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER -from orcapod.errors import PipelineJobRequiredError, StoreWriteError +from orcapod.errors import ( + PipelineJobRequiredError, + StoreReadError, + StoreWriteError, +) from orcapod.protocols.core_protocols import ( DataProtocol, StreamProtocol, @@ -851,6 +855,9 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: record_id_column=self.HASH_COLUMN_NAME, skip_duplicates=True, ) + except StoreReadError: + # The stored table could not be read: report the read failure itself. + raise except Exception as exc: raise StoreWriteError( self.node_identity_path, node_label=self.label diff --git a/src/orcapod/core/result_cache.py b/src/orcapod/core/result_cache.py index ec9bb788f..89bd0311e 100644 --- a/src/orcapod/core/result_cache.py +++ b/src/orcapod/core/result_cache.py @@ -12,7 +12,7 @@ from datetime import datetime, timezone from typing import TYPE_CHECKING, Any -from orcapod.errors import SchemaVersionError, StoreWriteError +from orcapod.errors import SchemaVersionError, StoreReadError, StoreWriteError from orcapod.protocols.core_protocols import DataProtocol from orcapod.protocols.database_protocols import ArrowDatabaseProtocol from orcapod.system_constants import constants, RESULT_DB_SCHEMA_VERSION @@ -294,11 +294,17 @@ def store( data_table, skip_duplicates=skip_duplicates, ) + except StoreReadError: + # The stored table could not be read: report the read failure itself. + raise except Exception as exc: raise StoreWriteError(self._versioned_record_path) from exc if self._auto_flush: try: self._result_database.flush() + except StoreReadError: + # The stored table could not be read: report the read failure itself. + raise except Exception as exc: # Views of one store share their pending writes, so the batch that # failed may be an earlier row's or another table's. diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index c7dec5dcf..8c34c1f04 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -7,6 +7,7 @@ from typing import TYPE_CHECKING, Any, Literal, cast from orcapod.databases.utils import coerce_record_id +from orcapod.errors import InputValidationError, StoreReadError from orcapod.databases.storage_utils import is_cloud_uri, parse_base_path from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -27,6 +28,259 @@ logger = logging.getLogger(__name__) +def _comparable_storage_type(arrow_type: pa.DataType) -> pa.DataType: + """Return ``arrow_type`` reduced to what the append check compares. + + Extension types are replaced by their storage type and every nested field is + made nullable, at every depth. The result is used only to compare a batch + with a stored table (ITL-884); it is never stored, so dropping nested + non-null flags here changes no schema or hash. + + Args: + arrow_type: The Arrow type to reduce. + + Returns: + The storage type with all nested fields nullable. + """ + if isinstance(arrow_type, pa.ExtensionType): + return _comparable_storage_type(arrow_type.storage_type) + if pa.types.is_struct(arrow_type): + return pa.struct( + [pa.field(f.name, _comparable_storage_type(f.type)) for f in arrow_type] + ) + if pa.types.is_large_list(arrow_type): + field = arrow_type.value_field + return pa.large_list(pa.field(field.name, _comparable_storage_type(field.type))) + if pa.types.is_list(arrow_type): + field = arrow_type.value_field + return pa.list_(pa.field(field.name, _comparable_storage_type(field.type))) + if pa.types.is_fixed_size_list(arrow_type): + field = arrow_type.value_field + return pa.list_( + pa.field(field.name, _comparable_storage_type(field.type)), + arrow_type.list_size, + ) + if pa.types.is_map(arrow_type): + key, item = arrow_type.key_field, arrow_type.item_field + # pyarrow accepts key and item Fields here; its stubs only list types. + return pa.map_( # pyright: ignore[reportCallIssue] + pa.field(key.name, _comparable_storage_type(key.type), nullable=False), + pa.field(item.name, _comparable_storage_type(item.type)), + keys_sorted=arrow_type.keys_sorted, + ) + return arrow_type + + +def _without_null_types(arrow_type: pa.DataType) -> pa.DataType | None: + """Return ``arrow_type`` without the parts typed ``null``, or ``None`` if none is left. + + A ``null``-typed value fits a column of any type, and Delta has no ``null`` + type to compare it with, so the append check leaves it out at every depth: a + ``null`` struct field is dropped (its siblings are still compared, inside lists + and maps too), and a type that is ``null``, or whose list items or map keys or + values are entirely ``null``, is not compared. + + Args: + arrow_type: A type already reduced by ``_comparable_storage_type``. + + Returns: + The type to compare, or ``None`` when it should not be compared. + """ + if pa.types.is_null(arrow_type): + return None + if pa.types.is_struct(arrow_type): + kept = [ + pa.field(f.name, t) + for f in arrow_type + if (t := _without_null_types(f.type)) is not None + ] + return pa.struct(kept) if kept else None + if pa.types.is_large_list(arrow_type) or pa.types.is_list(arrow_type): + item = _without_null_types(arrow_type.value_type) + if item is None: + return None + field = pa.field(arrow_type.value_field.name, item) + return ( + pa.large_list(field) + if pa.types.is_large_list(arrow_type) + else pa.list_(field) + ) + if pa.types.is_fixed_size_list(arrow_type): + item = _without_null_types(arrow_type.value_type) + if item is None: + return None + return pa.list_( + pa.field(arrow_type.value_field.name, item), arrow_type.list_size + ) + if pa.types.is_map(arrow_type): + key = _without_null_types(arrow_type.key_type) + value = _without_null_types(arrow_type.item_type) + if key is None or value is None: + return None + # pyarrow accepts key and item Fields here; its stubs only list types. + return pa.map_( # pyright: ignore[reportCallIssue] + pa.field(arrow_type.key_field.name, key, nullable=False), + pa.field(arrow_type.item_field.name, value), + keys_sorted=arrow_type.keys_sorted, + ) + return arrow_type + + +def _as_delta_writer_stores(schema: pa.Schema) -> pa.Schema: + """Return ``schema`` with the types the Delta writer would store it as. + + Applies the conversion ``deltalake.write_deltalake`` and ``DeltaTable.merge`` + apply to incoming data (unsigned to signed integers, every timestamp to + microseconds, fixed-size binary to binary), then Delta's own type mapping. + Calling the writer's conversion instead of copying it keeps the append check + in step with what the writer accepts; ``test_identical_schema_appends`` + guards it across ``deltalake`` versions. + + Args: + schema: The schema of the rows about to be written. + + Returns: + The schema as the Delta table would store it. + """ + from arro3.core import Schema as Arro3Schema + from deltalake.writer._conversion import ( # pyright: ignore[reportPrivateUsage] # the writer's own conversion, see the docstring + _convert_arro3_schema_to_delta, # pyright: ignore[reportPrivateUsage] + ) + + converted = _convert_arro3_schema_to_delta(Arro3Schema.from_arrow(schema)) + return pa.schema( + deltalake.Schema.from_arrow(converted).to_arrow() # pyright: ignore[reportArgumentType] # arro3 schema; pyarrow reads it through the C data interface + ) + + +def _without_field_metadata(delta_type: Any) -> Any: + """Return a Delta type with the metadata of every nested field dropped. + + Args: + delta_type: A ``deltalake.schema`` type. + + Returns: + The same type with no field metadata at any depth. + """ + from deltalake.schema import ArrayType, Field, MapType, StructType + + if isinstance(delta_type, StructType): + return StructType( + [ + Field(f.name, _without_field_metadata(f.type), nullable=f.nullable) + for f in delta_type.fields + ] + ) + if isinstance(delta_type, ArrayType): + return ArrayType( + _without_field_metadata(delta_type.element_type), + contains_null=delta_type.contains_null, + ) + if isinstance(delta_type, MapType): + return MapType( + _without_field_metadata(delta_type.key_type), + _without_field_metadata(delta_type.value_type), + value_contains_null=delta_type.value_contains_null, + ) + return delta_type + + +def _stored_arrow_schema(delta_table: deltalake.DeltaTable) -> pa.Schema: + """Return the Arrow schema of the table's stored Delta schema, without metadata. + + Delta keeps the Arrow field metadata it was written with, including the + extension type name of a logical type. Reading that back into pyarrow would + rebuild the registered extension type over Delta's own storage type (``list`` + where Orcapod wrote ``large_list``), which the extension type rejects. The + append check compares storage types only, so the metadata is dropped first. + + Args: + delta_table: The loaded stored table. + + Returns: + The stored schema as plain Arrow types. + """ + from deltalake.schema import Field, Schema + + stored = delta_table.schema() + plain = Schema( + [ + Field(f.name, _without_field_metadata(f.type), nullable=f.nullable) + for f in stored.fields + ] + ) + return pa.schema(plain.to_arrow()) # pyright: ignore[reportArgumentType] # arro3 schema; pyarrow reads it through the C data interface + + +def _types_conflict(stored: pa.DataType, incoming: pa.DataType) -> bool: + """Return ``True`` if ``incoming`` changes a type that ``stored`` already has. + + Both types must already be reduced by ``_comparable_storage_type``. Struct + fields are matched by name at every depth, through list and map types. A + field present on only one side is not a conflict, just as a top-level column + present on only one side is not: adding or omitting it is schema evolution, + which Delta's merge accepts or rejects itself. Any other difference is a + conflict. + + Args: + stored: The type the stored table has. + incoming: The type the new rows have. + + Returns: + Whether a type the stored table already has would change. + """ + if pa.types.is_struct(stored) and pa.types.is_struct(incoming): + stored_fields = {f.name: f.type for f in stored} + return any( + _types_conflict(stored_fields[f.name], f.type) + for f in incoming + if f.name in stored_fields + ) + if (pa.types.is_large_list(stored) and pa.types.is_large_list(incoming)) or ( + pa.types.is_list(stored) and pa.types.is_list(incoming) + ): + return _types_conflict(stored.value_type, incoming.value_type) + if pa.types.is_fixed_size_list(stored) and pa.types.is_fixed_size_list(incoming): + return stored.list_size != incoming.list_size or _types_conflict( + stored.value_type, incoming.value_type + ) + if pa.types.is_map(stored) and pa.types.is_map(incoming): + return _types_conflict(stored.key_type, incoming.key_type) or _types_conflict( + stored.item_type, incoming.item_type + ) + return stored != incoming + + +def _store_error_to_raise(errors: list[Exception]) -> Exception | None: + """Pick the store error a failed flush raises as itself. + + A ``StoreReadError`` is preferred, because it is the error that fails a run + whatever the orchestrator's ``error_policy``; failing that, an + ``InputValidationError`` from a refused append. + + Args: + errors: The error of every batch that failed, in flush order. + + Returns: + The first ``StoreReadError``, else the first ``InputValidationError``, or + ``None`` when no batch failed with a store error. + """ + for kind in (StoreReadError, InputValidationError): + chosen = next((e for e in errors if isinstance(e, kind)), None) + if chosen is not None: + return chosen + return None + + +class _DeltaTableReadError(Exception): + """A Delta table's data could not be read; the backend error is the ``__cause__``. + + Raised by ``DeltaTableDatabase._read_delta_table`` so that its callers, which + know the table's path, can tell a failed read from a bad filter and raise + ``StoreReadError``. + """ + + class DeltaTableDatabase: """ A Delta table database with clear insert vs update semantics. @@ -213,11 +467,23 @@ def _get_delta_table( # Table doesn't exist return None except Exception as e: - logger.error(f"Error loading Delta table for {record_key}: {e}") - # Try to clear any corrupted cache - if record_key in self._delta_table_cache: - self._delta_table_cache.pop(record_key) - raise + # The table exists but could not be loaded: never report it as absent. + self._delta_table_cache.pop(record_key, None) + raise self._read_failed(record_path) from e + + def _read_failed(self, record_path: tuple[str, ...]) -> StoreReadError: + """Build the error for a table at ``record_path`` that exists but cannot be read. + + Args: + record_path: Path components relative to this database view. + + Returns: + A ``StoreReadError`` naming the table; the caller raises it from the + backend's exception. + """ + return StoreReadError( + self._get_table_uri(record_path), self._path_prefix + record_path + ) def table_exists(self, record_path: tuple[str, ...]) -> bool: """Return ``True`` if a Delta table exists at ``record_path``. @@ -317,28 +583,26 @@ def _refresh_existing_ids_cache(self, record_path: tuple[str, ...]) -> None: # Get all existing IDs from Delta table using standard RECORD_ID_COLUMN # TODO: replace this with more targetted loading of only the target column and in batches arrow_table = delta_table.to_pyarrow_table() - if arrow_table.num_rows == 0: - self._existing_ids_cache[record_key] = set() - elif self.RECORD_ID_COLUMN not in arrow_table.column_names: - # TODO: replace this with proper checking of the table schema first! - logger.warning(f"Delta table missing {self.RECORD_ID_COLUMN} column") - self._existing_ids_cache[record_key] = set() - else: - existing_ids = cast( - set[str], set(arrow_table[self.RECORD_ID_COLUMN].to_pylist()) - ) - self._existing_ids_cache[record_key] = existing_ids + except Exception as e: + # Leave the cache dirty so the next call reads the table again. + raise self._read_failed(record_path) from e - self._cache_dirty[record_key] = False - logger.debug( - f"Refreshed existing IDs cache: {len(self._existing_ids_cache)} IDs" + if arrow_table.num_rows == 0: + self._existing_ids_cache[record_key] = set() + elif self.RECORD_ID_COLUMN not in arrow_table.column_names: + # TODO: replace this with proper checking of the table schema first! + logger.warning(f"Delta table missing {self.RECORD_ID_COLUMN} column") + self._existing_ids_cache[record_key] = set() + else: + existing_ids = cast( + set[str], set(arrow_table[self.RECORD_ID_COLUMN].to_pylist()) ) + self._existing_ids_cache[record_key] = existing_ids - except Exception as e: - logger.error(f"Failed to refresh existing IDs cache: {e}") - self._existing_ids_cache[record_key] = set() - self._cache_dirty[record_key] = False - raise + self._cache_dirty[record_key] = False + logger.debug( + f"Refreshed existing IDs cache: {len(self._existing_ids_cache)} IDs" + ) def _get_existing_ids(self, record_path: tuple[str, ...]) -> set[str]: """Get the set of existing IDs in the Delta table, using cache when possible.""" @@ -631,6 +895,9 @@ def get_all_records( Returns: Combined Arrow table with all records, or None if no records exist + + Raises: + StoreReadError: If the Delta table exists but could not be read. """ record_key = self._get_record_key(record_path) @@ -642,10 +909,10 @@ def get_all_records( delta_table_data = delta_table.to_pyarrow_dataset( as_large_types=True ).to_table() - if delta_table_data.num_rows > 0: - tables_to_combine.append(delta_table_data) except Exception as e: - logger.warning(f"Error reading Delta table: {e}") + raise self._read_failed(record_path) from e + if delta_table_data.num_rows > 0: + tables_to_combine.append(delta_table_data) # Add pending batch data if ( @@ -700,21 +967,17 @@ def get_records_with_column_value( if delta_table is None: return None + # Use schema-preserving read try: - # Use schema-preserving read result = self._read_delta_table(delta_table, expression=combined_expression) + except _DeltaTableReadError as e: + raise self._read_failed(record_path) from e.__cause__ - if len(result) == 0: - return None + if len(result) == 0: + return None - # Handle (remove/rename) the record id column before returning - return self._handle_record_id_column(result, record_id_column) - - except Exception as e: - logger.error( - f"Error getting record with {column_values} from {'/'.join(record_path)}: {e}" - ) - raise e + # Handle (remove/rename) the record id column before returning + return self._handle_record_id_column(result, record_id_column) def get_record_by_id( self, @@ -758,22 +1021,18 @@ def get_record_by_id( if delta_table is None: return None + # Use schema-preserving read + filter_expr = self._create_record_id_filter(record_id) try: - # Use schema-preserving read - filter_expr = self._create_record_id_filter(record_id) result = self._read_delta_table(delta_table, filters=filter_expr) + except _DeltaTableReadError as e: + raise self._read_failed(record_path) from e.__cause__ - if len(result) == 0: - return None + if len(result) == 0: + return None - # Handle (remove/rename) the record id column before returning - return self._handle_record_id_column(result, record_id_column) - - except Exception as e: - logger.error( - f"Error getting record {record_id} from {'/'.join(record_path)}: {e}" - ) - raise e + # Handle (remove/rename) the record id column before returning + return self._handle_record_id_column(result, record_id_column) def get_records_by_ids( self, @@ -815,23 +1074,19 @@ def get_records_by_ids( delta_table = self._get_delta_table(record_path) if delta_table is None: return None + # Use schema-preserving read with filters + filter_expr = self._create_record_ids_filter(record_ids_list) try: - # Use schema-preserving read with filters - filter_expr = self._create_record_ids_filter(record_ids_list) result = self._read_delta_table(delta_table, filters=filter_expr) + except _DeltaTableReadError as e: + raise self._read_failed(record_path) from e.__cause__ - if len(result) == 0: - return None - - # Handle record_id column based on parameter - return self._handle_record_id_column(result, record_id_column) - - except Exception as e: - logger.error( - f"Error getting records by IDs from {'/'.join(record_path)}: {e}" - ) + if len(result) == 0: return None + # Handle record_id column based on parameter + return self._handle_record_id_column(result, record_id_column) + def _read_delta_table( self, delta_table: deltalake.DeltaTable, @@ -844,13 +1099,22 @@ def _read_delta_table( Args: delta_table: The Delta table to read from filters: Optional filters to apply + expression: Optional filter expression, used instead of ``filters`` Returns: Arrow table with preserved schema + + Raises: + _DeltaTableReadError: If the table's data could not be read, with the + backend's error as its ``__cause__``. An error in the filter itself + (an unknown column, say) is raised as is. """ filter_expr = None # Use to_pyarrow_dataset with as_large_types for Polars compatible arrow table loading - dataset = delta_table.to_pyarrow_dataset(as_large_types=True) + try: + dataset = delta_table.to_pyarrow_dataset(as_large_types=True) + except Exception as e: + raise _DeltaTableReadError from e if filters and expression is None: for filt in filters: if len(filt) == 3: @@ -876,7 +1140,11 @@ def _read_delta_table( # Convert view types before filtering: PyArrow has no comparison/sort # kernels for string_view, so filtering the scan directly crashes # (ENG-601). Trades predicate pushdown for correctness. - table = arrow_utils.normalize_table_view_types(dataset.to_table()) + try: + scanned = dataset.to_table() + except Exception as e: + raise _DeltaTableReadError from e + table = arrow_utils.normalize_table_view_types(scanned) if filter_expr is not None: return table.filter(filter_expr) return table @@ -925,7 +1193,21 @@ def from_config(cls, config: dict[str, Any]) -> "DeltaTableDatabase": ) def flush(self) -> None: - """Flush all pending batches.""" + """Flush all pending batches. + + Every pending batch is attempted, even after one fails. A batch that fails + stays queued, so a later flush tries it again, and each failure is logged. + + Raises: + StoreReadError: If a batch's existing table could not be loaded. It is + raised as itself; every other batch that failed in this flush is + added to it as a note. + InputValidationError: If no batch failed with ``StoreReadError`` but a + batch's column types differ from its stored table's. Other failed + batches are added to it as notes. + RuntimeError: If batches failed with other errors only, naming the + failed batches, with the first failure as its ``__cause__``. + """ errors: list[tuple[str, Exception]] = [] for record_key in list(self._pending_batches.keys()): try: @@ -934,6 +1216,16 @@ def flush(self) -> None: logger.error(f"Error flushing batch for {record_key}: {e}") errors.append((record_key, e)) if errors: + store_error = _store_error_to_raise([e for _, e in errors]) + if store_error is not None: + # Keep every other failure visible on the error that is raised. + for record_key, other in errors: + if other is not store_error: + store_error.add_note( + f"Another batch also failed to flush ({record_key}): " + f"{type(other).__name__}: {other}" + ) + raise store_error failed_keys = [k for k, _ in errors] raise RuntimeError( f"Failed to flush {len(errors)} batch(es): {failed_keys}" @@ -991,18 +1283,19 @@ def _flush_by_key(self, record_key: str) -> None: os.makedirs(table_uri, exist_ok=True) - # Check if Delta table exists at this URI - try: - import deltalake as _dl + # Check if Delta table exists at this URI. Only TableNotFoundError + # means "absent": a table that exists but cannot be loaded must never + # be replaced by the mode="overwrite" write below (ITL-884). + from deltalake.exceptions import TableNotFoundError - delta_table = _dl.DeltaTable( + try: + delta_table = deltalake.DeltaTable( table_uri, storage_options=self._storage_options or None ) - except ( - _dl.exceptions.TableNotFoundError, - _dl.exceptions.DeltaProtocolError, - ): + except TableNotFoundError: delta_table = None + except Exception as load_error: + raise StoreReadError(table_uri, abs_path_components) from load_error if delta_table is None: # TODO: reconsider mode="overwrite" here @@ -1017,6 +1310,16 @@ def _flush_by_key(self, record_key: str) -> None: f"Created new Delta table for {record_key} with {len(combined_table)} records" ) else: + mismatch = self._incompatible_append_column(delta_table, combined_table) + if mismatch is not None: + column, stored_type, incoming_type = mismatch + raise InputValidationError( + f"Cannot append to the stored table {table_uri!r}: column " + f"{column!r} is stored as {stored_type} but the new rows have " + f"{incoming_type} (input type " + f"{combined_table.schema.field(column).type}). " + "The stored rows are unchanged and nothing was written." + ) delta_table.merge( source=combined_table, predicate=f"target.{self.RECORD_ID_COLUMN} = source.{self.RECORD_ID_COLUMN}", @@ -1046,6 +1349,58 @@ def _flush_by_key(self, record_key: str) -> None: self._pending_record_ids[record_key] = pending_ids raise + @staticmethod + def _incompatible_append_column( + delta_table: deltalake.DeltaTable, batch: pa.Table + ) -> tuple[str, pa.DataType, pa.DataType] | None: + """Find a column whose type in ``batch`` differs from the stored table's. + + Delta's merge either casts such a batch to the stored type (``int32`` or + ``float64`` rows into an ``int64`` column) or, when an extension type is + involved, writes data files the table can no longer read. ``null``-typed + parts of the batch fit any stored type and are not compared (see + ``_without_null_types``). The batch's types + are first converted exactly as the Delta writer converts them (see + ``_as_delta_writer_stores``), then both sides are compared with extension + types reduced to storage, large-normalized and nullability ignored, so a + type Delta stores differently (``uint64`` as ``int64``, ``timestamp[ns]`` + as ``timestamp[us]``) is not reported. Columns and nested + struct fields present on only one side are allowed (schema evolution); + see ``_types_conflict``. The stored schema comes from the loaded log; no + data is read. + + Args: + delta_table: The loaded stored table. + batch: The rows about to be merged into it. + + Returns: + ``(column, stored_type, incoming_type)`` for the first column whose + compared types differ, or ``None`` when the batch can be appended. + """ + stored = _stored_arrow_schema(delta_table) + compared = [ + (f.name, _without_null_types(_comparable_storage_type(f.type))) + for f in batch.schema + ] + incoming = _as_delta_writer_stores( + pa.schema([pa.field(name, t) for name, t in compared if t is not None]) + ) + for field in incoming: + index = stored.get_field_index(field.name) + if index < 0: + continue + stored_type = _comparable_storage_type( + arrow_utils.normalize_to_large_types( + stored.field(index).type, keep_null=True + ) + ) + incoming_type = _comparable_storage_type( + arrow_utils.normalize_to_large_types(field.type, keep_null=True) + ) + if _types_conflict(stored_type, incoming_type): + return field.name, stored_type, incoming_type + return None + @property def base_path(self) -> tuple[str, ...]: """The current relative root of this database view (always () for root instances).""" diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 8528c5ba9..50fe2aaf4 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -1,5 +1,5 @@ from collections.abc import Mapping -from typing import Any +from typing import Any, Self class InputValidationError(Exception): @@ -201,17 +201,85 @@ def __init__( ) -class StoreWriteError(RuntimeError): - """Raised when a node's results or records could not be written to the store. +class StoreError(RuntimeError): + """Base of the errors raised when the store could not be read or written. - A failed store write always fails the run, whatever the orchestrator's + A store error always fails the run, whatever the orchestrator's ``error_policy``: ``error_policy="continue"`` skips rows whose *function* - raised, never rows that could not be stored. The run stops at the first - failed write. Rows stored before the failure are kept, and re-running the - job does not recompute stored function results. + raised, never rows whose stored data could not be read or written. Catch + ``StoreError`` to handle both ``StoreReadError`` and ``StoreWriteError``. The backend's own exception is chained as ``__cause__``. + Attributes: + node_label: Label of the node the failure belongs to, or ``None`` when it + is not known or not tied to one node. + tag: Tag values of the row being processed, or ``None`` when the failure + is not tied to one row. + """ + + def __init__( + self, + *args: object, + node_label: str | None = None, + tag: Mapping[str, Any] | None = None, + ) -> None: + """Create the error. + + Args: + *args: The exception arguments, as for any exception. + node_label: Label of the node the failure belongs to, if known. + tag: Tag values of the row being processed, if known. + """ + self.node_label: str | None = node_label + self.tag: dict[str, Any] | None = dict(tag) if tag is not None else None + super().__init__(*args) + + @classmethod + def in_group(cls, group: BaseExceptionGroup) -> list[Self]: + """Return every error of this class in an exception group, nested ones included. + + Args: + group: An exception group, such as the one an ``asyncio.TaskGroup`` + raises. + + Returns: + The matching leaves in the order the group holds them. + """ + found: list[Self] = [] + for exc in group.exceptions: + if isinstance(exc, BaseExceptionGroup): + found.extend(cls.in_group(exc)) + elif isinstance(exc, cls): + found.append(exc) + return found + + def add_context( + self, node_label: str | None = None, tag: Mapping[str, Any] | None = None + ) -> None: + """Fill in the node label and row tag where the raising site did not know them. + + Called by the code that runs the node or the row as the error passes + through it. Values that are already set are kept. + + Args: + node_label: Label of the node the failure belongs to. + tag: Tag values of the row being processed. + """ + if self.node_label is None: + self.node_label = node_label + if self.tag is None and tag is not None: + self.tag = dict(tag) + + +class StoreWriteError(StoreError): + """Raised when a node's results or records could not be written to the store. + + The run stops at the first failed write. Rows stored before the failure are + kept, and re-running the job does not recompute stored function results. + A write that fails because the stored table cannot be read raises + ``StoreReadError`` instead. + Attributes: record_path: Path of the table the write was for, relative to the database view the writer holds. For a failed flush, the base path of @@ -243,29 +311,10 @@ def __init__( flush: Whether committing pending writes failed. """ self.record_path: tuple[str, ...] = tuple(record_path) - self.node_label: str | None = node_label - self.tag: dict[str, Any] | None = dict(tag) if tag is not None else None self.flush: bool = flush - super().__init__(self.record_path, node_label, tag) - - @classmethod - def in_group(cls, group: BaseExceptionGroup) -> "list[StoreWriteError]": - """Return every ``StoreWriteError`` in an exception group, nested ones included. - - Args: - group: An exception group, such as the one an ``asyncio.TaskGroup`` - raises. - - Returns: - The ``StoreWriteError`` leaves in the order the group holds them. - """ - found: list[StoreWriteError] = [] - for exc in group.exceptions: - if isinstance(exc, BaseExceptionGroup): - found.extend(cls.in_group(exc)) - elif isinstance(exc, cls): - found.append(exc) - return found + super().__init__( + self.record_path, node_label, tag, node_label=node_label, tag=tag + ) def add_context( self, node_label: str | None = None, tag: Mapping[str, Any] | None = None @@ -279,10 +328,7 @@ def add_context( node_label: Label of the node that owns the write. tag: Tag values of the row being stored. """ - if self.node_label is None: - self.node_label = node_label - if self.tag is None and tag is not None and not self.flush: - self.tag = dict(tag) + super().add_context(node_label, None if self.flush else tag) def __str__(self) -> str: """Describe the failed write: node, row, table and the backend's reason.""" @@ -309,3 +355,46 @@ def __str__(self) -> str: f"Could not write to the store for {where}{row} " f"(table {'/'.join(self.record_path)!r}){reason}. " + retry ) + + +class StoreReadError(StoreError): + """Raised when a stored table exists but could not be read. + + A table that does not exist reads as empty. A table that exists but cannot be + read (a missing or damaged data file, a log the backend cannot load, columns + whose stored data no longer matches the table schema) raises this error + instead, so stored results never look dropped. Its rows are not recomputed + and the table is not overwritten. + + Attributes: + table: Location of the table that could not be read (a directory or a + cloud URI). + record_path: Path of the table, including the database view's prefix. + node_label: Label of the node whose read failed, or ``None`` when the + read did not happen while a node ran. + tag: Tag values of the row being processed when the read failed, or + ``None`` when the read was not for one row. + """ + + def __init__(self, table: str, record_path: tuple[str, ...]) -> None: + """Create the error for one table that could not be read. + + Args: + table: Location of the table (a directory or a cloud URI). + record_path: Path of the table, including the database view's prefix. + """ + self.table: str = table + self.record_path: tuple[str, ...] = tuple(record_path) + super().__init__(table, self.record_path) + + def __str__(self) -> str: + """Describe the failed read: table, node, row and the backend's reason.""" + node = f" for node {self.node_label!r}" if self.node_label else "" + row = f" for row {self.tag!r}" if self.tag else "" + cause = self.__cause__ + reason = f": {type(cause).__name__}: {cause}" if cause is not None else "" + return ( + f"Could not read the stored table at {self.table!r}{node}{row}{reason}. " + "The table exists, so its rows are not treated as missing and the table " + "was not overwritten." + ) diff --git a/src/orcapod/pipeline/async_orchestrator.py b/src/orcapod/pipeline/async_orchestrator.py index 8020d3d03..87ffed444 100644 --- a/src/orcapod/pipeline/async_orchestrator.py +++ b/src/orcapod/pipeline/async_orchestrator.py @@ -11,10 +11,11 @@ import uuid import logging from collections import defaultdict +from collections.abc import Coroutine from typing import TYPE_CHECKING, Any, Literal from orcapod.channels import BroadcastChannel, Channel -from orcapod.errors import StoreWriteError +from orcapod.errors import StoreError, StoreWriteError from orcapod.pipeline.result import OrchestratorResult from orcapod.utils.store_errors import keep_store_error from orcapod.protocols.node_protocols import ( @@ -140,7 +141,7 @@ async def _run_async( effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) topo_order: list[Any] = [] - store_error: StoreWriteError | None = None + store_error: StoreError | None = None try: topo_order = list(graph.topological_sort()) buf = self._buffer_size @@ -191,8 +192,10 @@ async def _run_async( writer = _CollectingWriter(writer, collector) if is_source_node(node): - tg.create_task( - node.async_execute(writer, observer=effective_observer) + _start_node( + tg, + node, + node.async_execute(writer, observer=effective_observer), ) elif is_function_node(node) or is_side_effect_function_node(node): predecessors = in_edges.get(node, []) @@ -201,13 +204,15 @@ async def _run_async( f"FunctionNode expects exactly 1 upstream, got {len(predecessors)}" ) input_reader = edge_readers[(predecessors[0], node)] - tg.create_task( + _start_node( + tg, + node, node.async_execute( input_reader, writer, observer=effective_observer, run_id=run_id, - ) + ), ) elif is_operator_node(node): predecessors = in_edges.get(node, []) @@ -222,10 +227,12 @@ async def _run_async( input_readers = [ edge_readers[(upstream, node)] for upstream in sorted_preds ] - tg.create_task( + _start_node( + tg, + node, node.async_execute( input_readers, writer, observer=effective_observer - ) + ), ) elif is_side_effect_node(node): predecessors = in_edges.get(node, []) @@ -234,13 +241,15 @@ async def _run_async( "SideEffectNode expects exactly 1 upstream, got 0" ) input_reader = edge_readers[(predecessors[0], node)] - tg.create_task( + _start_node( + tg, + node, node.async_execute( [input_reader], writer, observer=effective_observer, run_id=run_id, - ) + ), ) else: raise TypeError( @@ -255,31 +264,65 @@ async def _run_async( node_outputs=collectors if materialize_results else {} ) except ExceptionGroup as group: - # A failed store write stops the run. Raise it on its own, as the sync - # orchestrator does, rather than inside the TaskGroup's exception group. + # A failed store read or write stops the run. Raise it on its own, as the + # sync orchestrator does, rather than inside the TaskGroup's exception + # group. store_error = _single_store_error(group, topo_order) if store_error is None: raise raise store_error finally: - # A failing on_run_end must not replace a failed store write. + # A failing on_run_end must not replace a failed store read or write. with keep_store_error(store_error, "Observer on_run_end"): effective_observer.on_run_end(run_id) +def _start_node( + tg: asyncio.TaskGroup, node: Any, work: Coroutine[Any, Any, None] +) -> None: + """Start a node's coroutine in ``tg``, naming the node on the store errors it raises. + + A store error raised on its own or inside an exception group (a node that runs + its own task group) gets the node's label when it has none. + + Args: + tg: The run's task group. + node: The node the coroutine runs. + work: The node's ``async_execute`` coroutine. + """ + + async def run() -> None: + try: + await work + except StoreError as exc: + # Name the node the failure belongs to, if the raising site did not, + # so _single_store_error can order it like the sync orchestrator. + exc.add_context(node_label=node.label) + raise + except BaseExceptionGroup as group: + # A node that runs its own task group raises its store errors inside + # a group; name the node on each and re-raise the group unchanged. + for store_error in StoreError.in_group(group): + store_error.add_context(node_label=node.label) + raise + + tg.create_task(run()) + + def _single_store_error( group: ExceptionGroup, topo_order: list[Any] -) -> StoreWriteError | None: - """Pick the one ``StoreWriteError`` an async run raises from its exception group. +) -> StoreError | None: + """Pick the one ``StoreError`` an async run raises from its exception group. Concurrent node tasks can fail together, so the ``TaskGroup``'s group may hold - several store errors and other exceptions. The caller gets a single - ``StoreWriteError``, as from the sync orchestrator: the one whose node comes - first in topological order (the one the sync orchestrator would meet first), - with ties and node-less errors broken by their order in the group. Every other - exception stays attached for diagnosis: each is noted on the chosen error, - other store errors are also logged at WARNING, and the whole group is the - chosen error's ``__context__``. Its ``__cause__`` stays the backend error. + several store errors, read and write ones alike, and other exceptions. The + caller gets a single ``StoreError``, as from the sync orchestrator: the one + whose node comes first in topological order (the one the sync orchestrator + would meet first, when both failures happen), with ties and node-less errors + broken by their order in the group. Every other exception stays attached for + diagnosis: each is noted on the chosen error, other store errors are also + logged at WARNING, and the whole group is the chosen error's ``__context__``. + Its ``__cause__`` stays the backend error. Args: group: The exception group the run's ``TaskGroup`` raised. @@ -288,7 +331,7 @@ def _single_store_error( Returns: The store error to raise, or ``None`` when the group holds none. """ - store_errors = StoreWriteError.in_group(group) + store_errors = StoreError.in_group(group) if not store_errors: return None rank = {getattr(node, "label", None): i for i, node in enumerate(topo_order)} @@ -296,13 +339,15 @@ def _single_store_error( for other in _leaves(group): if other is chosen: continue - if isinstance(other, StoreWriteError): + if isinstance(other, StoreError): + kind = "write" if isinstance(other, StoreWriteError) else "read" logger.warning( - "Another store write failed in the same run: %s", + "Another store %s failed in the same run: %s", + kind, other, exc_info=other, ) - chosen.add_note(f"Another store write failed in the same run: {other}") + chosen.add_note(f"Another store {kind} failed in the same run: {other}") else: chosen.add_note( f"Another node failed in the same run: {type(other).__name__}: {other}" diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index 7fe706f82..b2bb855ba 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -12,7 +12,7 @@ from orcapod.core.nodes.source_node import SourceJobNode from orcapod.side_effects import SideEffectJobNode from orcapod.core.tracker import AutoRegisteringContextBasedTracker -from orcapod.errors import StoreWriteError +from orcapod.errors import StoreReadError, StoreWriteError from orcapod.pipeline.base import AbstractPipelineBase from orcapod.pipeline.dag import OrcaDAG from orcapod.protocols import core_protocols as cp @@ -771,6 +771,10 @@ def run( stops at the first failed write. Writes that succeeded before it are flushed first, so rows stored before the failure are kept, and re-running the job does not recompute stored function results. + StoreReadError: If a stored table exists but could not be read, + whatever the orchestrator's ``error_policy``, also when the read + was needed to write a node's results. Its rows are not recomputed + and the table is not overwritten. """ import hashlib import uuid @@ -880,19 +884,23 @@ def run( _flush_best_effort(run_error, pipeline_db, result_db) raise - # Flush databases. + # Flush databases. Every view of the store shares one set of pending + # batches, so a failed batch can leave the ones after it unwritten. + pending_note = ( + "Writes that were still pending when the flush failed, such as a " + + "cache_mode=LOG operator's output or a node's last records, may not " + + "have been stored." + ) try: pipeline_db.flush() result_db.flush() + except StoreReadError as exc: + # A stored table could not be read: report the read failure itself. + exc.add_note(pending_note) + raise except Exception as exc: flush_error = StoreWriteError(pipeline_db.base_path, flush=True) - # Every view of the store shares one set of pending batches, so a failed - # batch can leave the ones after it unwritten. - flush_error.add_note( - "Writes that were still pending when the flush failed, such as a " - + "cache_mode=LOG operator's output or a node's last records, may not " - + "have been stored." - ) + flush_error.add_note(pending_note) raise flush_error from exc # Mutate self in place. diff --git a/src/orcapod/pipeline/sync_orchestrator.py b/src/orcapod/pipeline/sync_orchestrator.py index 46c2ab1f3..dfe238f5e 100644 --- a/src/orcapod/pipeline/sync_orchestrator.py +++ b/src/orcapod/pipeline/sync_orchestrator.py @@ -10,7 +10,7 @@ import uuid from typing import TYPE_CHECKING, Any, Literal -from orcapod.errors import StoreWriteError +from orcapod.errors import StoreError from orcapod.pipeline.result import OrchestratorResult from orcapod.utils import arrow_utils from orcapod.utils.store_errors import keep_store_error @@ -79,13 +79,15 @@ def run( effective_observer = observer if observer is not None else NoOpObserver() effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) - store_error: StoreWriteError | None = None + store_error: StoreError | None = None + current: Any = None try: topo_order = list(graph.topological_sort()) buffers: dict[Any, list[tuple[TagProtocol, DataProtocol]]] = {} processed: set[Any] = set() for node in topo_order: + current = node if is_source_node(node): buffers[node] = node.execute(observer=effective_observer) elif is_function_node(node) or is_side_effect_function_node(node): @@ -134,7 +136,10 @@ def run( buffers.clear() return OrchestratorResult(node_outputs=buffers) - except StoreWriteError as exc: + except StoreError as exc: + # Name the node the failure belongs to, if the raising site did not. + if current is not None: + exc.add_context(node_label=current.label) store_error = exc raise finally: diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 8b0c0ba16..854eab5ba 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -19,7 +19,7 @@ from orcapod.core.nodes.function_node import _build_record_id_preimage from orcapod.core.streams.base import StreamBase from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER -from orcapod.errors import StoreWriteError +from orcapod.errors import StoreError, StoreReadError, StoreWriteError from orcapod.invocation import ( InvocationContext, InvocationHashConfig, @@ -270,8 +270,8 @@ def _execute_side_effect_row( run_id=run_id, ) return (tag, data) - except StoreWriteError as exc: - # A failed store write fails the run, even under on_error="log". + except StoreError as exc: + # A failed store read or write fails the run, even under on_error="log". exc.add_context(node_label=pod_name, tag=tag.as_dict()) raise except Exception as exc: @@ -326,6 +326,9 @@ def _write_invocation_row( record, skip_duplicates=True, ) + except StoreReadError: + # The stored table could not be read: report the read failure itself. + raise except Exception as exc: raise StoreWriteError(table_path) from exc diff --git a/src/orcapod/utils/store_errors.py b/src/orcapod/utils/store_errors.py index d7cec2df8..a28c1a640 100644 --- a/src/orcapod/utils/store_errors.py +++ b/src/orcapod/utils/store_errors.py @@ -1,6 +1,6 @@ -"""Helpers that keep a ``StoreWriteError`` intact on its way out of a run. +"""Helpers that keep a ``StoreError`` intact on its way out of a run. -A failed store write always fails the run (ITL-878). User callbacks that run while +A failed store read or write always fails the run (ITL-878, ITL-884). User callbacks that run while the error propagates (a post-run hook, an observer callback) may fail too; their failure is logged and noted on the store error, never raised in its place. """ @@ -11,13 +11,13 @@ from collections.abc import Iterator from contextlib import contextmanager -from orcapod.errors import StoreWriteError +from orcapod.errors import StoreError logger = logging.getLogger(__name__) def note_callback_failure( - store_error: StoreWriteError, during: str, failure: Exception + store_error: StoreError, during: str, failure: Exception ) -> None: """Log a callback failure and record it on the propagating store error. @@ -28,7 +28,7 @@ def note_callback_failure( failure: The exception the callback raised. """ logger.warning( - "%s raised while a store write failure was propagating, which is raised " + "%s raised while a store failure was propagating, which is raised " + "instead: %s", during, failure, @@ -41,9 +41,7 @@ def note_callback_failure( @contextmanager -def keep_store_error( - store_error: StoreWriteError | None, during: str -) -> Iterator[None]: +def keep_store_error(store_error: StoreError | None, during: str) -> Iterator[None]: """Run a callback so that it cannot replace a propagating store error. With ``store_error`` set, an ``Exception`` the block raises is logged and noted on diff --git a/superpowers/plans/2026-11-10-itl-884-coderabbit-flush-and-async-group.md b/superpowers/plans/2026-11-10-itl-884-coderabbit-flush-and-async-group.md new file mode 100644 index 000000000..89c7a0f42 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-884-coderabbit-flush-and-async-group.md @@ -0,0 +1,165 @@ +# ITL-884 follow-up: store errors keep their type through `flush()` and out of the async TaskGroup + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan +> task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +> **Superseded in part (November 11, 2026):** the developer's review of PR #290 removed every +> parameter used only for an error message. `_read_delta_table` no longer takes `record_path`, +> `_require_compatible_append(table_uri, ...)` became `_incompatible_append_column(...)`, +> `_store_error_to_raise` takes only the errors, and `_single_store_error` no longer takes +> `topo_order`. See `2026-11-11-itl-884-no-message-only-parameters.md`. + +**Goal:** Answer the two open CodeRabbit threads on PR #290 (ids 4534230764 and 4534230775). +A `StoreReadError` or an append-refusal `InputValidationError` raised while flushing reaches the +caller as itself, so a store read failure in a per-row result-cache flush fails the run under +`error_policy="continue"`. The async orchestrator raises a lone `StoreReadError` even when an +unrelated node failed in the same `TaskGroup`. + +**Architecture:** `DeltaTableDatabase.flush()` keeps attempting every pending batch and logging +each failure. When at least one batch failed with a store error, it raises the first +`StoreReadError` (else the first `InputValidationError`) itself, with a note for every other +failed batch. Only non-store failures keep the `RuntimeError` wrapper. The async orchestrator +replaces `except* StoreReadError` with a plain `except ExceptionGroup` that selects one +`StoreReadError` (`_single_store_error`), notes and logs the rest, and raises it, in the same +shape as PR #287's `_single_store_error` for `StoreWriteError` so the second PR to merge has a +trivial conflict. + +**Tech Stack:** deltalake 1.0.2, pyarrow, asyncio `TaskGroup`, pytest. + +Approved by the coordinator (2026-11-10) on condition that each finding reproduces. Both do: + +- (1b) Run a job, then run it again over two new rows, with the function corrupting the result + table's log (a new log version that requires an unknown reader feature) during its first call. + The per-row lookup still succeeds, the result-cache flush raises `StoreReadError` wrapped in + `RuntimeError`, the sync per-row handler under `continue` skips the row, the next row is + computed, and the run ends with `RuntimeError` from the job's end-of-run flush. +- (2) CI's 3.11 is 3.11.18, so the 3.11.0-3.11.3 single-leaf wrapping does not occur there. The + concurrent case reproduces deterministically on 3.11.18 and 3.13.18: node `dbl` raises + `StoreReadError` in `_load_cached_entries` and node `tri` fails in the observer's + `on_node_start`, both before their first `await`, so both land in one group and `job.run` + raises `ExceptionGroup([StoreReadError, ExceptionGroup([ValueError])])`. + +No hash, path or stored-schema changes. No public API added. + +--- + +## File map + +- `src/orcapod/databases/delta_lake_databases.py`: `flush()` and a module helper + `_store_error_to_raise(errors)`. +- `src/orcapod/pipeline/async_orchestrator.py`: plain `except ExceptionGroup`, helpers + `_single_store_error(group, topo_order)` and `_leaves(group)` (identical to #287's `_leaves`). +- Tests: `tests/test_databases/test_delta_read_failures.py` (assert the bare error types, add a + multi-batch test), `tests/test_pipeline/test_store_read_failure.py` (flush during per-row + processing, both orchestrators; concurrent unrelated failure, async). +- Docs: `docs/concepts/function-pods.md` admonition, `CHANGELOG.md` ITL-884 entries, + `DESIGN_ISSUES.md` DR1 fix note. + +## Task 1: Failing tests for finding (1) + +- [ ] `tests/test_databases/test_delta_read_failures.py`: the `refusal` helper is replaced by + `pytest.raises(InputValidationError)` at each refusal site; the two unloadable-table flush tests + use `pytest.raises(StoreReadError)`; the absent `__cause__` assertions go. +- [ ] New `test_flush_raises_the_store_error_and_notes_every_other_failed_batch`: two tables + with pending batches (one whose stored `x` is int64 and receives float64, one whose log requires + an unknown reader feature); `flush()` raises the `StoreReadError`, one note names the other + table's key and `InputValidationError`, and both batches stay queued (a second `flush()` raises + again). +- [ ] New `test_flush_keeps_runtime_error_for_other_failures`: control case. Monkeypatch + `deltalake.write_deltalake` to raise `OSError` for a new table; `flush()` raises `RuntimeError` + with the `OSError` as `__cause__` (unchanged behaviour). +- [ ] `tests/test_pipeline/test_store_read_failure.py`: + `test_store_read_failure_in_a_row_flush_fails_the_run`, both orchestrators with + `error_policy="continue"`. Rerun over `a..e` with the function corrupting the result table on + its first call; `job.run` raises `StoreReadError` naming the result table, and the function ran + for exactly one new row. +- [ ] Run `uv run pytest tests/test_databases/test_delta_read_failures.py + tests/test_pipeline/test_store_read_failure.py -v`. Expected: the new and changed tests FAIL with + `RuntimeError`, and the per-row test also shows two calls. + +## Task 2: Fix `flush()` + +```python +def flush(self) -> None: + errors: list[tuple[str, Exception]] = [] + for record_key in list(self._pending_batches.keys()): + try: + self._flush_by_key(record_key) + except Exception as e: + logger.error(f"Error flushing batch for {record_key}: {e}") + errors.append((record_key, e)) + if errors: + store_error = _store_error_to_raise(errors) + if store_error is not None: + raise store_error + failed_keys = [k for k, _ in errors] + raise RuntimeError( + f"Failed to flush {len(errors)} batch(es): {failed_keys}" + ) from errors[0][1] + + +def _store_error_to_raise(errors): + """First StoreReadError, else first InputValidationError, with notes; else None.""" + for kind in (StoreReadError, InputValidationError): + chosen = next((e for _, e in errors if isinstance(e, kind)), None) + if chosen is not None: + break + else: + return None + for key, other in errors: + if other is not chosen: + chosen.add_note( + f"Another batch also failed to flush ({key}): " + f"{type(other).__name__}: {other}" + ) + return chosen +``` + +`StoreReadError` is preferred because it is the one the per-row handlers treat as fatal. The +docstring gets a Raises section. Run Task 1's tests: PASS. + +## Task 3: Failing test for finding (2) + +- [ ] `test_concurrent_unrelated_failure_still_raises_store_read_error` (async only): an + observer (a `NoOpObserver` subclass whose `contextualize` returns itself and whose + `on_node_start` raises `ValueError` for `tri`), two function nodes over one source, the `dbl` + result table's data files deleted. `job.run` raises `StoreReadError` (not a group) naming the + result table, `__cause__` is set, and a note names `ValueError: observer failed for tri`. +- [ ] Run it: FAIL with `ExceptionGroup`. + +## Task 4: Fix the async orchestrator + +```python + except ExceptionGroup as group: + # A failed store read stops the run. Raise it on its own, as the sync + # orchestrator does, rather than inside the TaskGroup's exception group. + # A plain except (not except*) so the raise is never wrapped again. + store_error = _single_store_error(group, topo_order) + if store_error is None: + raise + raise store_error +``` + +`_single_store_error(group, topo_order)`: `StoreReadError.in_group(group)`; none gives `None`; the first +is chosen; every other leaf (`_leaves`) is noted on it ("Another store read failed in the same +run: ..." also logged at WARNING; "Another node failed in the same run: : "). Its +`__cause__` stays the backend error; the group is its `__context__`. Run: PASS. + +## Task 5: Docs, design log, verification, commit, replies + +- [ ] Concept page and CHANGELOG: `flush()` / `add_records(..., flush=True)` raise + `StoreReadError` and `InputValidationError` themselves; when several batches fail, the first + store error is raised with a note per other batch, every failed batch stays queued and each + failure is logged; other flush failures still raise `RuntimeError`. +- [ ] `DESIGN_ISSUES.md` DR1: one sentence on the flush and async-group fix. +- [ ] `uv run --with ruff ruff check .`, `ruff format --check .`, basedpyright (exit code), the full + suite on 3.11, 3.12 and 3.13, `tests/test_hashing/golden` unchanged. +- [ ] One commit per finding, push, signed replies on both threads, wait for CI and CodeRabbit. + +## Interaction with PR #287 (report, do not fix here) + +#287's `ResultCache.store` wraps every `add_record`/`flush` failure in `StoreWriteError`, and its +`PipelineJob.run` wraps the end-of-run flush the same way. Once both PRs are in, a +`StoreReadError` from a flush reaches the caller as the `__cause__` of a `StoreWriteError`. It +is still fatal, but `except StoreReadError` misses it. Whichever PR merges second should decide +whether those wrappers let `StoreReadError` through. diff --git a/superpowers/plans/2026-11-10-itl-884-delta-read-failures.md b/superpowers/plans/2026-11-10-itl-884-delta-read-failures.md new file mode 100644 index 000000000..e2c5a9fe5 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-884-delta-read-failures.md @@ -0,0 +1,210 @@ +# ITL-884: Delta read failures are never reported as "no rows" — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan +> task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +> **Superseded in part (November 11, 2026):** the developer's review of PR #290 removed every +> parameter used only for an error message. `_read_delta_table` no longer takes `record_path`, +> `_require_compatible_append(table_uri, ...)` became `_incompatible_append_column(...)`, +> `_store_error_to_raise` takes only the errors, and `_single_store_error` no longer takes +> `topo_order`. See `2026-11-11-itl-884-no-message-only-parameters.md`. + +**Goal:** A Delta table that exists but cannot be read raises `StoreReadError`, never an +empty result. A flush that would make an existing table unreadable, or silently cast the +stored column type, is refused before anything is written. + +**Architecture:** Every read path in `DeltaTableDatabase` separates "table absent" +(`TableNotFoundError` → `None`, unchanged) from "table present but unreadable" (raise +`StoreReadError` naming the table URI, with the backend exception chained). `_flush_by_key` +creates a table only on `TableNotFoundError`. Before it merges into an existing table, it compares +the incoming batch's column types with the stored Delta schema. Both sides go through Delta's +own type mapping, with extension types stripped to storage, large-normalized and +nullability-blind. A difference raises `InputValidationError` naming the table, column and both +types. A store read failure fails the run whatever `error_policy` is (same shape as PR #287's +`StoreWriteError`). + +**Tech Stack:** deltalake 1.0.2, pyarrow, pytest. + +Approved by the coordinator (2026-11-10): contract as proposed. D1 adds a sibling +`StoreReadError(RuntimeError)`, and the append refusal raises `InputValidationError`. D2 is +option A: re-raise in the two per-row handlers and unwrap in the async orchestrator, with the +same shape as #287. D3 makes the comparison nullability-blind, plus a test that pins +nullability-only behaviour. D4 fixes the `DeltaProtocolError` → overwrite path. + +No hash, path or stored-schema changes. + +--- + +## File map + +- `src/orcapod/errors.py` gets `StoreReadError` (`table`, `record_path`, `__str__` from + `__cause__`, `in_group`). +- `src/orcapod/databases/delta_lake_databases.py` changes the read paths and + `_flush_by_key`, and adds `_require_compatible_append` and the module helper + `_comparable_storage_type`. +- `src/orcapod/core/nodes/function_node.py` (`execute` per-row handler) and + `src/orcapod/core/function_pod.py` (async `process_one`) both get + `except StoreReadError: ...; raise`. +- `src/orcapod/pipeline/async_orchestrator.py` catches the `TaskGroup`'s `ExceptionGroup` + with a plain `except`, selects one `StoreReadError` with `_single_store_error`, and raises + it on its own, noting the other failures on it. +- Tests: `tests/test_databases/test_delta_read_failures.py` and + `tests/test_pipeline/test_store_read_failure.py`. +- Docs: `docs/concepts/function-pods.md` gets an admonition, `CHANGELOG.md` gets Fixed and + Breaking entries, and `DESIGN_ISSUES.md` gets a new entry at the end. + +## Task 1: Failing regression tests (database level) + +`tests/test_databases/test_delta_read_failures.py`, public `DeltaTableDatabase` API only: + +- `unreadable_table` fixture: write 2 rows, flush, delete the table's `*.parquet` files (the + log stays intact). Use a fresh `DeltaTableDatabase` instance for the reads. +- `test_get_all_records_raises` checks `StoreReadError`: `exc.table` ends with the table + directory, `str(exc)` contains the URI, and `__cause__` is set. +- `test_get_records_by_ids_raises`, `test_get_record_by_id_raises`, + `test_get_records_with_column_value_raises`, and + `test_add_records_skip_duplicates_raises` (existing-ids refresh). Each raises + `StoreReadError`. +- `test_absent_table_reads_return_none` covers all four reads on a path never written: + `None`, `table_exists` is False. +- `test_unloadable_log_raises_on_read_and_flush_never_overwrites`: the table's log names an + unsupported reader feature (the load fails). Reads raise `StoreReadError`. `add_records(..., + flush=True)` raises, the log files and parquet files are byte-identical afterwards, and no + new log version exists. +- `test_protocol_error_on_load_is_not_treated_as_absent`: monkeypatch + `deltalake.DeltaTable` to raise `DeltaProtocolError` for this table (the error deltalake + 1.0.2 does not raise naturally from a local table). Flush raises and the stored files are + untouched. The comment explains the monkeypatch. +- Append refusal: + - `test_castable_type_change_is_refused` (implemented as the `int64->float64` case of + `test_plain_type_change`): int64 stored, float64 appended. `flush` raises + `InputValidationError` itself (Task 2 of `2026-11-10-itl-884-coderabbit-flush-and-async-group.md`); assert it + directly. + The message names the column `x`, `int64` and `double`. A re-read gives the original row + only, with type int64. + - `test_narrowing_change_is_refused`: int64 stored, int32 appended. + - `test_extension_storage_change_is_refused`: `ext` stored, `ext>` + appended. Refused, and the table is still readable afterwards (1 row). + - Legitimate cases still pass: identical schema; `string` vs `large_string` (and + `list` vs `large_list`); a new column; a missing column; an extension type + with unchanged storage (2 rows read back). + - `test_nullability_only_difference_is_accepted`: a nested non-null struct field vs a + nullable one is accepted. This pins today's nullability-blind behaviour (ITL-842 audit). + It also records what Delta stores for nullability after the merge, as an assertion on + the read-back schema. + +Run `uv run pytest tests/test_databases/test_delta_read_failures.py -v`. Expected: the read +tests FAIL (`None` / no raise / raw errors), and the refusal tests FAIL (the write is +accepted). + +## Task 2: Failing regression tests (pipeline level, both orchestrators) + +`tests/test_pipeline/test_store_read_failure.py`, parametrized over +`SyncPipelineOrchestrator()` and `AsyncPipelineOrchestrator()`: + +- Run `source -> FunctionPod(double) label="dbl"` on a `DeltaTableDatabase`, then delete + the parquet files of the `_result` table. +- `test_unreadable_result_table_fails_the_run`: a fresh store and job, `job.run(orchestrator=...)` + raises `StoreReadError`, and the function is not called. +- `test_unreadable_result_table_fails_the_run_under_continue`: also delete the pipeline + (tag) table, so that Phase 1 sees no tag rows and the per-row lookup is the read that fails. + With `error_policy="continue"` it still raises `StoreReadError` and makes 0 calls. +- `test_absent_tables_still_compute`: control case. Delete both tables entirely, then the + run computes all 3 rows. + +Run the tests and expect FAIL. + +## Task 3: `StoreReadError` + +Add the class to `src/orcapod/errors.py`: + +```python +class StoreReadError(RuntimeError): + """Raised when a stored table exists but could not be read. ...""" + + def __init__(self, table: str, record_path: tuple[str, ...]) -> None: + self.table = table + self.record_path = tuple(record_path) + super().__init__(table, self.record_path) + + def __str__(self) -> str: + cause = self.__cause__ + reason = f": {type(cause).__name__}: {cause}" if cause is not None else "" + return ( + f"Could not read the stored table at {self.table!r}{reason}. The table exists, " + "so its rows are not treated as missing and the table was not overwritten." + ) + + @classmethod + def in_group(cls, group: BaseExceptionGroup) -> "list[StoreReadError]": ... +``` + +## Task 4: Delta read paths + +- `_get_delta_table`: on `TableNotFoundError` return `None`. On any other exception, drop + the cache entry and `raise StoreReadError(table_uri, full_path) from exc`. +- Add the helper `_read_failed(record_path, exc) -> StoreReadError`, built from + `_get_table_uri(record_path)` and `self._path_prefix + record_path`. +- `get_all_records`: `except Exception as exc: raise self._read_failed(...) from exc` + (replaces the warning-and-skip). +- `get_records_by_ids`, `get_record_by_id`, `get_records_with_column_value`: narrow each + `try` to the read itself, then `raise self._read_failed(...) from exc` (replaces + `return None` / the raw re-raise). +- `_refresh_existing_ids_cache`: wrap the `to_pyarrow_table()` failure the same way. + +## Task 5: Delta flush path + +- In `_flush_by_key`, load only on `TableNotFoundError`, where `delta_table = None`. Any + other exception does `raise StoreReadError(table_uri, abs_path_components) from exc`. The + existing outer handler restores the pending batch. +- Before `merge`: `self._require_compatible_append(table_uri, delta_table, combined_table)`. +- `_comparable_storage_type(t)` is a module-level function. It strips extension types to + storage at every depth, rebuilds struct/list/fixed-size-list/map with nullable children + (comparison only, never stored), and recurses. +- `_require_compatible_append` builds the stored types from + `pa.schema(delta_table.schema().to_arrow())`. The incoming types go through + `deltalake.Schema.from_arrow(pa.schema([pa.field(name, _comparable_storage_type(type))]))` + and `.to_arrow()`. Both sides are passed through `normalize_to_large_types(..., + keep_null=True)` and `_comparable_storage_type`. Only columns present on both sides are + compared. On a mismatch it raises `InputValidationError(f"Cannot append to the stored table + {table_uri!r}: column {name!r} is stored as {stored} but the new rows have {incoming}. ...")`. + +Run the Task 1 and Task 2 tests and expect PASS. + +## Task 6: Run-level handling (same shape as #287) + +- `FunctionNode.execute` per-row handler: put `except StoreReadError as exc:` before + `except Exception`. It calls `ctx_obs.on_data_crash(...)` and `ctx_obs.on_node_end(...)`, + then `raise`. +- `FunctionPod` async `process_one`: put `except StoreReadError as exc:` before + `except Exception`. It calls `obs.on_data_crash(...)` and then `raise`. +- `AsyncPipelineOrchestrator._run_async`: add `except ExceptionGroup as group:` and pick + the error with `_single_store_error(group, topo_order)` (the `StoreReadError` of the + earliest node in topological order, the other failures noted on it); raise it on its own, + or re-raise the group unchanged when it holds no `StoreReadError`. A leaf raised inside + `except*` would be wrapped in a new group whenever another task failed concurrently. + +Run the Task 2 tests and expect PASS. + +## Task 7: Docs, changelog, design issues, full suite + +- `docs/concepts/function-pods.md` gets a `!!! note "When the store cannot be read"` + admonition after "Re-runs compute only what is new". It has no python block. +- `CHANGELOG.md`: a Fixed entry (ITL-884: read failures raise `StoreReadError`; an unloadable + table is never overwritten). Breaking Changes: a castable column type change appended to an + existing Delta table is now refused with `InputValidationError` instead of being silently + cast to the stored type. +- `DESIGN_ISSUES.md`: one self-contained entry at the end with status resolved and a Fix note. +- Then run `uv run ruff format .`, `uv run ruff check .`, + `uv run --with basedpyright basedpyright` (exit code 0) and `uv run pytest tests/`. +- Commit in Conventional Commits style: `test(databases): ...` for the failing tests first, + then `fix(databases): ...`, then `docs: ...`. + +## Resume notes (2026-11-11, after machine restart) + +- Arrived to a clean worktree at d32c84d1, in sync with origin; PR #290 had 0 open threads + but conflicted with main after #291 (ITL-847). +- Merged origin/main (b197544e) as b4387c00: only DESIGN_ISSUES.md conflicted; kept SJ1 (main) + then DR1 (this PR) at the end, additions-only against both sides. CHANGELOG auto-merged + (#291 did not touch it). Pushed. +- Next: full suite on 3.11/3.12/3.13, CI green, one CodeRabbit request, worker_done. diff --git a/superpowers/plans/2026-11-11-itl-884-no-message-only-parameters.md b/superpowers/plans/2026-11-11-itl-884-no-message-only-parameters.md new file mode 100644 index 000000000..a0def4030 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-884-no-message-only-parameters.md @@ -0,0 +1,105 @@ +# ITL-884 — remove message-only parameters (developer review of PR #290) + +**Review:** comment 4536812218 on `src/orcapod/databases/delta_lake_databases.py` +(`_read_delta_table`'s new `record_path`): a function must not receive a value used only to +build an error message; the caller, which has the context, wraps the error. + +**Goal:** no helper in the PR diff takes a parameter it uses only for a message. User-visible +behaviour and every `StoreReadError` / `InputValidationError` message stay exactly as they are. + +## Audit of the PR diff (`git diff origin/main...HEAD`) + +| Function | Parameter | Used for | Action | +|---|---|---|---| +| `DeltaTableDatabase._read_delta_table` | `record_path` | message only (`_read_failed`) | Remove; restore main's signature. The function raises a module-private `_DeltaTableReadError` (backend error as `__cause__`) for a failed scan; its three callers catch it and raise `self._read_failed(record_path) from exc.__cause__`. A bad filter still propagates as is. | +| `DeltaTableDatabase._require_compatible_append` | `table_uri` | message only | Becomes `_incompatible_append_column(delta_table, batch)`, returning `(column, stored_type, incoming_type)` or `None`; `flush_batch` (owns `table_uri`) raises the `InputValidationError` with the same message. | +| `_store_error_to_raise` | `record_key` in the `(record_key, error)` pairs | message only (the notes) | Takes `list[Exception]` and only selects; `flush` adds the notes naming the other batches' keys. | +| `async_orchestrator._single_store_error` | `topo_order` | nothing: ranks by `node_label`, which `StoreReadError` lacks, so the first error in the group always wins | Remove the parameter and the ranking (and the `topo_order` pre-init in `run`); select the first store error in group order. Same behaviour. | +| `DeltaTableDatabase._read_failed` | `record_path` | builds the error (its job); called only by owners of `record_path` | Keep. | +| `_get_delta_table`, `_refresh_existing_ids_cache`, `get_all_records` | `record_path` | locating the table (work) | Keep. | +| `flush_batch` | `table_uri`, `abs_path_components` | write location (work) | Keep. | +| `StoreReadError.__init__` | `record_path` | public attribute; not in the message, not read in src/tests | Keep as error payload (coordinator approved). | +| `FunctionJobNode` / `_FunctionPodBase` `StoreReadError` handlers | `node_label`, `pod_label` | observer callbacks (work) | Keep. | + +## Steps + +1. Ask the coordinator to approve the table (repo review process). **Status: approved** (item 8 kept: + payload for a caller that catches the error, same shape as `StoreWriteError` on PR #287). +2. Edit `delta_lake_databases.py`, `async_orchestrator.py`; restore + `tests/test_databases/test_delta_table_database.py:564` to main's call. +3. Tests unchanged in behaviour: `tests/test_databases/test_delta_read_failures.py`, + `tests/test_pipeline/test_store_read_failure.py`, full suite on 3.11–3.13, ruff, + basedpyright. +4. Update the earlier ITL-884 plans' descriptions where they name the removed parameters. +5. One commit, push, reply on thread 4536812218 (do not resolve), CodeRabbit round, CI. + +## Notes for successors + +- PR #293 calls `_read_delta_table`; after this it no longer passes `record_path`. + +## Status + +- Steps 2–4 done: suite 6926 passed / 96 skipped / 15 xfailed on 3.11, 3.12 and 3.13; ruff clean; + basedpyright 0 errors (warnings 1595 → 1587). +- `_single_store_error` here is now `(group) -> StoreReadError | None`, picking + `StoreReadError.in_group(group)[0]`. PR #287's version takes `topo_order` and ranks + `StoreWriteError` by `node_label`; whichever PR merges second reconciles the two. + +## CodeRabbit round on 0369cfcc (thread 4537325251, Major) + +The append check compared whole struct types, so adding a nested struct field (which main accepts +and Delta's merge adds) was refused: a regression from this PR. Fix (coordinator approved): the +module function `_types_conflict(stored, incoming)` matches struct fields by name at every depth, +through list and map types; a field on one side only is not a conflict, like a top-level column. +`_incompatible_append_column` still returns what differs; `flush_batch` raises. New tests: +`test_nested_field_type_change`, `test_new_struct_field`, `test_new_struct_field_inside_a_list` +(stored schema and null back-fill asserted). Suite 6929 passed / 96 skipped / 15 xfailed on 3.11–3.13. + +## Merge of main (#287, StoreWriteError) — approved design D1–D8 (coordinator, November 11, 2026) + +- D1 `StoreError(RuntimeError)` base (public, exported, documented; name may be renamed by the + developer), `in_group` on the base; all handlers type `StoreError`. +- D2 write sites let `StoreReadError` pass through; everything else still wrapped as + `StoreWriteError`. Test: refused append is a `StoreWriteError` (cause `InputValidationError`); + read failure during a write is a `StoreReadError` with its cause. +- D3 `node_label`/`tag` + `add_context` on the base; orchestrators annotate a `StoreError` + escaping a node with its label. Nothing threaded into databases or `ResultCache`. +- D4 `StoreReadError` message names node and row when known. +- D5 `_single_store_error(group, topo_order)` ranks every `StoreError` by node order. +- D6 sync orchestrator catches `StoreError`. +- D7 CHANGELOG / DESIGN_ISSUES / function-pods concept page. +- D8 `tests/test_pipeline/test_store_errors_together.py` (sync/async × both orders, async + order independence). + +## CodeRabbit round on 4f0b4f1c (coordinator approved) + +Accepted: CHANGELOG wording (refused append writes nothing *to that table*; new fields pass the +check, Delta adds them per `allow_schema_evolution`), concept page scoped to `DeltaTableDatabase` +and says a refused append in a run is a `StoreWriteError`; async `record_and_forward` names node +and row on a `StoreError`; `_start_node` labels store errors inside a node's exception group. +Declined: pending-rows read skip (fixed by #293). Deferred: table data without a `_delta_log` +and connector-DB read errors -> ITL-916. + +## CodeRabbit round on cf029bbf (coordinator approved) + +Regression: non-microsecond timestamps (and `list[Path]`, found by the new differential guard) +could not be appended to an existing Delta table. Fix: `_as_delta_writer_stores` applies the +writer's own conversion, `_stored_arrow_schema` reads the stored schema without field metadata. +Guard: `tests/test_databases/test_delta_append_types.py` (every Arrow/logical type, identical +schema, same view and reopened; 40 cases failed before the fix). Deferred: ExceptionGroup-wrapped +store errors -> ITL-918; fixed_size_list merge failure (also on main) -> ITL-916 (strict xfail). + +## CodeRabbit round on ffe44481 + +Regression: rows whose column (or nested field / list item) is `null`-typed could not be appended +to a typed column; `main` accepts them. Fix: `_without_null_types` leaves `null` parts out of the +comparison (a `null` struct field is dropped; a column holding `null` list items or map entries is +not compared). Tests in `test_delta_append_types.py` (three null cases, all passing on `main`; +a `null` field does not hide a changed sibling). +- Round on 51dc9050: `_without_null_types` keeps a list or map whose items still have comparable + fields (a changed sibling of a `null` field inside `list` is refused); the nested-null + tests now build `{"a": None}` / `[None]` and assert the stored value. A `null` field inside + `list` cannot be appended by Delta's merge itself (also on main), so it is not a case + of the append test. CI's "Run Tests" / "Objective" / "PostgreSQL" / license-check workflows fail + on every branch since ~14:25Z: the runners cannot resolve security.ubuntu.com (apt install of + graphviz); "Python — Required CI" passes. diff --git a/tests/test_databases/test_delta_append_types.py b/tests/test_databases/test_delta_append_types.py new file mode 100644 index 000000000..489388596 --- /dev/null +++ b/tests/test_databases/test_delta_append_types.py @@ -0,0 +1,329 @@ +"""Appending rows with the stored schema succeeds for every type Delta can store. + +ITL-884 added a check that refuses an append whose column types differ from the +stored table's. It twice refused appends that ``main`` accepted (a new nested +struct field, then non-microsecond timestamps), so this differential guard writes +one row of each Arrow type, appends a second row with the IDENTICAL schema, and +reads both back: once on the same database view and once after reopening the +store. A type the Delta writer cannot store at all is listed in +``NOT_STORABLE`` and must still fail on the first write, so the list stays true +as ``deltalake`` changes. +""" + +from __future__ import annotations + +import datetime +import decimal +import pathlib +import uuid +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import numpy as np +import pandas as pd +import pyarrow as pa +import pytest +from upath import UPath + +from orcapod.contexts import get_default_context +from orcapod.databases import DeltaTableDatabase +from orcapod.errors import InputValidationError +from orcapod.logical_types.directory_type import Directory +from orcapod.logical_types.file_type import File + +PATH = ("types", "t") + + +@dataclass +class Point: + """A dataclass output type, stored as a logical (extension) type.""" + + x: int + y: float + + +def _converter() -> Any: + """The default data context's type converter.""" + return get_default_context().type_converter + + +def _logical(python_type: Any) -> pa.DataType: + """The Arrow type Orcapod stores ``python_type`` as.""" + converter = _converter() + if python_type is Point: + # Dataclasses are logical types once registered with the converter. + converter.register_python_class(Point) + return converter.python_type_to_arrow_type(python_type) + + +def sample(arrow_type: pa.DataType) -> Any: + """Return one valid Python value of ``arrow_type`` (its storage, for extensions).""" + if isinstance(arrow_type, pa.BaseExtensionType): + return sample(arrow_type.storage_type) + if pa.types.is_boolean(arrow_type): + return True + if pa.types.is_integer(arrow_type): + return 1 + if pa.types.is_floating(arrow_type): + return 1.5 + if pa.types.is_decimal(arrow_type): + return decimal.Decimal("1.5") + if pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type): + return "a" + if pa.types.is_fixed_size_binary(arrow_type): + return b"a" * arrow_type.byte_width + if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): + return b"a" + if pa.types.is_date(arrow_type): + return datetime.date(2020, 1, 2) + if pa.types.is_time(arrow_type): + return datetime.time(1, 2, 3) + if pa.types.is_timestamp(arrow_type): + return datetime.datetime(2020, 1, 2, 3, 4, 5) + if pa.types.is_duration(arrow_type): + return datetime.timedelta(seconds=3) + if pa.types.is_dictionary(arrow_type): + return sample(arrow_type.value_type) + if pa.types.is_fixed_size_list(arrow_type): + return [sample(arrow_type.value_type)] * arrow_type.list_size + if pa.types.is_list(arrow_type) or pa.types.is_large_list(arrow_type): + return [sample(arrow_type.value_type)] + if pa.types.is_map(arrow_type): + return [(sample(arrow_type.key_type), sample(arrow_type.item_type))] + if pa.types.is_struct(arrow_type): + return {f.name: sample(f.type) for f in arrow_type} + raise NotImplementedError(arrow_type) + + +def column(arrow_type: pa.DataType) -> pa.Array[Any]: + """A one-row array of ``arrow_type``.""" + if isinstance(arrow_type, pa.BaseExtensionType): + return pa.ExtensionArray.from_storage( + arrow_type, column(arrow_type.storage_type) + ) + if pa.types.is_dictionary(arrow_type): + return pa.array([sample(arrow_type)], arrow_type.value_type).dictionary_encode() + return pa.array([sample(arrow_type)], arrow_type) + + +PLAIN_TYPES: dict[str, pa.DataType] = { + "int8": pa.int8(), + "int16": pa.int16(), + "int32": pa.int32(), + "int64": pa.int64(), + "uint8": pa.uint8(), + "uint16": pa.uint16(), + "uint32": pa.uint32(), + "uint64": pa.uint64(), + "float16": pa.float16(), + "float32": pa.float32(), + "float64": pa.float64(), + "bool": pa.bool_(), + "string": pa.string(), + "large_string": pa.large_string(), + "binary": pa.binary(), + "large_binary": pa.large_binary(), + "fixed_size_binary": pa.binary(4), + "date32": pa.date32(), + "date64": pa.date64(), + "time32[s]": pa.time32("s"), + "time32[ms]": pa.time32("ms"), + "time64[us]": pa.time64("us"), + "time64[ns]": pa.time64("ns"), + **{f"timestamp[{u}]": pa.timestamp(u) for u in ("s", "ms", "us", "ns")}, + **{f"timestamp[{u}, UTC]": pa.timestamp(u, "UTC") for u in ("s", "ms", "us", "ns")}, + "timestamp[us, Europe/Paris]": pa.timestamp("us", "Europe/Paris"), + **{f"duration[{u}]": pa.duration(u) for u in ("s", "ms", "us", "ns")}, + "decimal128(10, 2)": pa.decimal128(10, 2), + "decimal128(38, 10)": pa.decimal128(38, 10), + "decimal256(40, 2)": pa.decimal256(40, 2), + "list": pa.list_(pa.int64()), + "large_list": pa.large_list(pa.int64()), + "fixed_size_list": pa.list_(pa.int64(), 2), + "list": pa.list_(pa.field("item", pa.int64(), nullable=False)), + "struct": pa.struct([("a", pa.int64()), ("b", pa.string())]), + "struct": pa.struct([pa.field("a", pa.int64(), nullable=False)]), + "map": pa.map_(pa.string(), pa.int64()), + "dictionary": pa.dictionary(pa.int32(), pa.string()), + # Nested combinations, with the units the Delta writer converts. + "struct": pa.struct( + [("t", pa.timestamp("ns")), ("u", pa.uint32())] + ), + "large_list>": pa.large_list( + pa.struct([("t", pa.timestamp("ms", "UTC"))]) + ), + "list>": pa.list_(pa.large_list(pa.uint64())), + "struct>>": pa.struct( + [("l", pa.list_(pa.struct([("a", pa.binary(3))])))] + ), +} + +LOGICAL_TYPES: dict[str, Any] = { + "logical Path": pathlib.Path, + "logical UPath": UPath, + "logical UUID": uuid.UUID, + "logical File": File, + "logical Directory": Directory, + "logical numpy.ndarray": np.ndarray, + "logical pandas.DataFrame": pd.DataFrame, + "logical pandas.Series": pd.Series, + "logical dataclass": Point, + "logical list[Path]": list[pathlib.Path], +} + +# Types the Delta writer cannot store at all (deltalake 1.0.2): the first write +# fails, before the append check is ever reached, on main as on this branch. Kept +# explicit so a deltalake upgrade that starts storing one of them makes the test +# fail and the type join the guarded list. +NOT_STORABLE: set[str] = { + "float16", + "date64", + "time32[s]", + "time32[ms]", + "time64[us]", + "time64[ns]", + "duration[s]", + "duration[ms]", + "duration[us]", + "duration[ns]", + "decimal256(40, 2)", +} + +# Types Delta stores but whose append Delta's own merge rejects, on main as on +# this branch (not the append check): tracked in ITL-916. +MERGE_REJECTS: set[str] = {"fixed_size_list"} + + +def table(arrow_type: pa.DataType, rid: bytes, *, nullable: bool) -> pa.Table: + """A one-row table keyed by ``rid`` with an ``x`` column of ``arrow_type``.""" + schema = pa.schema( + [ + pa.field("rid", pa.large_binary(), nullable=False), + pa.field("x", arrow_type, nullable=nullable), + ] + ) + return pa.Table.from_arrays( + [pa.array([rid], pa.large_binary()), column(arrow_type)], schema=schema + ) + + +def all_types() -> list[Any]: + """Every (id, Arrow type) pair, nullable and non-nullable.""" + types = dict(PLAIN_TYPES) + types.update({name: _logical(t) for name, t in LOGICAL_TYPES.items()}) + merge_rejects = pytest.mark.xfail( + raises=RuntimeError, + strict=True, + reason="Delta's merge rejects this type, also on main (ITL-916)", + ) + return [ + pytest.param( + name, + arrow_type, + nullable, + id=f"{name}-{'nullable' if nullable else 'non-null'}", + marks=[merge_rejects] if name in MERGE_REJECTS else [], + ) + for name, arrow_type in types.items() + for nullable in (True, False) + ] + + +def write(db: DeltaTableDatabase, rows: pa.Table) -> None: + """Add ``rows`` at ``PATH`` and flush them.""" + db.add_records(PATH, rows, record_id_column="rid", flush=True) + + +@pytest.mark.parametrize(("name", "arrow_type", "nullable"), all_types()) +@pytest.mark.parametrize("reopen", [False, True], ids=["same-view", "reopened"]) +def test_identical_schema_appends( + tmp_path: Path, name: str, arrow_type: pa.DataType, nullable: bool, reopen: bool +) -> None: + """A second batch with the stored batch's exact schema appends and reads back.""" + db = DeltaTableDatabase(base_path=tmp_path) + if name in NOT_STORABLE: + # Delta cannot store the type: the first write already fails. + with pytest.raises(RuntimeError): + write(db, table(arrow_type, b"a", nullable=nullable)) + return + write(db, table(arrow_type, b"a", nullable=nullable)) + + if reopen: + db = DeltaTableDatabase(base_path=tmp_path) + write(db, table(arrow_type, b"b", nullable=nullable)) + + back = DeltaTableDatabase(base_path=tmp_path).get_all_records( + PATH, record_id_column="rid" + ) + assert back is not None + assert back.num_rows == 2 + assert set(back.column("rid").to_pylist()) == {b"a", b"b"} + + +@pytest.mark.parametrize( + ("stored_type", "incoming_type", "value"), + [ + (pa.int64(), pa.null(), None), + (pa.struct([("a", pa.int64())]), pa.struct([("a", pa.null())]), {"a": None}), + (pa.large_list(pa.int64()), pa.large_list(pa.null()), [None]), + # A null field inside list is not here: Delta's own merge cannot + # coerce it (also on main), whatever the append check decides. + ], + ids=[ + "int64<-null", + "struct-field<-null", + "list-item<-null", + ], +) +def test_null_typed_values_append_to_a_typed_column( + tmp_path: Path, stored_type: pa.DataType, incoming_type: pa.DataType, value: Any +) -> None: + """Rows whose column (or nested field) is ``null``-typed append to a typed column.""" + db = DeltaTableDatabase(base_path=tmp_path) + write(db, table(stored_type, b"a", nullable=True)) + nulls = pa.Table.from_arrays( + [pa.array([b"b"], pa.large_binary()), pa.array([value], incoming_type)], + names=["rid", "x"], + ) + write(db, nulls) + + back = DeltaTableDatabase(base_path=tmp_path).get_all_records( + PATH, record_id_column="rid" + ) + assert back is not None + assert back.num_rows == 2 + rows = dict(zip(back.column("rid").to_pylist(), back.column("x").to_pylist())) + assert rows[b"b"] == value + + +@pytest.mark.parametrize( + ("stored_type", "incoming_type", "value"), + [ + ( + pa.struct([("a", pa.int64()), ("b", pa.int64())]), + pa.struct([("a", pa.null()), ("b", pa.float64())]), + {"a": None, "b": 2.5}, + ), + ( + pa.large_list(pa.struct([("a", pa.int64()), ("b", pa.int64())])), + pa.large_list(pa.struct([("a", pa.null()), ("b", pa.float64())])), + [{"a": None, "b": 2.5}], + ), + ], + ids=["struct", "list"], +) +def test_null_field_does_not_hide_a_changed_sibling( + tmp_path: Path, stored_type: pa.DataType, incoming_type: pa.DataType, value: Any +) -> None: + """A ``null`` struct field is skipped, but a changed sibling field is refused.""" + db = DeltaTableDatabase(base_path=tmp_path) + write(db, table(stored_type, b"a", nullable=True)) + changed = pa.Table.from_arrays( + [pa.array([b"b"], pa.large_binary()), pa.array([value], incoming_type)], + names=["rid", "x"], + ) + with pytest.raises(InputValidationError, match="'x'") as info: + write(db, changed) + assert "int64" in str(info.value) + assert "double" in str(info.value) diff --git a/tests/test_databases/test_delta_read_failures.py b/tests/test_databases/test_delta_read_failures.py new file mode 100644 index 000000000..e40ca3514 --- /dev/null +++ b/tests/test_databases/test_delta_read_failures.py @@ -0,0 +1,498 @@ +"""Regression tests for ITL-884: a Delta table that cannot be read is never "no rows". + +Covers: +- A table that exists but cannot be read raises ``StoreReadError`` on every read + path, naming the table; a table that does not exist still reads as ``None``. +- A table whose log cannot be loaded is never overwritten by a flush. +- ``flush`` raises ``StoreReadError`` and the append refusal's + ``InputValidationError`` themselves, not wrapped in ``RuntimeError``, and + notes every other batch that failed in the same flush. +- An append whose column types differ from the stored table is refused before + anything is written, naming the column and both types; legitimate appends + (same types in another layout, new or missing columns, unchanged extension + storage) still succeed. +""" + +from __future__ import annotations + +import datetime +import json +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import deltalake +import pyarrow as pa +import pytest +from deltalake.exceptions import DeltaProtocolError + +from orcapod.databases import DeltaTableDatabase +from orcapod.errors import InputValidationError, StoreReadError + +PATH = ("results", "t") + + +def rows(ids: list[bytes], x: pa.Array[Any], **extra: pa.Array[Any]) -> pa.Table: + """Build a table keyed by ``rid`` with an ``x`` column and optional extras.""" + return pa.table({"rid": pa.array(ids, pa.large_binary()), "x": x, **extra}) + + +def table_dir(base: Path) -> Path: + """Return the directory of the table at ``PATH`` under ``base``.""" + return base.joinpath(*PATH) + + +def snapshot(directory: Path) -> dict[str, bytes]: + """Every file under ``directory`` with its bytes, keyed by relative path.""" + return { + str(p.relative_to(directory)): p.read_bytes() + for p in sorted(directory.rglob("*")) + if p.is_file() + } + + +class _Ext(pa.ExtensionType): + """A minimal extension type standing in for a logical type's storage.""" + + def __init__(self, storage: pa.DataType) -> None: + """Wrap ``storage`` in the test extension type.""" + super().__init__(storage, "orcapod.test.itl884") + + def __arrow_ext_serialize__(self) -> bytes: + """Return the (empty) serialized parameters of the type.""" + return b"" + + @classmethod + def __arrow_ext_deserialize__( + cls, storage_type: pa.DataType, serialized: bytes + ) -> "_Ext": + """Rebuild the type over ``storage_type``.""" + return cls(storage_type) + + +def ext_array(values: list[Any], storage: pa.DataType) -> pa.ExtensionArray[Any]: + """Build an extension array of ``values`` over the given storage type.""" + return pa.ExtensionArray.from_storage(_Ext(storage), pa.array(values, storage)) + + +@pytest.fixture +def base(tmp_path: Path) -> Path: + """Return the store root, inside the test's temporary directory.""" + return tmp_path / "store" + + +@pytest.fixture +def stored(base: Path) -> DeltaTableDatabase: + """A store holding two rows (``a``, ``b``) of ``x: int64`` at ``PATH``.""" + db = DeltaTableDatabase(base_path=base) + db.add_records( + PATH, + rows([b"a", b"b"], pa.array([1, 2], pa.int64())), + record_id_column="rid", + flush=True, + ) + return db + + +@pytest.fixture +def unreadable(base: Path, stored: DeltaTableDatabase) -> DeltaTableDatabase: + """A fresh view of a table whose log is intact but whose data files are gone.""" + data_files = list(table_dir(base).glob("*.parquet")) + assert data_files + for f in data_files: + f.unlink() + return DeltaTableDatabase(base_path=base) + + +def unsupported_reader_feature(base: Path) -> None: + """Make the table's log require a reader feature deltalake does not know.""" + log = table_dir(base) / "_delta_log" / "00000000000000000000.json" + actions = [json.loads(line) for line in log.read_text().splitlines()] + for action in actions: + if "protocol" in action: + action["protocol"] = { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": ["orcapodMadeUpFeature"], + "writerFeatures": ["orcapodMadeUpFeature"], + } + log.write_text("\n".join(json.dumps(a) for a in actions)) + + +def assert_names_table(exc: StoreReadError, base: Path) -> None: + """Assert that ``exc`` names the table at ``PATH`` and chains its cause.""" + assert Path(exc.table) == table_dir(base) + assert exc.table in str(exc) + assert exc.__cause__ is not None + + +# --------------------------------------------------------------------------- +# Reads: unreadable table raises, absent table is None +# --------------------------------------------------------------------------- + +READS: dict[str, Callable[[DeltaTableDatabase], object]] = { + "get_all_records": lambda db: db.get_all_records(PATH), + "get_records_by_ids": lambda db: db.get_records_by_ids(PATH, [b"a"]), + "get_record_by_id": lambda db: db.get_record_by_id(PATH, b"a"), + "get_records_with_column_value": lambda db: db.get_records_with_column_value( + PATH, {"x": 1} + ), +} + + +class TestUnreadableTable: + """A table that exists but cannot be read raises on every read path.""" + + @pytest.mark.parametrize("read", READS.values(), ids=list(READS)) + def test_read_raises_store_read_error(self, base, unreadable, read): + """Each read of the unreadable table raises ``StoreReadError`` naming it.""" + with pytest.raises(StoreReadError) as info: + read(unreadable) + assert_names_table(info.value, base) + + def test_skip_duplicates_append_raises_store_read_error(self, base, unreadable): + """The existing-ids read for ``skip_duplicates`` raises, never assumes none.""" + # skip_duplicates reads the stored record ids before buffering the rows. + with pytest.raises(StoreReadError) as info: + unreadable.add_records( + PATH, + rows([b"c"], pa.array([3], pa.int64())), + record_id_column="rid", + skip_duplicates=True, + ) + assert_names_table(info.value, base) + + def test_table_still_exists(self, unreadable): + """The unreadable table is still reported as existing.""" + assert unreadable.table_exists(PATH) + + +def test_bad_filter_is_not_reported_as_unreadable_table(stored): + """A bad filter column is the caller's error, not a ``StoreReadError``.""" + # The table reads fine; the filter names a column it does not have. + with pytest.raises(pa.ArrowInvalid): + stored.get_records_with_column_value(PATH, {"no_such_column": 1}) + + +class TestAbsentTable: + """A table that does not exist still reads as ``None``.""" + + @pytest.mark.parametrize("read", READS.values(), ids=list(READS)) + def test_read_returns_none(self, base, read): + """Each read of a table that was never written returns ``None``.""" + db = DeltaTableDatabase(base_path=base) + assert read(db) is None + assert not db.table_exists(PATH) + + +# --------------------------------------------------------------------------- +# Flush never overwrites a table it could not load +# --------------------------------------------------------------------------- + + +class TestUnloadableTableIsNeverOverwritten: + """A table whose log cannot be loaded is never replaced by a flush.""" + + def test_unsupported_reader_feature(self, base, stored): + """An unloadable log makes reads and the flush raise; nothing is written.""" + unsupported_reader_feature(base) + before = snapshot(table_dir(base)) + db = DeltaTableDatabase(base_path=base) + + with pytest.raises(StoreReadError): + db.get_all_records(PATH) + with pytest.raises(StoreReadError) as info: + db.add_records( + PATH, + rows([b"c"], pa.array([3], pa.int64())), + record_id_column="rid", + flush=True, + ) + assert_names_table(info.value, base) + assert snapshot(table_dir(base)) == before + + def test_protocol_error_on_load_is_not_treated_as_absent( + self, base, stored, monkeypatch + ): + """A ``DeltaProtocolError`` on load makes the flush raise, not overwrite.""" + # deltalake 1.0.2 does not raise DeltaProtocolError for a local table we can + # build here, so the load is replaced by one that raises it for this table. + # Before ITL-884 the flush treated this error as "no table" and wrote the + # batch with mode="overwrite", replacing the stored rows. + real = deltalake.DeltaTable + target = table_dir(base) + + def failing_load(table_uri, *args, **kwargs): + """Raise ``DeltaProtocolError`` for the target table only.""" + if Path(str(table_uri)) == target: + raise DeltaProtocolError("simulated") + return real(table_uri, *args, **kwargs) + + before = snapshot(target) + db = DeltaTableDatabase(base_path=base) + monkeypatch.setattr(deltalake, "DeltaTable", failing_load) + with pytest.raises(StoreReadError) as info: + db.add_records( + PATH, + rows([b"c"], pa.array([3], pa.int64())), + record_id_column="rid", + flush=True, + ) + monkeypatch.undo() + + assert isinstance(info.value.__cause__, DeltaProtocolError) + assert snapshot(target) == before + stored_rows = DeltaTableDatabase(base_path=base).get_all_records(PATH) + assert stored_rows is not None + assert stored_rows.num_rows == 2 + + +class TestFlushRaisesStoreErrorsThemselves: + """``flush`` raises store errors as themselves, with the other failures noted.""" + + def test_store_read_error_is_raised_with_a_note_per_other_failed_batch( + self, base, stored + ): + """The ``StoreReadError`` is raised and the refused batch is noted on it.""" + other = ("results", "u") + stored.add_records( + other, + rows([b"a"], pa.array([1], pa.int64())), + record_id_column="rid", + flush=True, + ) + unsupported_reader_feature(base) + db = DeltaTableDatabase(base_path=base) + # Queued first: refused at flush, because x is stored as int64. + db.add_records(other, rows([b"b"], pa.array([2.0])), record_id_column="rid") + # Queued second: its table's log cannot be loaded. + db.add_records( + PATH, rows([b"c"], pa.array([3], pa.int64())), record_id_column="rid" + ) + + with pytest.raises(StoreReadError) as info: + db.flush() + + # The StoreReadError wins over the earlier refusal, because it is the one + # that fails a run whatever the error policy; the refusal is noted on it. + assert_names_table(info.value, base) + notes = getattr(info.value, "__notes__", []) + assert len(notes) == 1 + assert "results/u" in notes[0] + assert "InputValidationError" in notes[0] + # Both failed batches stay queued, so the next flush fails the same way. + with pytest.raises(StoreReadError): + db.flush() + + def test_other_flush_failures_still_raise_runtime_error(self, base, monkeypatch): + """A flush that fails for another reason still raises ``RuntimeError``.""" + + def failing_write(*args, **kwargs): + """Raise ``OSError`` for every write.""" + raise OSError("simulated disk failure") + + db = DeltaTableDatabase(base_path=base) + db.add_records( + PATH, rows([b"a"], pa.array([1], pa.int64())), record_id_column="rid" + ) + monkeypatch.setattr(deltalake, "write_deltalake", failing_write) + + with pytest.raises(RuntimeError, match="Failed to flush 1 batch") as info: + db.flush() + assert isinstance(info.value.__cause__, OSError) + + +# --------------------------------------------------------------------------- +# Append refusal: a column's type differs from the stored table +# --------------------------------------------------------------------------- + + +def append(db: DeltaTableDatabase, table: pa.Table) -> None: + """Add ``table`` at ``PATH`` and flush it.""" + db.add_records(PATH, table, record_id_column="rid", flush=True) + + +class TestChangedColumnTypeIsRefused: + """An append whose column type differs from the stored table is refused.""" + + @pytest.mark.parametrize( + ("new_type", "value", "named"), + [ + (pa.float64(), 2.0, "double"), + (pa.int32(), 2, "int32"), + (pa.large_string(), "2", "string"), + (pa.timestamp("ns"), datetime.datetime(2020, 1, 2), "timestamp"), + ], + ids=[ + "int64->float64", + "int64->int32", + "int64->large_string", + "int64->timestamp[ns]", + ], + ) + def test_plain_type_change(self, base, stored, new_type, value, named): + """A changed column type is refused, naming the column and both types.""" + with pytest.raises(InputValidationError) as info: + append(stored, rows([b"c"], pa.array([value], new_type))) + message = str(info.value) + assert "'x'" in message + assert "int64" in message + assert named in message + assert str(table_dir(base)) in message + + back = DeltaTableDatabase(base_path=base).get_all_records(PATH) + assert back is not None + assert back.num_rows == 2 + assert back.schema.field("x").type == pa.int64() + + def test_extension_storage_change(self, base): + """A changed extension storage type is refused and the table stays readable.""" + db = DeltaTableDatabase(base_path=base) + append(db, rows([b"a"], ext_array([b"x"], pa.large_binary()))) + + with pytest.raises(InputValidationError) as info: + append( + db, + rows([b"b"], ext_array([[1.0]], pa.large_list(pa.float32()))), + ) + message = str(info.value) + assert "'x'" in message + assert "binary" in message + assert "list" in message + + back = DeltaTableDatabase(base_path=base).get_all_records(PATH) + assert back is not None + assert back.num_rows == 1 + + def test_nested_field_type_change(self, base: Path) -> None: + """A changed type of an existing struct field is refused.""" + db = DeltaTableDatabase(base_path=base) + append(db, rows([b"a"], pa.array([{"a": 1}], pa.struct([("a", pa.int64())])))) + + with pytest.raises(InputValidationError) as info: + append( + db, + rows([b"b"], pa.array([{"a": 2.0}], pa.struct([("a", pa.float64())]))), + ) + message = str(info.value) + assert "'x'" in message + assert "struct" in message + assert "struct" in message + + back = DeltaTableDatabase(base_path=base).get_all_records(PATH) + assert back is not None + assert back.num_rows == 1 + assert back.schema.field("x").type == pa.struct([("a", pa.int64())]) + + +class TestLegitimateAppendsStillPass: + """Appends whose types match the stored table, as Delta stores them, succeed.""" + + def read_back(self, base: Path) -> pa.Table: + """Read every row at ``PATH`` with a fresh database view.""" + back = DeltaTableDatabase(base_path=base).get_all_records(PATH) + assert back is not None + return back + + def test_identical_schema(self, base, stored): + """An append with the stored schema succeeds.""" + append(stored, rows([b"c"], pa.array([3], pa.int64()))) + assert self.read_back(base).num_rows == 3 + + @pytest.mark.parametrize( + ("first", "second"), + [ + (pa.array(["a"], pa.string()), pa.array(["b"], pa.large_string())), + (pa.array(["a"], pa.large_string()), pa.array(["b"], pa.string())), + ( + pa.array([[1]], pa.list_(pa.int64())), + pa.array([[2]], pa.large_list(pa.int64())), + ), + (pa.array([b"a"], pa.binary()), pa.array([b"b"], pa.large_binary())), + ], + ids=["string->large", "large->string", "list->large", "binary->large"], + ) + def test_large_and_non_large_layouts(self, base, first, second): + """Large and non-large layouts of one type count as the same type.""" + db = DeltaTableDatabase(base_path=base) + append(db, rows([b"a"], first)) + append(db, rows([b"b"], second)) + assert self.read_back(base).num_rows == 2 + + def test_new_column(self, base, stored): + """An append that adds a column succeeds.""" + append( + stored, + rows([b"c"], pa.array([3], pa.int64()), y=pa.array(["new"])), + ) + back = self.read_back(base) + assert back.num_rows == 3 + assert "y" in back.column_names + + def test_new_struct_field(self, base: Path) -> None: + """An append that adds a field to a stored struct column succeeds.""" + db = DeltaTableDatabase(base_path=base) + append(db, rows([b"a"], pa.array([{"a": 1}], pa.struct([("a", pa.int64())])))) + evolved = pa.struct([("a", pa.int64()), ("b", pa.int64())]) + append(db, rows([b"b"], pa.array([{"a": 2, "b": 3}], evolved))) + + back = DeltaTableDatabase(base_path=base).get_all_records( + PATH, record_id_column="rid" + ) + assert back is not None + back = back.sort_by("rid") + assert back.schema.field("x").type == evolved + assert back.column("x").to_pylist() == [ + {"a": 1, "b": None}, + {"a": 2, "b": 3}, + ] + + def test_new_struct_field_inside_a_list(self, base: Path) -> None: + """An append that adds a field to a struct inside a list column succeeds.""" + db = DeltaTableDatabase(base_path=base) + stored_type = pa.large_list(pa.struct([("a", pa.int64())])) + append(db, rows([b"a"], pa.array([[{"a": 1}]], stored_type))) + evolved = pa.large_list(pa.struct([("a", pa.int64()), ("b", pa.int64())])) + append(db, rows([b"b"], pa.array([[{"a": 2, "b": 3}]], evolved))) + + back = DeltaTableDatabase(base_path=base).get_all_records( + PATH, record_id_column="rid" + ) + assert back is not None + back = back.sort_by("rid") + assert back.schema.field("x").type == evolved + assert back.column("x").to_pylist() == [ + [{"a": 1, "b": None}], + [{"a": 2, "b": 3}], + ] + + def test_missing_column(self, base): + """An append that omits a stored column succeeds.""" + db = DeltaTableDatabase(base_path=base) + append(db, rows([b"a"], pa.array([1], pa.int64()), y=pa.array(["old"]))) + append(db, rows([b"b"], pa.array([2], pa.int64()))) + assert self.read_back(base).num_rows == 2 + + def test_extension_type_with_unchanged_storage(self, base): + """An extension type with unchanged storage appends.""" + db = DeltaTableDatabase(base_path=base) + append(db, rows([b"a"], ext_array([b"x"], pa.large_binary()))) + append(db, rows([b"b"], ext_array([b"y"], pa.large_binary()))) + assert self.read_back(base).num_rows == 2 + + def test_nullability_only_difference_is_accepted(self, base): + """A nullability-only difference is accepted (pinned, ITL-842).""" + # Pins today's behaviour: the append check ignores nullability (ITL-884, + # decision D3). Whether a nullability change should be refused belongs to + # the nullability audit, ITL-842. The read-back assertion records what + # Delta does on merge: the stored non-null child becomes nullable, so the + # rows written first now read back with a different (looser) type. + strict = pa.struct([pa.field("a", pa.int64(), nullable=False)]) + loose = pa.struct([pa.field("a", pa.int64(), nullable=True)]) + db = DeltaTableDatabase(base_path=base) + append(db, rows([b"a"], pa.array([{"a": 1}], strict))) + append(db, rows([b"b"], pa.array([{"a": 2}], loose))) + + back = self.read_back(base) + assert back.num_rows == 2 + assert back.schema.field("x").type == loose diff --git a/tests/test_pipeline/test_store_errors_together.py b/tests/test_pipeline/test_store_errors_together.py new file mode 100644 index 000000000..9b9863eb0 --- /dev/null +++ b/tests/test_pipeline/test_store_errors_together.py @@ -0,0 +1,333 @@ +"""Store read and write failures in one run (ITL-884 with ITL-878). + +``StoreReadError`` and ``StoreWriteError`` share the ``StoreError`` base and one +error path: whichever fails first in node order fails the run, the same on the +sync and the async orchestrator, and the error names its node. A read failure +met while writing is reported as the read failure; any other failed write is a +``StoreWriteError``. +""" + +from __future__ import annotations + +import asyncio +import json +from collections.abc import Callable, Iterator +from pathlib import Path +from typing import Any + +import deltalake +import pyarrow as pa +import pytest +from deltalake.exceptions import DeltaError + +from orcapod import function_pod +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase +from orcapod.errors import ( + InputValidationError, + StoreError, + StoreReadError, + StoreWriteError, +) +from orcapod.pipeline import PipelineJob +from orcapod.pipeline.async_orchestrator import ( + AsyncPipelineOrchestrator, + _single_store_error, # pyright: ignore[reportPrivateUsage] + _start_node, # pyright: ignore[reportPrivateUsage] +) +from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator + +# Tables the read-failing function damages on its next call, standing in for +# another process breaking a table while the job runs. +_DAMAGE_ON_NEXT_CALL: list[Path] = [] + + +def require_unknown_reader_feature(table: Path) -> None: + """Commit a log version that requires a reader feature deltalake does not know.""" + log = table / "_delta_log" + version = max(int(p.stem) for p in log.glob("*.json")) + 1 + protocol = { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": ["orcapodMadeUpFeature"], + "writerFeatures": ["orcapodMadeUpFeature"], + } + (log / f"{version:020d}.json").write_text(json.dumps({"protocol": protocol})) + + +@function_pod(output_keys="y") +async def double(x: int) -> int: + """Damage any queued table, then return twice ``x`` (on the event loop).""" + while _DAMAGE_ON_NEXT_CALL: + require_unknown_reader_feature(_DAMAGE_ON_NEXT_CALL.pop()) + return 2 * x + + +@function_pod(output_keys="z") +async def triple(x: int) -> int: + """Return three times ``x`` (on the event loop).""" + return 3 * x + + +ORCHESTRATORS: dict[str, Callable[[], Any]] = { + "sync": lambda: SyncPipelineOrchestrator(error_policy="continue"), + "async": lambda: AsyncPipelineOrchestrator(error_policy="continue"), +} + + +@pytest.fixture(autouse=True) +def _reset_damage() -> Iterator[None]: + """Start and end every test with no queued damage.""" + _DAMAGE_ON_NEXT_CALL.clear() + yield + _DAMAGE_ON_NEXT_CALL.clear() + + +def build_job(store_path: Path, keys: str, *, reader_first: bool) -> PipelineJob: + """Two sibling nodes over one source: ``rd`` (doubles) and ``wr`` (triples). + + ``reader_first`` decides which node is added first, and so comes first in + node order. + """ + source = ArrowTableSource( + pa.table({"k": list(keys), "x": list(range(1, len(keys) + 1))}), + tag_columns=["k"], + infer_nullable=True, + ) + job = PipelineJob(name="together", store=DeltaTableDatabase(base_path=store_path)) + with job: + if reader_first: + _ = double.pod(source, label="rd") + _ = triple.pod(source, label="wr") + else: + _ = triple.pod(source, label="wr") + _ = double.pod(source, label="rd") + return job + + +def result_table(store_path: Path, function_name: str) -> Path: + """The directory of ``function_name``'s result table.""" + dirs = { + p.parent + for p in (store_path / "together" / "_result" / function_name).rglob( + "*.parquet" + ) + } + assert len(dirs) == 1, sorted(dirs) + return dirs.pop() + + +def fail_result_writes_of( + monkeypatch: pytest.MonkeyPatch, function_name: str +) -> list[tuple[str, ...]]: + """Make every result-table write of ``function_name`` fail with ``OSError``. + + Returns: + The record paths of the writes that were failed. + """ + failed: list[tuple[str, ...]] = [] + original = DeltaTableDatabase.add_record + + def add_record(self: DeltaTableDatabase, record_path, *args, **kwargs): + if "_result" in self.base_path and record_path[0] == function_name: + failed.append(tuple(record_path)) + raise OSError(f"disk unavailable while writing {'/'.join(record_path)}") + return original(self, record_path, *args, **kwargs) + + monkeypatch.setattr(DeltaTableDatabase, "add_record", add_record) + return failed + + +def run_with_both_failures( + tmp_path: Path, + make_orchestrator: Callable[[], Any], + *, + reader_first: bool, +) -> StoreError: + """Run a job in which ``rd``'s read and ``wr``'s write fail on the new row d. + + The first run stores rows a, b and c for both nodes. In the second run row d + is new: ``rd``'s result table breaks while d is computed, so storing d's + result fails to read the table, and ``wr``'s write of d's result fails. + + Returns: + The store error the second run raised. + """ + store_path = tmp_path / "store" + build_job(store_path, "abc", reader_first=reader_first).run( + orchestrator=make_orchestrator() + ) + + _DAMAGE_ON_NEXT_CALL.append(result_table(store_path, "double")) + with pytest.MonkeyPatch.context() as monkeypatch: + failed = fail_result_writes_of(monkeypatch, "triple") + job = build_job(store_path, "abcd", reader_first=reader_first) + with pytest.raises(StoreError) as info: + job.run(orchestrator=make_orchestrator()) + if info.value.node_label == "wr": + assert failed, "wr's error should come from its failed write" + return info.value + + +@pytest.mark.parametrize("reader_first", [True, False], ids=["rd-first", "wr-first"]) +def test_sync_and_async_raise_the_same_error( + tmp_path: Path, reader_first: bool +) -> None: + """Both orchestrators raise the error of the node that comes first.""" + raised = { + name: run_with_both_failures(tmp_path / name, make, reader_first=reader_first) + for name, make in ORCHESTRATORS.items() + } + + expected_type, expected_node = ( + (StoreReadError, "rd") if reader_first else (StoreWriteError, "wr") + ) + for name, err in raised.items(): + assert type(err) is expected_type, (name, err) + assert err.node_label == expected_node, (name, err) + assert err.tag == {"k": "d"}, (name, err) + sync, async_ = raised["sync"], raised["async"] + assert str(sync).replace(str(tmp_path / "sync"), "") == str(async_).replace( + str(tmp_path / "async"), "" + ) + + +@pytest.mark.parametrize("reader_first", [True, False], ids=["rd-first", "wr-first"]) +def test_async_choice_does_not_depend_on_completion_order( + tmp_path: Path, reader_first: bool +) -> None: + """Repeated async runs always raise the first node's error, noting the other.""" + for attempt in range(5): + err = run_with_both_failures( + tmp_path / str(attempt), + ORCHESTRATORS["async"], + reader_first=reader_first, + ) + assert err.node_label == ("rd" if reader_first else "wr"), attempt + notes = "\n".join(getattr(err, "__notes__", [])) + other = "write" if reader_first else "read" + assert f"Another store {other} failed in the same run" in notes, attempt + + +def test_selection_is_independent_of_group_order() -> None: + """``_single_store_error`` ranks by node order, whatever order the group has.""" + # Calls the private helper directly: the test is about its ordering rule, + # which an end-to-end run cannot force into both group orders. + + class Node: + def __init__(self, label: str) -> None: + self.label = label + + topo_order = [Node("first"), Node("second")] + for build in (lambda r, w: [r, w], lambda r, w: [w, r]): + read = StoreReadError("table", ("t",)) + read.add_context(node_label="second") + write = StoreWriteError(("t",), node_label="first") + chosen = _single_store_error( + ExceptionGroup("run", build(read, write)), topo_order + ) + assert chosen is write + + +def test_read_failure_while_writing_is_a_store_read_error(tmp_path: Path) -> None: + """A write that cannot read its stored table raises ``StoreReadError``.""" + store_path = tmp_path / "store" + build_job(store_path, "abc", reader_first=True).run( + orchestrator=ORCHESTRATORS["sync"]() + ) + table = result_table(store_path, "double") + _DAMAGE_ON_NEXT_CALL.append(table) + + with pytest.raises(StoreReadError) as info: + build_job(store_path, "abcd", reader_first=True).run( + orchestrator=ORCHESTRATORS["sync"]() + ) + + err = info.value + assert Path(err.table) == table + assert err.node_label == "rd" + assert isinstance(err.__cause__, DeltaError) + assert "node 'rd'" in str(err) + assert "row {'k': 'd'}" in str(err) + + +def test_refused_append_is_a_store_write_error(tmp_path: Path) -> None: + """A refused append (a changed column type) fails the run as a ``StoreWriteError``.""" + store_path = tmp_path / "store" + build_job(store_path, "abc", reader_first=True).run( + orchestrator=ORCHESTRATORS["sync"]() + ) + # Store the result column as int32, so the new row's int64 result differs. + table = result_table(store_path, "double") + stored = deltalake.DeltaTable(str(table)).to_pyarrow_table() + index = stored.schema.get_field_index("y") + narrowed = stored.set_column(index, "y", stored.column("y").cast(pa.int32())) + deltalake.write_deltalake( + str(table), narrowed, mode="overwrite", schema_mode="overwrite" + ) + + with pytest.raises(StoreWriteError) as info: + build_job(store_path, "abcd", reader_first=True).run( + orchestrator=ORCHESTRATORS["sync"]() + ) + + err = info.value + assert err.node_label == "rd" + assert isinstance(err.__cause__, InputValidationError) + assert "'y'" in str(err.__cause__) + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_pipeline_record_read_failure_names_node_and_row( + tmp_path: Path, orchestrator: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """A read failure while recording a row's pipeline entry names the node and row.""" + # The async node records pipeline entries outside the pod's per-row handler, + # so this checks that the error still gets its node and row. + store_path = tmp_path / "store" + build_job(store_path, "abc", reader_first=True).run( + orchestrator=ORCHESTRATORS[orchestrator]() + ) + original = DeltaTableDatabase.add_record + + def add_record(self: DeltaTableDatabase, record_path, record_id, record, **kwargs): + # Stand in for a pipeline table that cannot be read while row d's entry + # is recorded (the skip_duplicates check reads the stored record ids). + if "_result" not in self.base_path and record.column("k")[0].as_py() == "d": + raise StoreReadError("pipeline-table", tuple(record_path)) from OSError( + "data file missing" + ) + return original(self, record_path, record_id, record, **kwargs) + + monkeypatch.setattr(DeltaTableDatabase, "add_record", add_record) + with pytest.raises(StoreReadError) as info: + build_job(store_path, "abcd", reader_first=True).run( + orchestrator=ORCHESTRATORS[orchestrator]() + ) + + err = info.value + assert err.table == "pipeline-table" + assert err.node_label == "rd" + assert err.tag == {"k": "d"} + + +def test_store_error_inside_a_node_group_gets_the_node_label() -> None: + """A store error a node raises inside its own exception group is labelled.""" + # Calls the private helper directly: no public pipeline raises a store error + # from a node's inner task group without an annotation on the way. + + class Node: + label = "inner" + + read = StoreReadError("table", ("t",)) + + async def node_work() -> None: + raise ExceptionGroup("node", [read]) + + async def main() -> None: + async with asyncio.TaskGroup() as tg: + _start_node(tg, Node(), node_work()) + + with pytest.raises(ExceptionGroup): + asyncio.run(main()) + assert read.node_label == "inner" diff --git a/tests/test_pipeline/test_store_read_failure.py b/tests/test_pipeline/test_store_read_failure.py new file mode 100644 index 000000000..6a3c52c8d --- /dev/null +++ b/tests/test_pipeline/test_store_read_failure.py @@ -0,0 +1,248 @@ +"""Regression tests for ITL-884 at the job level: an unreadable store fails the run. + +Before ITL-884 a Delta result table that existed but could not be read was +returned as "no rows": the job warned that the data "may have been deleted +externally", swallowed the per-row read errors under ``error_policy="continue"``, +finished without raising, and the node showed no records. Both orchestrators +must now raise ``StoreReadError`` instead, and a table that does not exist must +still be computed. +""" + +from __future__ import annotations + +import json +import shutil +from collections.abc import Callable +from pathlib import Path + +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase +from orcapod.errors import StoreReadError +from orcapod.pipeline import PipelineJob +from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator +from orcapod.pipeline.observer import NoOpObserver +from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator + +_CALLS: list[int] = [] +# Tables the function damages on its next call, standing in for another process +# breaking a table while the job runs. +_DAMAGE_ON_NEXT_CALL: list[Path] = [] + + +@function_pod(output_keys="y") +def double(x: int) -> int: + """Record the call, damage any queued table, and return twice ``x``.""" + _CALLS.append(x) + while _DAMAGE_ON_NEXT_CALL: + require_unknown_reader_feature(_DAMAGE_ON_NEXT_CALL.pop()) + return 2 * x + + +@function_pod(output_keys="z") +def triple(x: int) -> int: + """Return three times ``x``.""" + return 3 * x + + +ORCHESTRATORS: dict[str, Callable[[], object]] = { + "sync": lambda: SyncPipelineOrchestrator(error_policy="continue"), + "async": lambda: AsyncPipelineOrchestrator(error_policy="continue"), +} + + +@pytest.fixture(autouse=True) +def _reset_calls(): + """Start and end every test with no recorded calls and no queued damage.""" + _CALLS.clear() + _DAMAGE_ON_NEXT_CALL.clear() + yield + _CALLS.clear() + _DAMAGE_ON_NEXT_CALL.clear() + + +@pytest.fixture(params=list(ORCHESTRATORS)) +def make_orchestrator(request) -> Callable[[], object]: + """Return a factory for the parametrized orchestrator.""" + return ORCHESTRATORS[request.param] + + +def build_job(store_path: Path, keys: str = "abc") -> PipelineJob: + """Build a job that doubles ``x`` for one row per key in ``keys``.""" + source = ArrowTableSource( + pa.table({"k": list(keys), "x": list(range(1, len(keys) + 1))}), + tag_columns=["k"], + infer_nullable=True, + ) + job = PipelineJob(name="itl884", store=DeltaTableDatabase(base_path=store_path)) + with job: + _ = double.pod(source, label="dbl") + return job + + +def table_dirs(store_path: Path) -> tuple[Path, Path]: + """The (result table, pipeline table) directories the job wrote.""" + dirs = {p.parent for p in store_path.rglob("*.parquet")} + result = [d for d in dirs if "_result" in d.parts] + pipeline = [d for d in dirs if "_result" not in d.parts] + assert len(result) == 1 and len(pipeline) == 1, sorted(dirs) + return result[0], pipeline[0] + + +def delete_data_files(table: Path) -> None: + """Delete a table's data files, leaving its log intact.""" + for f in table.glob("*.parquet"): + f.unlink() + + +def require_unknown_reader_feature(table: Path) -> None: + """Commit a log version that requires a reader feature deltalake does not know. + + Data files already loaded stay readable through the job's open table handle, + but loading the table again, as every flush does, fails. + """ + log = table / "_delta_log" + version = max(int(p.stem) for p in log.glob("*.json")) + 1 + protocol = { + "minReaderVersion": 3, + "minWriterVersion": 7, + "readerFeatures": ["orcapodMadeUpFeature"], + "writerFeatures": ["orcapodMadeUpFeature"], + } + (log / f"{version:020d}.json").write_text(json.dumps({"protocol": protocol})) + + +@pytest.fixture +def stored(tmp_path: Path, make_orchestrator) -> Path: + """A store in which the job has computed and stored all three rows.""" + store_path = tmp_path / "store" + build_job(store_path).run(orchestrator=make_orchestrator()) + assert len(_CALLS) == 3 + _CALLS.clear() + return store_path + + +def test_unreadable_result_table_fails_the_run(stored, make_orchestrator): + """An unreadable result table fails the run before any function call.""" + result_table, _ = table_dirs(stored) + delete_data_files(result_table) + + job = build_job(stored) + with pytest.raises(StoreReadError) as info: + job.run(orchestrator=make_orchestrator()) + + assert Path(info.value.table) == result_table + assert _CALLS == [] + + +def test_unreadable_result_table_fails_the_run_under_continue( + stored, make_orchestrator +): + """The per-row lookup's read failure fails the run under ``continue``.""" + # With the pipeline table gone, the job has no stored tag rows to join, so the + # first read of the result table is the per-row cache lookup, which runs inside + # the error_policy="continue" handler. + result_table, pipeline_table = table_dirs(stored) + delete_data_files(result_table) + shutil.rmtree(pipeline_table) + + job = build_job(stored) + with pytest.raises(StoreReadError) as info: + job.run(orchestrator=make_orchestrator()) + + assert Path(info.value.table) == result_table + assert _CALLS == [] + + +def test_store_read_failure_in_a_row_flush_fails_the_run(stored, make_orchestrator): + """A read failure in the flush that stores a row's result fails the run.""" + # Rows d and e are new. The result table breaks while d is computed, after + # d's cache lookup succeeded, so the failure surfaces in the flush that stores + # d's result, inside the error_policy="continue" per-row handler. Before the + # fix the flush wrapped the StoreReadError in a RuntimeError, the handler + # skipped d as a failed row, computed e, and the run ended with RuntimeError. + result_table, _ = table_dirs(stored) + _DAMAGE_ON_NEXT_CALL.append(result_table) + + orchestrator = make_orchestrator() + job = build_job(stored, keys="abcde") + with pytest.raises(StoreReadError) as info: + job.run(orchestrator=orchestrator) + + assert Path(info.value.table) == result_table + if isinstance(orchestrator, SyncPipelineOrchestrator): + # The run stops at d: e is never computed. + assert _CALLS == [4] + else: + # The async pod computes rows concurrently, so e may already be in flight + # when d's flush fails. + assert 4 in _CALLS and len(_CALLS) <= 2 + + +def test_absent_tables_are_computed(stored, make_orchestrator): + """Tables that do not exist are computed and stored, not reported unreadable.""" + result_table, pipeline_table = table_dirs(stored) + shutil.rmtree(result_table) + shutil.rmtree(pipeline_table) + + build_job(stored).run(orchestrator=make_orchestrator()) + assert sorted(_CALLS) == [1, 2, 3] + + # The recomputed rows were stored: a fresh job over the same store computes + # nothing. (Reading the node right after its own run can miss rows, ITL-864.) + _CALLS.clear() + build_job(stored).run(orchestrator=make_orchestrator()) + assert _CALLS == [] + + +class _FailsToStartNode(NoOpObserver): + """An observer whose ``on_node_start`` raises for one node label.""" + + def __init__(self, label: str) -> None: + """Fail ``on_node_start`` for the node labelled ``label``.""" + self.label = label + + def on_node_start( + self, node_label: str, node_hash: str, tag_schema: object = None + ) -> None: + """Raise ``ValueError`` when the node labelled ``self.label`` starts.""" + if node_label == self.label: + raise ValueError(f"observer failed for {node_label}") + + +def test_unrelated_concurrent_failure_does_not_hide_store_read_error(tmp_path): + """An unrelated concurrent failure does not hide the ``StoreReadError``.""" + # Node "dbl" raises StoreReadError while loading its stored results and node + # "tri" fails in the observer's on_node_start. Both fail before their first + # await, so both land in the async TaskGroup's exception group. Before the fix, + # re-raising inside "except* StoreReadError" left the unmatched ValueError in + # the group and the caller got an ExceptionGroup, whatever the Python version. + stored = tmp_path / "store" + _ = build_job(stored).run() + result_table, _ = table_dirs(stored) + delete_data_files(result_table) + + source = ArrowTableSource( + pa.table({"k": ["a", "b", "c"], "x": [1, 2, 3]}), + tag_columns=["k"], + infer_nullable=True, + ) + job = PipelineJob(name="itl884", store=DeltaTableDatabase(base_path=stored)) + with job: + _ = double.pod(source, label="dbl") + _ = triple.pod(source, label="tri") + + with pytest.raises(StoreReadError) as info: + job.run( + orchestrator=AsyncPipelineOrchestrator(), + observer=_FailsToStartNode("tri"), + ) + + assert type(info.value) is StoreReadError + assert Path(info.value.table) == result_table + assert info.value.__cause__ is not None + notes = getattr(info.value, "__notes__", []) + assert any("ValueError: observer failed for tri" in note for note in notes), notes