diff --git a/.basedpyright/baseline.json b/.basedpyright/baseline.json index 9de5b7cdc..ebffbbc51 100644 --- a/.basedpyright/baseline.json +++ b/.basedpyright/baseline.json @@ -45553,7 +45553,7 @@ "code": "reportAttributeAccessIssue", "range": { "startColumn": 21, - "endColumn": 40, + "endColumn": 43, "lineCount": 1 } }, diff --git a/.zed/rules b/.zed/rules index 39f79b31a..e50d39c04 100644 --- a/.zed/rules +++ b/.zed/rules @@ -854,7 +854,7 @@ data. The rules: the rows of that run's inputs, sync or async (ITL-1093). Before a run in this process (a fresh or loaded job), after a run that raised, after clear_cache(), and while stale, every function and operator job node reads empty (zero typed rows; head() is not_computed and - points to run() and as_source()), in every cache mode: no read touches the store. A + points to run() and as_source()), in every cache mode: no read touches the database. A never-run or cleared node has no modified time, so it is stale; a job node is stale also when any function or operator job node it depends on is (core/nodes/staleness.py), and a SourceJobNode is modified when it is bound or its bound source is. A run skips inputs diff --git a/CLAUDE.md b/CLAUDE.md index 3ecc733fb..da5c02829 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -896,7 +896,7 @@ data. The rules: the rows of that run's inputs, sync or async (ITL-1093). Before a run in this process (a fresh or loaded job), after a run that raised, after `clear_cache()`, and while stale, every function and operator job node reads empty (zero typed rows; `head()` is `not_computed` and - points to `run()` and `as_source()`), in every cache mode: no read touches the store. A + points to `run()` and `as_source()`), in every cache mode: no read touches the database. A never-run or cleared node has no modified time, so it is stale; a job node is stale also when any function or operator job node it depends on is (`core/nodes/staleness.py`), and a `SourceJobNode` is modified when it is bound or its bound source is. A run skips inputs diff --git a/README.md b/README.md index 43c4168d4..046592701 100644 --- a/README.md +++ b/README.md @@ -238,7 +238,7 @@ While the following is subject to change based on future development, it represe │ on success: │ ctx_observer.on_data_end(node_label, tag, data, result, cached=False) │ emit (tag_out, result) downstream - │ on StoreError (the store could not be read or written): + │ on OrcapodDatabaseError (the database could not be read or written): │ ctx_observer.on_data_crash(node_label, tag, data, exc) │ raise, whatever error_policy is — the run stops (ITL-878, ITL-884) │ on any other exception (the function failed): diff --git a/changelog.d/ITL-1087.md b/changelog.d/ITL-1087.md new file mode 100644 index 000000000..896b6446f --- /dev/null +++ b/changelog.d/ITL-1087.md @@ -0,0 +1,17 @@ +### Breaking changes and migration + +#### The remaining "store" names are "database" names (ITL-1087) + +`PipelineJob(database=...)` already names the Arrow database; the names around it now do too. There is no alias for any old name. A saved job file, a stored table, a column, a layout and every hash are unchanged. + +- `set_ephemeral_store(store)` on a pipeline, a node and the node protocols is `set_ephemeral_database(database)`. Rename the call and pass the database as `database=` if you used the keyword. The node attribute `ephemeral_result_store` is `ephemeral_result_database`. +- `StoreError` is `OrcapodDatabaseError`. Rename every `except StoreError`. `SideEffectLogDataContextError` keeps its name and derives from `OrcapodDatabaseError`. +- `StoreWriteError` is `OrcapodDatabaseWriteError` and `StoreReadError` is `OrcapodDatabaseReadError`. +- `StoreContentWarning` is `OrcapodDatabaseContentWarning`. Rename it in `warnings.filterwarnings` and `pytest.warns`. +- `describe_store_table` is `describe_database_table`, and the `store_path` parameter of `account_for_stale_tables` and `find_stale_tables` is `database_path` (all in `orcapod.contexts.stored`). +- The module `orcapod.utils.store_errors` is `orcapod.utils.database_errors`; `store_error_of` and `keep_store_error` are `database_error_of` and `keep_database_error`. +- Error and warning messages say "database" where they said "store". + +### Fixed + +- `PipelineJob.set_ephemeral_database(...)` no longer raises `AttributeError` on a job that contains a side-effect node, live or loaded. The side-effect node takes no ephemeral database and is skipped; function and operator nodes receive it as before (ITL-1147). diff --git a/design_issues/ITL-1040-DG2.md b/design_issues/ITL-1040-DG2.md index 4f17208d7..5ed8373f8 100644 --- a/design_issues/ITL-1040-DG2.md +++ b/design_issues/ITL-1040-DG2.md @@ -15,7 +15,7 @@ time with `data.drop`. Their sync paths drop the `_source_` columns from the tab explicitly. So an async run stored an extra `_source_` in the function node's pipeline records that a sync run did not, and the stored table's schema depended on the orchestrator. This happened after the automatic select in front of a pod, and after any -explicit select or drop. On one store, mixing orchestrators then either widened a Delta table +explicit select or drop. On one database, mixing orchestrators then either widened a Delta table (with nulls) or failed an in-memory commit with a schema mismatch. The cell values were always the row's own. The row-alignment suite found it (ITL-1034). It had been reported for `Data` alone as ITL-838. diff --git a/design_issues/ITL-1042-NS1.md b/design_issues/ITL-1042-NS1.md index 645152838..fd30f9cc1 100644 --- a/design_issues/ITL-1042-NS1.md +++ b/design_issues/ITL-1042-NS1.md @@ -14,7 +14,7 @@ run. Its first `iter_data()` / `as_table()` then threw away the in-memory result node's whole stored table (two `get_all_records` calls) and stamped the node. That made every operator node downstream of it stale, and their `head()` reported `not_computed` ("inputs changed since the last computation") with no rows, although `as_table()` and `iter_data()` -still returned every row. `head()` of the function node itself read the store instead of +still returned every row. `head()` of the function node itself read the database instead of memory. No function was called again and no operator recomputed. **Rule:** a job node stamps its modified time once per completed run, after it has read all diff --git a/design_issues/ITL-1043-LJ1.md b/design_issues/ITL-1043-LJ1.md index d5a5794ee..7734770f5 100644 --- a/design_issues/ITL-1043-LJ1.md +++ b/design_issues/ITL-1043-LJ1.md @@ -32,7 +32,7 @@ source's identity config record `"nullable": {"tag": [...], "data": [...]}`, res `T | None` through the type converter's own rule (`serialization.serialize_nullability`, `deserialize_output_schema`, `deserialize_schema(nullable=...)`). `PipelineJob.save` records the run content hash of operator nodes in `run.node_job_content_hashes`; a loaded -operator node returns it, and the pipeline hash saved in its descriptor. The store is the +operator node returns it, and the pipeline hash saved in its descriptor. The database is the source of truth (developer, November 12, 2026): a loaded REPLAY node that knows its key reads as the node built from code does, so no rows under its key (no table, or none of its rows) is an empty result. A node whose file lacks the run content hash cannot name its @@ -42,7 +42,7 @@ written before them) loads as before, lossy, and saving it again records nothing not have. No hash, stored path or golden value of a running job changes. Facts about stored data stay out of the definition: the job file holds only an optional, advisory `run.expected_rows` section (rows under each node's run key at save time, written only -when certain); a loaded node returns what the store holds and issues one -`StoreContentWarning` when the number differs. A loaded function node no longer raises on -a store without its run's results (the ITL-904 `node_has_stored_records` flag is no longer +when certain); a loaded node returns what the database holds and issues one +`OrcapodDatabaseContentWarning` when the number differs. A loaded function node no longer raises on +a database without its run's results (the ITL-904 `node_has_stored_records` flag is no longer written or read). diff --git a/design_issues/ITL-1045-FP1.md b/design_issues/ITL-1045-FP1.md index 5462f436b..ff9e6c924 100644 --- a/design_issues/ITL-1045-FP1.md +++ b/design_issues/ITL-1045-FP1.md @@ -14,6 +14,6 @@ and no row's `_content_hash` was the hash of its own data. **Fix:** `as_table` clears a stale cache first, before any of it is used, so the table and the hashes come from the same rows and the call returns the current rows. For a job node no -public sequence changes the values between the two passes (its results come from the store), +public sequence changes the values between the two passes (its results come from the database), so the defect was latent there; the same few lines fix its structure. No hash, golden value, stored table or public API changes. diff --git a/design_issues/ITL-1052-F18.md b/design_issues/ITL-1052-F18.md index 60bca5704..a9811815f 100644 --- a/design_issues/ITL-1052-F18.md +++ b/design_issues/ITL-1052-F18.md @@ -37,9 +37,9 @@ current (the result may mix two states of the input), so: unnoticed by the last stream, hidden by the false staleness above. The generation and the noted state are in-memory bookkeeping only: they enter no hash, no -saved file and no store. Regressions: `tests/test_core/function_pod/test_pod_stream_staleness.py` +saved file and no database. Regressions: `tests/test_core/function_pod/test_pod_stream_staleness.py` (chains of two and three, sequential and concurrent, the cached pod stream, a chain through a -join; second reads call no function and read no store; a reset upstream recomputes; a reset +join; second reads call no function and read no database; a reset upstream recomputes; a reset during a read, or made by the function while the stream computes, leaves the reader stale and its next read is of one state). diff --git a/design_issues/ITL-1053-PV2.md b/design_issues/ITL-1053-PV2.md index 027b76c69..353aa7cec 100644 --- a/design_issues/ITL-1053-PV2.md +++ b/design_issues/ITL-1053-PV2.md @@ -21,7 +21,7 @@ ITL-1029) and slice, so `head(n, cfg) == as_table(cfg).slice(0, n)`. A bounded r (`SpiralDBTableSource`, a partly computed stream, a polled source's batches) sorts the rows it has: "the rows read, in tag order", not "the first `n` in tag order". Nothing computes or reads more than before. When ITL-1002 bounds a node's stored read, a node's sorted head becomes "the -rows read, sorted" unless the store returns rows in tag order. +rows read, sorted" unless the database returns rows in tag order. The notebook preview asks for the tag-sorted head (developer, November 12, 2026: "preview should sort by tags"; `preview_html`), and its header says "sorted by tags", or for a bounded read diff --git a/design_issues/ITL-1060-FN1.md b/design_issues/ITL-1060-FN1.md index 69cacfc27..c5690ce91 100644 --- a/design_issues/ITL-1060-FN1.md +++ b/design_issues/ITL-1060-FN1.md @@ -1,6 +1,6 @@ -## `src/orcapod/core/nodes/function_node.py` — Rows read back from a function node's store +## `src/orcapod/core/nodes/function_node.py` — Rows read back from a function node's database -### FN1 — A result row read back from the store got a fresh datagram id instead of its stored `__data_id` +### FN1 — A result row read back from the database got a fresh datagram id instead of its stored `__data_id` **Status:** resolved **Severity:** low **Issue:** ITL-1060 diff --git a/design_issues/ITL-1095-CR1.md b/design_issues/ITL-1095-CR1.md index ff78f3191..e5c1e712a 100644 --- a/design_issues/ITL-1095-CR1.md +++ b/design_issues/ITL-1095-CR1.md @@ -17,7 +17,7 @@ one level up only, so a source rebound two nodes up left the nodes below it "cur **Fix:** (developer, November 13, 2026) a job node shows its current run and `node.as_source()` shows everything stored. Function and operator job nodes read empty (typed zero rows, `head()` `not_computed` pointing to `run()` and `as_source()`) until a run in this process, after -`clear_cache()`, after a run that raised and while stale; no read touches the store. A never-run +`clear_cache()`, after a run that raised and while stale; no read touches the database. A never-run or cleared node has no modified time (stale); `core/nodes/staleness.py` makes a job node stale when any function or operator job node above it is; a `SourceJobNode` is modified when bound or when its bound source is. `job.run(load_only=True)` fills the nodes from what is stored and diff --git a/design_issues/ITL-1122-LN1.md b/design_issues/ITL-1122-LN1.md index 0d75f9339..87f5f96f3 100644 --- a/design_issues/ITL-1122-LN1.md +++ b/design_issues/ITL-1122-LN1.md @@ -6,7 +6,7 @@ **Issue:** ITL-1122 The rule (developer, November 14, 2026, replacing an earlier form with exceptions for a user's -Arrow table and for a store that keeps the distinction): every column of every table Orcapod +Arrow table and for a database that keeps the distinction): every column of every table Orcapod presents or stores has its string, binary and list types in their large variants, with no exception. Measured on `main` 3a7f3850: `ArrowTableSource` and an explicit Arrow schema kept `string`; `CSVSource` gave `string` (a CSV file keeps no distinction); `CachedSource` gave the diff --git a/design_issues/ITL-1133-ZR3.md b/design_issues/ITL-1133-ZR3.md index 9bde42acb..90d2d624a 100644 --- a/design_issues/ITL-1133-ZR3.md +++ b/design_issues/ITL-1133-ZR3.md @@ -11,8 +11,8 @@ own (`OperatorJobNode.run()`, `_compute_and_store`) kept its operator's result a the operator's own order (a join lists its data before its system tags; every operator that keeps its input's order did the same over a source, which lists system tags after data). So one node had two populated orders, its table before a run (row order) matched only one of them, and -a store holding rows of both refused the second append (`StoreWriteError`, schema mismatch, in -the in-memory store and in a Delta store with a pending batch). Only the operator cache +a database holding rows of both refused the second append (`OrcapodDatabaseWriteError`, schema mismatch, in +the in-memory store and in a Delta database with a pending batch). Only the operator cache (`odb_v1`) was affected: function pipeline and result tables, side-effect logs and cached sources store one order on every path. `odb_v1` dates from 2026-11-12 (ITL-967) and was never released. diff --git a/design_issues/ITL-864-DW1.md b/design_issues/ITL-864-DW1.md index f6757bc09..2322d986c 100644 --- a/design_issues/ITL-864-DW1.md +++ b/design_issues/ITL-864-DW1.md @@ -26,7 +26,7 @@ pipeline; the async orchestrator runs only user functions in worker threads. Reg `tests/test_databases/test_delta_read_your_writes.py` and `tests/test_pipeline/test_delta_node_records_after_run.py`. Stored ids come from `_stored_ids`: the snapshot of the loaded table version reads them on first use, through an -id-only scan (a one-column schema); a failed scan raises `StoreReadError` and leaves the +id-only scan (a one-column schema); a failed scan raises `OrcapodDatabaseReadError` and leaves the snapshot's ids unset (`None`), so the next read scans again instead of treating every stored id as unstored and returning duplicates. A handle that loaded the table before it had any stored ids (a fresh root over an existing table) reads them on its first read with pending diff --git a/design_issues/ITL-878-SW1.md b/design_issues/ITL-878-SW1.md index 385cf8bb8..49adca2e5 100644 --- a/design_issues/ITL-878-SW1.md +++ b/design_issues/ITL-878-SW1.md @@ -1,11 +1,11 @@ -## `src/orcapod/pipeline/job.py`, orchestrators, node write paths — store write failures +## `src/orcapod/pipeline/job.py`, orchestrators, node write paths — database write failures -### SW1 — A failed store write did not fail `PipelineJob.run()` +### SW1 — A failed database write did not fail `PipelineJob.run()` **Status:** resolved **Severity:** critical **Issue:** ITL-878 -A node's write to the store ran inside the per-row handler meant for function failures: +A node's write to the database ran inside the per-row handler meant for function failures: `FunctionNode.execute` (sync) and `_FunctionPodBase.async_execute.process_one` (async) wrap compute *and* the result-table and pipeline-table writes in one `except Exception`. With the default `error_policy="continue"` a failed write was skipped like a crashed function: the sync @@ -15,30 +15,30 @@ the node read back fewer rows (none when every write failed, e.g. a logical-type final flush, so the upstream function node's pending pipeline rows were never committed while its result rows were; async callers got an `ExceptionGroup`. -**Fix:** Contract, for now (developer, November 10, 2026): a failed store write always fails +**Fix:** Contract, for now (developer, November 10, 2026): a failed database write always fails the run, whatever `error_policy` is. Every node write site (`ResultCache.store`, `FunctionNode.add_pipeline_record`, `OperatorNode._store_output_stream`, the side-effect -invocation log) raises `orcapod.errors.StoreWriteError`, chained from the backend error and +invocation log) raises `orcapod.errors.OrcapodDatabaseWriteError`, chained from the backend error and naming the node, the row's tag and the table. The per-row handlers re-raise it, so the run stops at the first failed write; `error_policy` still decides only about rows whose function raised. `AsyncPipelineOrchestrator` unwraps it from the `TaskGroup` exception group, so both orchestrators raise the same error. `PipelineJob.run()` flushes the writes that succeeded before re-raising (a flush failure there is logged, not raised over the original), and wraps a failure of its final flush. Nothing stored is rewritten; no hash, path or schema change. An -opt-in mode that tolerates store failures is a possible later option. A fail-loud post-run -hook or an observer callback that raises while the store error propagates is logged and noted -on it (`orcapod.utils.store_errors`), not raised in its place, so a node function that runs -another cached pod cannot turn that pod's store failure into a skipped row. When concurrent -node tasks fail together, the async orchestrator raises the store error of the node first in -topological order, notes the other failures on it and logs other store errors at WARNING. +opt-in mode that tolerates database failures is a possible later option. A fail-loud post-run +hook or an observer callback that raises while the database error propagates is logged and noted +on it (`orcapod.utils.database_errors`), not raised in its place, so a node function that runs +another cached pod cannot turn that pod's database failure into a skipped row. When concurrent +node tasks fail together, the async orchestrator raises the database error of the node first in +topological order, notes the other failures on it and logs other database errors at WARNING. A failed flush (the result cache's per-row auto-flush or the job's final flush) raises -`StoreWriteError(flush=True)` naming the store and no row: views of one store share their +`OrcapodDatabaseWriteError(flush=True)` naming the database and no row: views of one database share their pending writes, so the failing batch may be an earlier row's or another table's. -The rule (developer, November 11, 2026): store failures fail the run; a function output that +The rule (developer, November 11, 2026): database failures fail the run; a function output that cannot be stored (a naive `datetime`, a value not matching the declared type, rejected while `ResultCache.store` prepares the record, before any write) is a row failure governed by -`error_policy` exactly like a function that raised, not a `StoreWriteError`: the value is -specific to that row, the store is healthy, and every re-run would fail at the same row. The +`error_policy` exactly like a function that raised, not a `OrcapodDatabaseWriteError`: the value is +specific to that row, the database is healthy, and every re-run would fail at the same row. The async orchestrator ignored `fail_fast` for such rows (ITL-879), unchanged here and fixed since by ITL-879 (`ITL-879-EP1`). Regressions: `tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async diff --git a/design_issues/ITL-879-EP1.md b/design_issues/ITL-879-EP1.md index 3f10abac7..f9023b74f 100644 --- a/design_issues/ITL-879-EP1.md +++ b/design_issues/ITL-879-EP1.md @@ -9,7 +9,7 @@ handler (`_FunctionPodBase.async_execute.process_one`) skipped every row whose function raised, so under `error_policy="fail_fast"` an async run computed every other row and finished, with a `RunSummary` and a `RowFailuresWarning`, where the sync run raised the row's -exception after storing the rows before it. Any other node failure that was not a store +exception after storing the rows before it. Any other node failure that was not a database error (a side-effect pod with `on_error="raise"`, an operator that rejected its inputs) reached the caller as the task group's `ExceptionGroup`, where the sync run raised the exception itself. Under `"continue"` the two orchestrators already agreed: no raise, the @@ -23,12 +23,12 @@ and the orchestrator passes its own. Under `"fail_fast"` the pod re-raises the r exception after `on_data_crash`; the existing task groups cancel the rows and nodes still running and every node closes its output in its existing `finally`; the failing node calls `on_node_end` before it raises, as `execute` does. The orchestrator's cancel and close -machinery is unchanged. Its exception-group handler now always raises ONE exception: a store +machinery is unchanged. Its exception-group handler now always raises ONE exception: a database error, then a `TypeIdentityError` (both as before), then the exception of the node earliest in topological order (ties: the order the failures happened), the others noted on it (`_single_failure`). A function node cancelled by another node's failure commits what it wrote (`_commit_after_failure`) before the cancellation propagates, as a failing node does: -the job's flush commits neither its ephemeral store nor its result table ahead of its +the job's flush commits neither its ephemeral database nor its result table ahead of its pipeline records (CodeRabbit, PR #379); it then holds no rows, as a node whose run raised (ITL-1093). An observer whose `on_node_end` raises as the node fails does not skip that commit either, as on the sync path, and is not called a second time. Rows run concurrently, diff --git a/design_issues/ITL-884-DR1.md b/design_issues/ITL-884-DR1.md index 2bb639ac9..2ffbeab09 100644 --- a/design_issues/ITL-884-DR1.md +++ b/design_issues/ITL-884-DR1.md @@ -18,7 +18,7 @@ affected: the in-memory and connector databases (SQLite, PostgreSQL, SpiralDB) r only for a table that does not exist and let read errors propagate. **Fix:** only `TableNotFoundError` means absent. Every other load or read error raises -`orcapod.errors.StoreReadError`, which names the table and chains the cause. `FunctionNode.execute`, the +`orcapod.errors.OrcapodDatabaseReadError`, which names the table and chains the cause. `FunctionNode.execute`, the async `FunctionPod` row handler and the async orchestrator let it fail the run whatever the `error_policy`. Before merging into an existing table, `_flush_by_key` compares the batch with the stored Delta schema through Delta's own type mapping (extension types by storage, @@ -30,16 +30,16 @@ are converted with the Delta writer's own conversion (`deltalake.writer._convers field metadata, so a type the writer stores differently (timestamps in microseconds, unsigned as signed) or a logical type whose Delta storage differs is never refused; the differential guard `tests/test_databases/test_delta_append_types.py` appends an identical schema for every Arrow and -logical type Delta can store. `flush()` raises a `StoreReadError` or +logical type Delta can store. `flush()` raises a `OrcapodDatabaseReadError` or that `InputValidationError` as itself, not wrapped in `RuntimeError`, with every other failed batch noted on it, so a read failure in a per-row result-cache flush still fails the run. The -async orchestrator selects the `StoreReadError` from the `TaskGroup`'s exception group with a +async orchestrator selects the `OrcapodDatabaseReadError` from the `TaskGroup`'s exception group with a plain `except`, noting the other failures, so a concurrent unrelated failure cannot turn it into -an `ExceptionGroup`. With ITL-878 (SW1) both errors derive from `orcapod.errors.StoreError` -and share one path: the per-row handlers, `orcapod.utils.store_errors` and both orchestrators -handle `StoreError`. A write that fails because the stored table cannot be read raises -`StoreReadError`, not `StoreWriteError`; a refused append is a `StoreWriteError` caused by the -`InputValidationError`. The orchestrators label a store error with the node it escaped from, +an `ExceptionGroup`. With ITL-878 (SW1) both errors derive from `orcapod.errors.OrcapodDatabaseError` +and share one path: the per-row handlers, `orcapod.utils.database_errors` and both orchestrators +handle `OrcapodDatabaseError`. A write that fails because the stored table cannot be read raises +`OrcapodDatabaseReadError`, not `OrcapodDatabaseWriteError`; a refused append is a `OrcapodDatabaseWriteError` caused by the +`InputValidationError`. The orchestrators label a database error with the node it escaped from, and the async orchestrator ranks read and write errors alike by node order. Regressions: `tests/test_databases/test_delta_read_failures.py`, `tests/test_pipeline/test_store_read_failure.py` and diff --git a/design_issues/ITL-895-FO1.md b/design_issues/ITL-895-FO1.md index ff3cf468b..d6b8b72ec 100644 --- a/design_issues/ITL-895-FO1.md +++ b/design_issues/ITL-895-FO1.md @@ -12,7 +12,7 @@ unchanged. pyarrow's integer converter calls `__int__` on a non-`int`, so `1.5`, `numpy.float64(1.5)`, `Decimal` and `Fraction` were truncated. Numbers were read as epoch ticks for date and timestamp fields, `bytes` and `str` were decoded and encoded into each other, and `None` became a NULL in a not-null field. The `Data` kept the raw dict, so `as_dict()` gave -`1.5` while `as_table()`, the store and the hash gave `1`. Downstream pods read `as_dict()`, so +`1.5` while `as_table()`, the database and the hash gave `1`. Downstream pods read `as_dict()`, so the same graph gave different results standalone, on the sync orchestrator and on the async orchestrator. Present in every release up to v0.2.0. Root-cause evidence: `superpowers/specs/itl-895-float-to-int/FINDINGS.md`. diff --git a/design_issues/ITL-896-OR1.md b/design_issues/ITL-896-OR1.md index ef8ffd80b..aaa0abbfa 100644 --- a/design_issues/ITL-896-OR1.md +++ b/design_issues/ITL-896-OR1.md @@ -20,7 +20,7 @@ path or stored schema changes. A run with no rows clears the previous run's outp never serves stale rows (CodeRabbit on PR #302). `get_all_records()` still returns `None` for an operator node in a job, since with the default `cache_mode=OFF` (the only mode a job sets) it stores nothing. `clear_cache()` still returns the node to its zero-row "not yet run" -state on both paths (a function node instead reloads from its store). +state on both paths (a function node instead reloads from its database). Regressions: `tests/test_pipeline/test_operator_node_async_readback.py`, which compares the sync and async reads of every operator. Open, pinned by strict xfails there: a zero-row async result reads back the table built from the Python output schema (ITL-858, ITL-865); that table widens diff --git a/design_issues/ITL-904-PJLOAD1.md b/design_issues/ITL-904-PJLOAD1.md index 919a4351c..060d257b2 100644 --- a/design_issues/ITL-904-PJLOAD1.md +++ b/design_issues/ITL-904-PJLOAD1.md @@ -8,7 +8,7 @@ Saving a job and loading it with `PipelineJob.load` failed in four ways: - On a loaded function node, `as_table()` and `iter_data()` raised "no function pod and no - database attached", although the store was attached and `get_all_records()` read the rows. + database attached", although the database was attached and `get_all_records()` read the rows. The cause: `FunctionJobNode.from_descriptor` set `LoadStatus.UNAVAILABLE`, and attaching the store never changed that status. - A loaded node's `output_schema()` returned the serialized type strings (`'int64'`), and its @@ -21,16 +21,16 @@ Saving a job and loading it with `PipelineJob.load` failed in four ways: **Fix:** -- Attaching a store makes a loaded function node `READ_ONLY`. `iter_data()` yields the stored +- Attaching a database makes a loaded function node `READ_ONLY`. `iter_data()` yields the stored rows: those whose `_node_content_hash` is the content hash saved in `run.node_job_content_hashes`, the same set `get_all_records()` returns. `as_table()` is an `ArrowTableStream` over the stored Arrow records, so stored types are kept and `all_info` includes system tags, source info and the data context. -- Without a store, reading names `PipelineJob.load(path, store=...)` and `job.bind(store=...)`. -- `save()` records `run.node_has_stored_records`: whether the store held at least one pipeline +- Without a database, reading names `PipelineJob.load(path, database=...)` and `job.bind(database=...)`. +- `save()` records `run.node_has_stored_records`: whether the database held at least one pipeline record carrying each function node's content hash. This is an additive, optional field, and - the format version is unchanged. A node saved with records raises, naming the store and the - table, when the attached store holds none of them, instead of reading zero rows. Records + the format version is unchanged. A node saved with records raises, naming the database and the + table, when the attached database holds none of them, instead of reading zero rows. Records that other runs wrote to a shared table do not count. Files saved without the field read zero rows, as before. - A loaded node's `output_schema()` returns Python types (`deserialize_output_schema`), and its @@ -45,20 +45,20 @@ Saving a job and loading it with `PipelineJob.load` failed in four ways: registration is reported by node and column, leaving affected operators as stubs. Incompatible identifiers raise `LogicalTypeIdentifierError`; legacy name-only text raises an earlier-version error. This narrow saved-file change was approved November 12, 2026. -- `get_all_records()` on a loaded node raises the same store errors as `as_table()`. +- `get_all_records()` on a loaded node raises the same database errors as `as_table()`. - Re-saving a loaded job keeps each node's saved expectation of stored records. - One row per base entry, for `as_table()` and `iter_data()` alike. When an entry has several records (it was recomputed), the record with the greatest recomputation index wins. This - also changes a live node's hot-load from the store: it used to keep whichever record the + also changes a live node's hot-load from the database: it used to keep whichever record the join returned last. -- A loaded node whose results lived in an ephemeral store yields `EmptyData` placeholders from - `iter_data()`. Its `as_table()` raises an error saying the ephemeral store is not saved. +- A loaded node whose results lived in an ephemeral database yields `EmptyData` placeholders from + `iter_data()`. Its `as_table()` raises an error saying the ephemeral database is not saved. A newer missing ephemeral recomputation takes precedence over an older persistent row. - A loaded operator node with nothing stored raises, names its input nodes, and says to bind an unsaved input source first. - Stored-record expectations constrain both the node content hash and the saved data context. Rows of another data context cannot satisfy the expectation while reads omit those rows. -- Loading logs in-memory sources at INFO, and any other source or store failure at WARNING +- Loading logs in-memory sources at INFO, and any other source or database failure at WARNING with the exception. Each message states the consequence and the remedy. - `PipelineJob.run()` rejects every loaded job, including an operator-only one whose operators were rebuilt: a loaded job is read-only by design (developer, November 11, @@ -90,11 +90,11 @@ when their cache table is absent; loaded function `run()` rejects computation, a loaded jobs report `is_runnable()` as false even after binding sources. Source-schema restoration uses the saved context converter in both normal and fallback branches. -**Superseded in part (ITL-1043, developer decision, November 12, 2026):** the store is the +**Superseded in part (ITL-1043, developer decision, November 12, 2026):** the database is the source of truth, with one rule for function and operator nodes. `run.node_has_stored_records` is no longer written, and is ignored when an older file has it: a loaded function node no -longer raises on a store without its run's records; it returns what the store holds. The job -file's optional, advisory `run.expected_rows` makes a difference one `StoreContentWarning` +longer raises on a database without its run's records; it returns what the database holds. The job +file's optional, advisory `run.expected_rows` makes a difference one `OrcapodDatabaseContentWarning` per node. Likewise, a loaded REPLAY operator whose job file records its run's content hash reads an absent cache table as no rows instead of raising. See `design_issues/ITL-1043-LJ1.md`. diff --git a/design_issues/ITL-905-DO1.md b/design_issues/ITL-905-DO1.md index 574de6f16..2fc856193 100644 --- a/design_issues/ITL-905-DO1.md +++ b/design_issues/ITL-905-DO1.md @@ -30,7 +30,7 @@ On top of the v0.3 hash step, hashes move only for values not already in canonical order (two golden entries, `pipeline:index_pick.index` row data hashes); see the `0.3.0` changelog. This storage order is a separate layer from the semantic hasher's ordering of its hash input (ITL-942), which sorts by canonical JSON text and stores nothing. Regressions: `tests/test_hashing/test_dict_order_hash.py` (including a -Delta store written by `0.2.0`, which v0.3 does not read) and `tests/test_hashing/test_set_order_hash.py`. Open: a +Delta database written by `0.2.0`, which v0.3 does not read) and `tests/test_hashing/test_set_order_hash.py`. Open: a key/value list that arrives already in Arrow form keeps its stored order and hash, by the same ruling; dicts inside dataclass or pydantic objects keep their own order in memory (their stored and hashed form is canonical). A fresh user-class instance iterates its diff --git a/design_issues/ITL-906-O7.md b/design_issues/ITL-906-O7.md index 956170db4..d536e64c3 100644 --- a/design_issues/ITL-906-O7.md +++ b/design_issues/ITL-906-O7.md @@ -14,7 +14,7 @@ operator's or a function pod's output the async names, `MergeJoin`'s canonical i and every record id derived from them differed from the sync path. The `input_pipeline_hashes` keyword the operator node passes was ignored. A function node downstream then wrote two sets of system-tag columns into one pipeline table, and a run on one orchestrator could not extend -a store written by the other (ITL-885). +a database written by the other (ITL-885). **Fix:** a barrier path rebuilds each input with `_materialize_input_stream(rows, pipeline_hash)`, a stream that reports the input's pipeline hash (`_RowsOfStream`). @@ -26,5 +26,5 @@ use it. `_materialize_to_stream(rows)` only holds output rows. Async names and r sync run produces moves, and no golden value moves (the golden cases run the sync path). `tests/test_core/operators/test_async_sync_equivalence.py` runs every operator with an async path over inputs behind an operator and behind a function pod; -`tests/test_pipeline/test_mixed_orchestrators.py` runs one store with both orchestrators in +`tests/test_pipeline/test_mixed_orchestrators.py` runs one database with both orchestrators in both orders. diff --git a/design_issues/ITL-910-HS2.md b/design_issues/ITL-910-HS2.md index 31c2e4a3e..8f5bf9867 100644 --- a/design_issues/ITL-910-HS2.md +++ b/design_issues/ITL-910-HS2.md @@ -21,7 +21,7 @@ their extension identity instead of as their storage type. Stored rows: under the default `table_scope`, table paths do not move (developer, November 11, 2026: only a change of a table's schema goes into its path; a value-only change stays in the same table), so a v0.3 node writes into the same tables as a v0.2 store. The first decision -that day was to refuse such a store (`StoreHashLabelError`); the developer reversed it in the +that day was to refuse such a database (`StoreHashLabelError`); the developer reversed it in the ITL-938 design session: v0.3 does not refuse a v0.2 store. It ignores rows hashed under `arrow_v0.1`, computes every input again under `arrow_v0.2`, and leaves the old rows in place, unchanged and distinguishable, for a later migration (ITL-913, ITL-914). @@ -50,7 +50,7 @@ later discussion of what side-effect logs hold. Known limits, follow-ups: - The warning is logged once per reader object (node, result cache, cached source) and - table, not once per process: stored paths are relative to the store, so a process-wide + table, not once per process: stored paths are relative to the database, so a process-wide record of reported tables would silence a second store with the same layout. - A global hash-version identifier stored with every row is being designed (not decided); it would also give operator-node and cached-source rows a label. @@ -61,7 +61,7 @@ Known limits, follow-ups: - The node content hash does not separate the generations: a source with an explicit `source_id` gives the same node content hash under both labels. - A `DeltaTableDatabase` object keeps the table version it read first, so a job object that - read a table before a run does not see the rows the run added until the store is reopened + read a table before a run does not see the rows the run added until the database is reopened (not specific to hash labels; seen when re-reading a node of the same job after an incremental run). - Tables placed under a content hash (`table_scope="content_hash"`, `CachedSource` without an diff --git a/design_issues/ITL-919-PQ1.md b/design_issues/ITL-919-PQ1.md index 6d39a159e..7e442903c 100644 --- a/design_issues/ITL-919-PQ1.md +++ b/design_issues/ITL-919-PQ1.md @@ -6,7 +6,7 @@ **Issue:** ITL-919 A function node in a `PipelineJob` does, per row: `ResultCache.lookup` (lookup by input data -hash in the result table), `ResultCache.store` with a flush of the whole store (both +hash in the result table), `ResultCache.store` with a flush of the whole database (both tables), and `add_pipeline_record` (lookup by base entry id for the recomputation index, then `add_record(skip_duplicates=True)`). Sync and async orchestrators share that path; operator nodes write with one `add_records` and are not affected. Measured on `c8efc092`, @@ -39,7 +39,7 @@ pod: 0.09 ms). async rerun of an ephemeral-result node skipped its rows (ITL-923, fixed). Remaining (ITL-923 item 3): a pending row written with `skip_duplicates=False` whose id is already stored is still dropped by the commit (Delta keeps the stored row; the in-memory store - overwrites it). Memory: a store object keeps the committed rows of each table it looks rows + overwrites it). Memory: a database object keeps the committed rows of each table it looks rows up in for as long as it lives, all of them at once (main read one whole table per lookup and released it), so memory is cumulative over the tables: measured for an `int -> int` pod, about 820 bytes per stored row for the result table and 746 for the pipeline-record diff --git a/design_issues/ITL-926-SP2.md b/design_issues/ITL-926-SP2.md index 8db315c69..a22f896d1 100644 --- a/design_issues/ITL-926-SP2.md +++ b/design_issues/ITL-926-SP2.md @@ -8,7 +8,7 @@ Re-verified against the EDI sample project (`edi-sample-v1-855632`, read-only) and a disk-backed pyspiral fake (`tests/test_databases/fake_pyspiral.py`). The workshop path works: a `SpiralDBTableSource` with a composite key, a `Join` of two Spiral sources, a function pod, -a Delta store, a re-run with no function calls, save and load, and the same hashes in a fresh +a Delta database, a re-run with no function calls, save and load, and the same hashes in a fresh process. Findings, with repro scripts, are in `superpowers/specs/itl-926-live/FINDINGS.md`. **Fixed:** diff --git a/design_issues/ITL-928-DT1.md b/design_issues/ITL-928-DT1.md index e1a1424cd..499a98ca1 100644 --- a/design_issues/ITL-928-DT1.md +++ b/design_issues/ITL-928-DT1.md @@ -1,6 +1,6 @@ ## `src/orcapod/databases/delta_lake_databases.py` -### DT1 — the Delta store gave back other Arrow types than it was given +### DT1 — the Delta database gave back other Arrow types than it was given **Status:** resolved **Severity:** critical **Issue:** ITL-928 @@ -8,7 +8,7 @@ The Delta writer converts what Delta Lake has no type for: an unsigned integer to the signed integer of the same width (a value above the signed maximum failed the flush), `timestamp[s]`, `[ms]` and `[ns]` to `timestamp[us]` (nanosecond digits silently dropped), a zone other than UTC -to UTC; it cannot store `date64`, `time32/64` or `duration` at all. Every read of a Delta store +to UTC; it cannot store `date64`, `time32/64` or `duration` at all. Every read of a Delta database (function node hot-load and `get_all_records`, a fresh or loaded job, operator `REPLAY`, `CachedSource`) gave back the converted type. A tag read back that way changes the record ids computed from it (the base entry id hashes the tag's Arrow table), so a re-run of @@ -22,6 +22,6 @@ written as the same-width signed integer bit for bit, and `timestamp[ns]`, `date `duration` as their integer value; s/ms timestamps and zones go through Delta's own lossless conversion. Every read path (`get_all_records`, the snapshot scan, the snapshot extension after a commit) casts recorded columns back. Logical-type columns are left to their extension metadata. -Tables written before the fix have no recorded type and read as before; such a store must be +Tables written before the fix have no recorded type and read as before; such a database must be rebuilt (developer, November 12, 2026). Not covered: Delta also gives back `fixed_size_binary` as `binary` and a dictionary column as its value type; those are unchanged. diff --git a/design_issues/ITL-928-FN1.md b/design_issues/ITL-928-FN1.md index 9da2a27b9..361f168e2 100644 --- a/design_issues/ITL-928-FN1.md +++ b/design_issues/ITL-928-FN1.md @@ -10,7 +10,7 @@ values, typed by the node's Python output schema through the type converter. A f never touches tags, yet the tag columns came back with types inferred from Python: every integer width as `int64`, `float32` as `double`, every naive or zoned timestamp as `timestamp[us, tz=UTC]`, `date64` as `date32`; a `time`, `duration`, `decimal`, `uuid` or -`Path` tag made it raise. It happened on every store, sync and async, while `iter_data()`, +`Path` tag made it raise. It happened on every database, sync and async, while `iter_data()`, `get_all_records()` and the stored pipeline rows kept the input types. Joining a node's output with another source on such a tag raised "Join inputs disagree on the Arrow type". The same build also dropped the `_source_` and `_context_key` columns that the pod's stream @@ -24,4 +24,4 @@ concatenated by name; rows that disagree on a column's Arrow type raise `InputValidationError` naming the column and both types, never a widened type. No stored value, record id or hash changes. The zero-row table of a node that has not run is still built from the Python output schema (the ITL-912 problem, for function nodes). -The Delta store's own type changes are entry DT1 (`design_issues/ITL-928-DT1.md`). +The Delta database's own type changes are entry DT1 (`design_issues/ITL-928-DT1.md`). diff --git a/design_issues/ITL-944-DC1.md b/design_issues/ITL-944-DC1.md index 982cede85..df13201b9 100644 --- a/design_issues/ITL-944-DC1.md +++ b/design_issues/ITL-944-DC1.md @@ -46,7 +46,7 @@ Known limits, follow-ups: - The content hash of a stream without a producer covers its `_context_key` column, so it moves with the data context key. - Old-layout tables are reported once per reader object, not once per process or job, and - counting their rows reads the whole table (the store protocol has no row count). + counting their rows reads the whole table (the database protocol has no row count). - The manifest has the fields of the ITL-938 design document, section 7.4 (`released_in`, `previous`, `definitions`, `libraries`, `golden`, `changes`), and the logical types and their value handlers moved to the unfrozen `contexts/logical_types.json`. The names under diff --git a/design_issues/ITL-954-DC2.md b/design_issues/ITL-954-DC2.md index a832193bb..c263cf227 100644 --- a/design_issues/ITL-954-DC2.md +++ b/design_issues/ITL-954-DC2.md @@ -29,7 +29,7 @@ would deliver everything again on a v0.2 store. job that no node claims by name (operator caches, cached sources, nodes that left the job). - A store that cannot list its tables: a side-effecting node logs one warning and proceeds (developer, November 11, 2026). -- Proven on a store written by Orcapod v0.2.0 itself +- Proven on a database written by Orcapod v0.2.0 itself (`tests/test_arrays/data/v02_full_store.tar.gz`), as written and with every hash in its paths replaced (`tests/test_pipeline/test_pre_v03_tables.py`). diff --git a/design_issues/ITL-972-UCK1.md b/design_issues/ITL-972-UCK1.md index aee8c4866..723ae341d 100644 --- a/design_issues/ITL-972-UCK1.md +++ b/design_issues/ITL-972-UCK1.md @@ -15,7 +15,7 @@ dropping its extra fields. PR #292 (ITL-881, not merged) identified every class by a declared name, kind and ordered fields, so two different classes with the same description were interchangeable. A measurement on November 12, 2026 showed the consequence: in memory a pod annotated with one -class received the other class's object unchanged, while through a store it received its +class received the other class's object unchanged, while through a database it received its own class. **Fix:** two explicit kinds (developer, November 12, 2026; spec @@ -35,7 +35,7 @@ own class. and the record is the extension metadata, so a structural class is rebuilt from metadata when no class is defined (`orcapod.synthesized`). - Kept from PR #292: a subclass has its own identity; field order is identity; - `TypeIdentityError` is not a `StoreReadError` and fails the run under every error policy; + `TypeIdentityError` is not a `OrcapodDatabaseReadError` and fails the run under every error policy; the registry binds an identifier to the newest class object with that identity. Removed: `op.register_type`, declared names for exact classes, equivalence of exact classes. diff --git a/design_issues/NOISSUE-O5.md b/design_issues/NOISSUE-O5.md index 393c6612e..9109353d7 100644 --- a/design_issues/NOISSUE-O5.md +++ b/design_issues/NOISSUE-O5.md @@ -12,7 +12,7 @@ executes, and its result is persisted with nothing marking it incomplete. The ne the full set and the group is recomputed correctly. Measured with two upstream pod stages (`source → sync_like → stringify → [GroupBy] → pod`), -one member of one group changed, same Delta store, fresh objects per run: +one member of one group changed, same Delta database, fresh objects per run: ``` CONTROL (per-row, no GroupBy) diff --git a/docs/concepts/arrays.md b/docs/concepts/arrays.md index 15aeb2eac..5fdb40bb3 100644 --- a/docs/concepts/arrays.md +++ b/docs/concepts/arrays.md @@ -33,11 +33,11 @@ for tag, data in scale.pod(source).iter_data(): hashed the file. v0.3 stores them under `orcapod.numpy.ndarray`, hashes the value as described below, and does not migrate older stores. Every node that handles arrays, and every node downstream of an array column, gets new tables in v0.3. Because v0.3 also - changed every hash of Arrow data, v0.3 reuses no result of a store written by v0.2: it + changed every hash of Arrow data, v0.3 reuses no result of a database written by v0.2: it ignores the v0.2 rows and computes every input again (see [A store written by an older version](function-pods.md#a-store-written-by-an-older-version)). Nodes that perform side effects are the exception: a side-effect pod, or a function pod - that takes the invocation context, does not run again on such a store but raises + that takes the invocation context, does not run again on such a database but raises `SideEffectLogDataContextError` before it delivers anything. The v0.2 tables and rows stay on disk unchanged, readable with v0.2. A migration is planned after v0.3 (ITL-889). @@ -96,7 +96,7 @@ Part of the hash by default; each has a name and can be put on the type's | `byte_order` | Little- or big-endian, for each scalar field. A native array hashes as the byte order it actually has, never as "native": the same array bytes hash the same on every machine, and an array created natively on a big-endian machine is a big-endian array. | | `struct_layout` | The field offsets and the item size of a structured dtype, at every nesting depth. An aligned (padded) dtype and the packed dtype with the same fields differ. | -Never part of the hash, because `.npy` does not keep it. A value read back from a store must +Never part of the hash, because `.npy` does not keep it. A value read back from a database must hash like the value that was written, or every cached run would compute again: - **Strides other than C or Fortran order.** A step slice (`a[:, ::2]`), a reversed view or @@ -109,7 +109,7 @@ hash like the value that was written, or every cached run would compute again: flag produced are kept and count, as `struct_layout`). - **Padding bytes between structured fields.** NumPy does not define their content (an `np.empty` array holds whatever the memory held). They are not part of the value: they are - written to the store as zero, so residual memory never reaches a store that may be shared, + written to the database as zero, so residual memory never reaches a database that may be shared, and they read back as zero. Every field reads back as written. ```python @@ -361,7 +361,7 @@ tensors hash the same only when all of these are equal. | Device, autograd history, `.grad`, position inside a larger buffer | never | | A property that storage does not keep is never in the hash: otherwise a value read back from -a store would hash differently from the value that was written, and every cached result would +a database would hash differently from the value that was written, and every cached result would be computed again. For the same reason the optional properties are those of the *stored* tensor: a step slice hashes like its contiguous clone. @@ -428,7 +428,7 @@ print(caught[0].message) # Function 'fit' output 'weights' returned a torch tensor with autograd history (a grad_fn); Orcapod stored a CPU tensor without autograd history, and the returned tensor is unchanged. Return value.detach().cpu() to avoid this warning. ``` -**What comes back.** A tensor read from a store or a source is a CPU leaf tensor with the +**What comes back.** A tensor read from a database or a source is a CPU leaf tensor with the stored dtype, shape, elements, strides, `requires_grad`, lazy bits and class (a `bfloat16` tensor reads back as `bfloat16`, an `nn.Parameter` as an `nn.Parameter`). Within a single in-memory run, though, the next pod receives the very object the previous pod returned, on its diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 988824420..3ffa69841 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -337,10 +337,10 @@ A job node shows the current run; `node.as_source()` shows everything stored for of that run's inputs, on either orchestrator. The database may hold more under the same node: results of an earlier run over other rows of the source stay stored and are reused when those rows come back, but they are not rows of this run. Before a run in this process -(a job built over a store that already holds results, or a loaded job) every node is empty: +(a job built over a database that already holds results, or a loaded job) every node is empty: `as_table()`, `iter_data()`, `head()` and the preview show no rows, and the preview says the node has not run and points to `run()` and to `as_source()`. A run fills it, also when every -result is read from the store and nothing is computed. A run that raised defines no rows, +result is read from the database and nothing is computed. A run that raised defines no rows, and so does `clear_cache()`; a node whose input changed since its run (an upstream node ran again, or a source was bound again) is stale and empty until it runs again. The stored view, `node.as_source()`, reads every record stored under the node's key, whether or not the node @@ -383,16 +383,16 @@ print("function calls:", len(calls)) version, e.g. `@function_pod(output_keys="bmi", version="v1.0")` (the default is `"v0.0"`; a minor bump such as `"v0.1"` is not enough), or rename the function. -!!! note "When the store cannot be read" +!!! note "When the database cannot be read" A table that does not exist yet reads as empty, and its rows are computed. In a `DeltaTableDatabase`, a table that exists but cannot be read (for example, a data file was deleted or damaged) is never treated as empty: the run stops with - `orcapod.errors.StoreReadError`, which names the + `orcapod.errors.OrcapodDatabaseReadError`, which names the table, the node and row being processed when they are known, and the underlying error, whatever the orchestrator's `error_policy`. Its stored rows are not recomputed and the table is not overwritten, so you can repair the table or restore the file and run again. - A write that fails because the stored table cannot be read raises `StoreReadError` too, - not `StoreWriteError` (see *When the store cannot be written* below). + A write that fails because the stored table cannot be read raises `OrcapodDatabaseReadError` too, + not `OrcapodDatabaseWriteError` (see *When the database cannot be written* below). In a `DeltaTableDatabase`, rows whose column types differ from the stored table's, as Delta stores them (`float64` values for a column stored as `int64`, say), are refused with @@ -406,11 +406,11 @@ print("function calls:", len(calls)) New columns and new struct fields pass this check; whether they are added follows the database's `allow_schema_evolution` setting. In a job run, a refused append fails the - run as a `StoreWriteError` whose cause is the `InputValidationError`. + run as a `OrcapodDatabaseWriteError` whose cause is the `InputValidationError`. When you use the database directly, both errors reach you as themselves, also from `flush()` and from `add_records(..., flush=True)`. When several pending tables fail in one flush, the - `StoreReadError` (or else the first `InputValidationError`) is raised, and each other + `OrcapodDatabaseReadError` (or else the first `InputValidationError`) is raised, and each other failure is attached to it as a note and logged. Failed rows stay queued, so a later flush tries them again. A flush that fails for any other reason raises `RuntimeError`, with the first failure as its cause. @@ -449,7 +449,7 @@ the next run computes only the rest. A function that is in the middle of a row c interrupted, so the cell waits until it returns. If it does not return (a function that loops forever, say), press the interrupt button again: the cell stops waiting at once, with a warning that the function in that node has not returned yet. The run is already cancelled, -so nothing more is written to the store when the function does return. To stop the function +so nothing more is written to the database when the function does return. To stop the function itself, restart the kernel. An `async def` function must not block (with `time.sleep`, a busy loop or another blocking @@ -482,7 +482,7 @@ print(job.nodes["bmi"].as_table().num_rows) ### Reading the run summary `job.run()` (and `await job.run_async()`) returns a `RunSummary`: for each function node, how -many rows were computed, how many were read from the store, and how many failed. At the end +many rows were computed, how many were read from the database, and how many failed. At the end of a notebook cell it is shown as a table; `print` shows the same as text. The job keeps it as `job.last_run`. The results themselves are read from `job.nodes`, as above. @@ -527,7 +527,7 @@ print(summary) `job.last_run.print_traceback("bmi")` prints the full traceback of the node's first failed row, ending in the line of your function that raised. Fix the function and run the job -again: the rows that succeeded before are read from the store, and only the failed ones +again: the rows that succeeded before are read from the database, and only the failed ones are computed. To stop a run at the first failed row instead, see "Stopping at the first failed row" below. @@ -543,13 +543,13 @@ print(summary.ok, summary.failed_rows) ### Loading without computing: `run(load_only=True)` -A job node shows the rows of its current run, so a job built over a store that already holds +A job node shows the rows of its current run, so a job built over a database that already holds its results is empty until it runs. A run that finds every result stored computes nothing, but it does compute the results that are missing. `job.run(load_only=True)` fills the nodes with what is already stored for this run's inputs and never computes: sources are read, operators run on the rows that reach them, each function node takes the stored results of its inputs and skips the inputs that have none, no side effect is delivered and nothing is -written to the store. The summary counts what each node loaded and skipped: +written to the database. The summary counts what each node loaded and skipped: ```python more_mice = mice + [{"subject_id": "mouse_04", "weight": 30.0, "height": 0.12}] @@ -599,14 +599,14 @@ the same for every node of the job. | Filled by a run | that run's rows (`rows`) | holds it | holds it | A node that is held is not run: its rows feed the nodes after it, and nothing is computed, -read from the store, written or delivered for it. So running an unchanged job again does +read from the database, written or delivered for it. So running an unchanged job again does nothing, also on the asynchronous orchestrator, and the summary says so per node: "rowwise: 6 rows: held from the previous run" (`summary.nodes["rowwise"].held`). A load-only run's rows are never held by a full run, nor are the rows of any node below them. A source is read only when a node that runs reads it. A node whose last run left failed rows (with the default `error_policy="continue"`) is not -held either: the next run computes only the failed rows and reads the rest from the store, +held either: the next run computes only the failed rows and reads the rest from the database, so running the job again after fixing a transient failure retries exactly those rows. A node is stale, and runs, when: @@ -674,7 +674,7 @@ first in input order only with `max_concurrency=1`, cancels every row and node s and closes their channels before it raises. A function already running in a worker thread cannot be interrupted: its result is discarded, never stored, and `job.run()` returns once it has finished. When rows of several nodes fail before the run stops, one error is raised and -the others are noted on it (`error.__notes__`): a store error first (see "When the store +the others are noted on it (`error.__notes__`): a database error first (see "When the database cannot be written" below), then a `TypeIdentityError`, and otherwise the error of the node that comes first in the pipeline. A node cancelled by the failure commits the results it wrote before it stops. A run that raised defines no rows: afterwards every node the run ran, @@ -710,11 +710,11 @@ job.run(commit_every_rows=500) # fewer, larger commits `commit_every_rows=1` commits after every row. A node or a cached pod used on its own, outside `job.run()`, also commits after every row. On a `DeltaTableDatabase`, -`commit_every_rows` must be smaller than the store's `batch_size` (default 1000): -the store commits every pending table by itself once one table holds that many pending +`commit_every_rows` must be smaller than the database's `batch_size` (default 1000): +the database commits every pending table by itself once one table holds that many pending rows, which would not keep the order described below, so `job.run()` refuses a larger value -with `ValueError`. Create the store with a larger `batch_size` to commit less often. For the same reason, `job.run()` first commits -whatever such a store already holds pending (writes made without a flush, or the batches of a +with `ValueError`. Create the database with a larger `batch_size` to commit less often. For the same reason, `job.run()` first commits +whatever such a database already holds pending (writes made without a flush, or the batches of a run that failed); if that fails, the run stops before computing anything. **Memory.** A `DeltaTableDatabase` keeps the committed rows of every table it looks rows up @@ -732,12 +732,12 @@ commit are lost, and nothing stored is damaged. Run the job again: it computes o whose results were not committed. A node commits its results before their pipeline records, so a crash between those two commits leaves results stored without pipeline records. Reading the node (`as_table()`, `get_all_records()`) does not show those rows until the next run, -which finds their results in the store, computes nothing for them and writes their pipeline +which finds their results in the database, computes nothing for them and writes their pipeline records. **Reads see the version this process loaded.** A `DeltaTableDatabase` keeps the committed rows of each table it has read in memory and answers lookups from them, extending them with -its own commits. Rows another process commits to the same table are not seen by a store +its own commits. Rows another process commits to the same table are not seen by a database object that has already loaded the table until its own next commit to that table reloads it; a new `DeltaTableDatabase` object reads them at once. This was already the case before batching; only the re-reading per lookup is gone. @@ -763,21 +763,21 @@ An operator node attached in `CacheMode.REPLAY` does not compute: its run replay stored under its key by `CacheMode.LOG` runs, every logged run of that key, not only the latest; before its run it shows nothing, like every node (a narrower replay is ITL-1098). -### When the store cannot be written +### When the database cannot be written -A failed write to the store always fails the run. If a node's results or records cannot be +A failed write to the database always fails the run. If a node's results or records cannot be stored -- the disk is full, the connection drops, or the database cannot hold one of the -output types -- `job.run()` raises `StoreWriteError` (from `orcapod.errors`) at the first +output types -- `job.run()` raises `OrcapodDatabaseWriteError` (from `orcapod.errors`) at the first failed write, on both the synchronous and the asynchronous orchestrator. A row's write that fails names the node, the row being stored and the table, and the database's own exception is chained as its cause. When committing pending writes fails, the error has `flush=True` -and names the node and the store instead of a row and table: a commit holds a batch of rows -(see *When results are committed*), and every view of a store shares its pending writes, so +and names the node and the database instead of a row and table: a commit holds a batch of rows +(see *When results are committed*), and every view of a database shares its pending writes, so the write that failed may belong to any row of the batch or another table. A run that returns normally has stored every result that could be turned into a record; a result rejected before that is a row failure (see below). -With batched commits, a store failure that only committing reveals (an unreadable table, a +With batched commits, a database failure that only committing reveals (an unreadable table, a full disk) surfaces at the node's next commit point, not at the row that caused it: the rows the node computed since its last commit are not stored and are computed again on the next run. `job.run(commit_every_rows=1)` commits after every row, so such a failure surfaces at @@ -787,12 +787,12 @@ This holds whatever `error_policy` the orchestrator was given. `error_policy="co default) skips rows whose *function* raised and keeps going; it never skips a row because it could not be stored. -`StoreWriteError` and `StoreReadError` both derive from `orcapod.errors.StoreError`, so -`except StoreError` handles either. The error you get is always one plain store error, never +`OrcapodDatabaseWriteError` and `OrcapodDatabaseReadError` both derive from `orcapod.errors.OrcapodDatabaseError`, so +`except OrcapodDatabaseError` handles either. The error you get is always one plain database error, never an exception group and never another exception raised in its place. A post-run hook or -observer callback that fails while the store error is on its way out is logged and recorded +observer callback that fails while the database error is on its way out is logged and recorded as a note on the error (`err.__notes__`). When several nodes fail at once on the asynchronous -orchestrator, read and write failures alike, you get the store error of the node that comes +orchestrator, read and write failures alike, you get the database error of the node that comes first in the pipeline, the one the synchronous orchestrator would have stopped at; the other failures are logged and noted on it. A node whose failure the asynchronous orchestrator cancelled before it happened is not reported. @@ -814,10 +814,10 @@ different case: a naive `datetime`, a `str` from a function declared to return ` prepared, before anything is written, and it is a *row failure*: it is handled exactly like a function that raised on that row. The observer gets `on_data_crash`, the failure is logged, and `error_policy` decides -- `"continue"` skips the row, `"fail_fast"` raises the error -the conversion raised. It is not a `StoreWriteError`, because the store is healthy and every +the conversion raised. It is not a `OrcapodDatabaseWriteError`, because the database is healthy and every re-run would fail at the same row. Both orchestrators handle these rows the same way. -This is the contract for now. An opt-in mode that tolerates store failures may be added +This is the contract for now. An opt-in mode that tolerates database failures may be added later. ### Side effects in a job: delivered by a run, never by a read @@ -860,7 +860,7 @@ print(len(sent)) - The rows a read returns are those the last run in this process passed on: under `on_error="log"`, a row whose delivery failed is left out with `drop_on_failure=True` (the default) and kept with `drop_on_failure=False`, as for the nodes downstream. -- Before any run in this process (a new job over a store that already records deliveries), +- Before any run in this process (a new job over a database that already records deliveries), after a run that raised, or once a node upstream has changed since the run, a read returns no rows and `head()` says that `run()` delivers. Run the job to get the rows. - Outside a job, the stream a side-effect pod returns keeps nothing, so every consuming read @@ -870,7 +870,7 @@ print(len(sent)) ### Saving and loading a job -`job.save(path)` writes the job to a JSON file: the graph, the store's location and how each +`job.save(path)` writes the job to a JSON file: the graph, the database's location and how each source was configured. `PipelineJob.load(path)` reads it back, for example in a later Python session, so that you can read the results the job stored. The file also records `orcapod_version`, the Orcapod release that wrote it (the same text as `orcapod.__version__`; @@ -909,9 +909,9 @@ print(loaded.nodes["bmi"].as_source().as_table().sort_by("subject_id").to_pandas ``` A loaded job is for reading stored results. Its nodes have not run in this process, so -they are empty; each node's stored view, `node.as_source()`, reads what the store holds. A +they are empty; each node's stored view, `node.as_source()`, reads what the database holds. A loaded job cannot compute: function pods are not saved with it, so `loaded.run()` raises. -`loaded.run(load_only=True)` fills its nodes from the store instead, as for any job (see +`loaded.run(load_only=True)` fills its nodes from the database instead, as for any job (see *Loading without computing* above): the sources are read (an in-memory source must be bound first, since its data is not saved), the rebuilt operators run, and each function node takes the results stored for those inputs. Nodes below a source that is not available stay empty. @@ -926,14 +926,14 @@ print(loaded.nodes["bmi"].as_table().num_rows, summary.nodes["bmi"].not_stored) To compute again, build the job from your code as before. -The store is the source of truth: a loaded job's stored views show what the store holds for +The database is the source of truth: a loaded job's stored views show what the database holds for it. The job file names each run's rows by the content hash the run stored them under, so a loaded node's stored view reads exactly what the same node built from your code would read -from that store, and no rows under that key is an empty result, not an error. +from that database, and no rows under that key is an empty result, not an error. - A function node's `as_source()` and `get_all_records()` return the rows stored under the saved run's key, with the types they were stored with: those of every run of that node - over the same sources, not only the saved run's inputs. Rows come in the order the store + over the same sources, not only the saved run's inputs. Rows come in the order the database returns them; sort them if order matters. The node itself (`as_table()`, `iter_data()`, `head()`) is empty: it has not run in this process. - Operators (a join, a filter) store outputs only when caching is enabled, under `odb_v1`. @@ -941,7 +941,7 @@ from that store, and no rows under that key is an empty result, not an error. loaded operator node in `CacheMode.REPLAY` to read the rows its run stored: `loaded.nodes["joined"].attach_databases(pipeline_database=store.at(""), cache_mode=CacheMode.REPLAY)`, then read its stored view, `as_source()`, which reads - exactly as the same node built from your code would; when the store holds no rows under + exactly as the same node built from your code would; when the database holds no rows under the node's key (no table, or none of its rows), the answer is an empty result, as for that node. A loaded operator node itself is empty, like every node that has not run. - A side-effect node is restored read-only, in its place in the graph, with its label, @@ -961,7 +961,7 @@ from that store, and no rows under that key is an empty result, not an error. carry. Whether a file has these records is decided by the records themselves, not by the file's format version: a file without them (format `0.1.0`, or a `0.2.0` file written before they existed) still loads, but lossy: its schemas read every column as non-null, - and a loaded operator node cannot name its run's rows. When the store holds none under the + and a loaded operator node cannot name its run's rows. When the database holds none under the content hash the loaded job computes, its stored view (`as_source()`, `get_all_records()`) raises and says why, rather than presenting stored rows as none. An empty record is not a missing one: it says that no column may be null. Saving such a loaded job again does @@ -983,14 +983,14 @@ from that store, and no rows under that key is an empty result, not an error. stream and bind the stored source. A pipeline built directly over a raw `ArrowTableStream` is refused the same way: wrap its table as a source, `ArrowTableSource(stream.as_table(), tag_columns=[...])`. -- The results are read from the store recorded in the file. Pass - `PipelineJob.load(path, database=...)` if the store has moved. +- The results are read from the database recorded in the file. Pass + `PipelineJob.load(path, database=...)` if the database has moved. - The job file may also hold an optional, advisory section, `run.expected_rows`: for each - node, how many rows the store held under its run's key when the job was saved (written + node, how many rows the database held under its run's key when the job was saved (written only when that is certain: after a run that finished). Reading a loaded node's stored - view always returns what the store holds; when the store holds another number of rows, - the read issues one `StoreContentWarning` per node, naming the node, both numbers and the store: - the store may not be the one the run wrote, or it may have changed since. Deleting the + view always returns what the database holds; when the database holds another number of rows, + the read issues one `OrcapodDatabaseContentWarning` per node, naming the node, both numbers and the database: + the database may not be the one the run wrote, or it may have changed since. Deleting the section from the file changes nothing but this check. Logical types in saved schema text carry their complete identifier, written as @@ -1017,7 +1017,7 @@ Tables written before v0.3 have no such column and are stored at an older table (`pdb_v1`, `rdb_v1`). Orcapod **does not read them as results**: it does not refuse the store, and it does not reuse those rows. -- The first run of v0.3 on a store written by v0.2 computes every input again and stores the +- The first run of v0.3 on a database written by v0.2 computes every input again and stores the rows in its own tables (`pdb_v2`, `rdb_v2`). - Reading a node (`as_table()`, `iter_data()`, `get_all_records()`) returns the rows of the current data context only: nothing before that first run, one row per input after it. @@ -1039,11 +1039,11 @@ store, and it does not reuse those rows. that take the invocation context (`ctx_arg`). A side effect is an external action, and completions stored by v0.2 cannot be matched with current inputs, so running the node would do everything a second time. Instead the node raises - `SideEffectLogDataContextError` (from `orcapod.errors`, a `StoreError`), naming the + `SideEffectLogDataContextError` (from `orcapod.errors`, a `OrcapodDatabaseError`), naming the tables, before it does anything. Every older table under the node's function or pod name - in the job counts. Keep using the version that wrote them. A fresh store is not a safe + in the job counts. Keep using the version that wrote them. A fresh database is not a safe way around the error: it holds no completions, so on it the node performs every side - effect again; use one only when a repeated delivery is acceptable. If the store cannot + effect again; use one only when a repeated delivery is acceptable. If the database cannot list its tables, the node warns that it could not check, and runs. Each result row also records the full version string of the function that produced it, in diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index 317fc5f11..74891b27a 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -273,7 +273,7 @@ A stream or node records that its results are current when it has read its input So after `job.run()` on either orchestrator, sync or async, every node shows the rows of that run: `head()`, `as_table()` and `iter_data()` serve them from memory, without reading the store. Before its run in this process a job node holds nothing and shows no rows, whatever -its store holds; `node.as_source()` reads the store. Likewise, once a chain of streams such as `g.pod(f.pod(source))` has been read to the +its database holds; `node.as_source()` reads the database. Likewise, once a chain of streams such as `g.pod(f.pod(source))` has been read to the end, reading it again computes nothing. A change upstream (`clear_cache()`) makes every stream after it out of date, and its next read computes it again. Two reads of one stream open at the same time, or a read that picks up one left partway, share diff --git a/docs/concepts/your-own-data-types.md b/docs/concepts/your-own-data-types.md index eb852118c..f535c0e33 100644 --- a/docs/concepts/your-own-data-types.md +++ b/docs/concepts/your-own-data-types.md @@ -19,7 +19,7 @@ has two kinds of class, and you choose the kind where you define the class. | Reading stored values without the class | works: the class is rebuilt from stored metadata | `ImportError` | | What a pod is promised | the fields | the class itself, with its methods and validators | -**Use `@op.dataclass`** for data you pass between pods, keep in a store and work on in +**Use `@op.dataclass`** for data you pass between pods, keep in a database and work on in notebooks: records, measurements, parameters. It is the kind to use while a class is still being edited. diff --git a/docs/metamorphic/plans/2026-07-08-itl-507-ephemeral-coverage-tests.md b/docs/metamorphic/plans/2026-07-08-itl-507-ephemeral-coverage-tests.md index 42c0c1a6a..9ec30814d 100644 --- a/docs/metamorphic/plans/2026-07-08-itl-507-ephemeral-coverage-tests.md +++ b/docs/metamorphic/plans/2026-07-08-itl-507-ephemeral-coverage-tests.md @@ -15,7 +15,7 @@ | Task | Lines in `function_node.py` | Code path | |------|----------------------------|-----------| | 1 | 1311–1342 | `_async_process_data_internal` ephemeral branch (happy path) | -| 2 | 1318–1324 | `_async_process_data_internal` RuntimeError guard (no store) | +| 2 | 1318–1324 | `_async_process_data_internal` RuntimeError guard (no database) | | 3 | 1430, 1436–1439 | `add_pipeline_record` early return when duplicate detected (`skip_cache_lookup=False`) | | 4 | 1611, 1637–1638 | `_fetch_joined_records` no-DB guard + backward-compat IS_EPHEMERAL_COL-absent branch | @@ -70,7 +70,7 @@ Covers `_async_process_data_internal` lines 1311–1342: the `if ephemeral_resul class TestAsyncEphemeralExecution: @pytest.mark.asyncio async def test_async_execute_ephemeral_happy_path(self): - """async_execute with is_result_ephemeral=True writes to ephemeral store and emits results.""" + """async_execute with is_result_ephemeral=True writes to ephemeral database and emits results.""" stream = _make_stream([{"id": 0, "x": 5}, {"id": 1, "x": 10}]) pipeline_db = InMemoryArrowDatabase() ephemeral_store = InMemoryArrowDatabase() @@ -99,7 +99,7 @@ Covers `_async_process_data_internal` lines 1311–1342: the `if ephemeral_resul values = sorted(data.as_dict()["result"] for _, data in results) assert values == [10, 20] - # Result records must be in the ephemeral store, not the persistent store + # Result records must be in the ephemeral database, not the persistent store eph_records = ephemeral_store.get_all_records( node._ephemeral_cached_pod.record_path, ) @@ -108,7 +108,7 @@ Covers `_async_process_data_internal` lines 1311–1342: the `if ephemeral_resul @pytest.mark.asyncio async def test_async_process_data_internal_raises_when_no_store(self): - """_async_process_data_internal raises RuntimeError when is_result_ephemeral=True but no store.""" + """_async_process_data_internal raises RuntimeError when is_result_ephemeral=True but no database.""" stream = _make_stream([{"id": 0, "x": 5}]) pipeline_db = InMemoryArrowDatabase() diff --git a/docs/metamorphic/plans/2026-07-08-itl-513-stale-entry-cleanup.md b/docs/metamorphic/plans/2026-07-08-itl-513-stale-entry-cleanup.md index e629aaeff..7bca6d211 100644 --- a/docs/metamorphic/plans/2026-07-08-itl-513-stale-entry-cleanup.md +++ b/docs/metamorphic/plans/2026-07-08-itl-513-stale-entry-cleanup.md @@ -94,7 +94,7 @@ Append the following method to the `TestEphemeralWritePath` class (after the exi stream = _make_stream([{"id": 0, "x": 10}]) pipeline_db = InMemoryArrowDatabase() - # Session 1: compute with ephemeral store 1 → pipeline DB gets index-0 record + # Session 1: compute with ephemeral database 1 → pipeline DB gets index-0 record pf = PythonDataFunction(counting_double, output_keys="result") cfg = NodeConfig(is_result_ephemeral=True) pod = FunctionPod(pf, node_config=cfg) @@ -107,7 +107,7 @@ Append the following method to the `TestEphemeralWritePath` class (after the exi node1.execute(stream) assert call_count["n"] == 1 - # Session 2: fresh ephemeral store → cross-session miss → recompute → index-1 record written + # Session 2: fresh ephemeral database → cross-session miss → recompute → index-1 record written pf2 = PythonDataFunction(counting_double, output_keys="result") pod2 = FunctionPod(pf2, node_config=cfg) ephemeral2 = InMemoryArrowDatabase() diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index c5f112b78..65b53215f 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -646,7 +646,7 @@ result. Within one dict or set the two compare equal, so they cannot both be pre **Arrow tables are hashed exactly as stored.** `StarfixArrowHasher` never sorts list contents. A `list>` column that arrives already in Arrow form (an -`ArrowTableSource`, a Delta table, rows read back from a store) is read by Python as a +`ArrowTableSource`, a Delta table, rows read back from a database) is read by Python as a `dict`, but at the Arrow level it is an ordered list and its stored order is part of its hash. A `list[T]` value is never sorted: its order is part of its identity. (Tables written by `0.2.0`, which stored dicts in insertion order, are not read by v0.3 at all.) @@ -1339,7 +1339,7 @@ result tables. Either raises `SideEffectLogDataContextError`, naming the tables, doing anything when it finds a table at an older layout anywhere under its name, or completions of another data context in its current tables. Because every older table under the name counts, an unrelated pod of the same name in the same job raises the error too; -the error names every table so that you can tell. When the store cannot list its tables, +the error names every table so that you can tell. When the database cannot list its tables, the node logs one warning saying that it could not look for older completions, and proceeds. @@ -1467,7 +1467,7 @@ column back, so the value is refused (see "What is refused" below). **The hash is strict.** Everything about a value that can be changed is part of its identity (developer, November 11 and 12, 2026), with one exception: **a property storage -does not restore never counts**, so that a value read back from a store hashes exactly like +does not restore never counts**, so that a value read back from a database hashes exactly like the value that was written. `PANDAS_IDENTITY_PROPERTIES` in `orcapod/hashing/pandas_value_hash.py` is the one list in code of every property below. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index b0e5f3d94..6d33e1a0c 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2,7 +2,7 @@ ## A script finishes its work but the process never exits -**Symptom.** A Python script that used a Delta store prints all of its output and then hangs, +**Symptom.** A Python script that used a Delta database prints all of its output and then hangs, and has to be killed. On Linux with Python 3.11 or 3.12 the process may instead end with `terminate called without an active exception` and exit code 134 (SIGABRT). Notebooks are not affected in the same way, because the kernel keeps running. @@ -25,7 +25,7 @@ Reading a Delta table yourself with `deltalake.DeltaTable(...).to_pyarrow_table( - Upgrade pyarrow once a release includes the fix ([apache/arrow#54467](https://github.com/apache/arrow/pull/54467)). -- In a script that reads a cloud Delta store, end it by flushing your outputs and calling +- In a script that reads a cloud Delta database, end it by flushing your outputs and calling `os._exit(0)`. That skips interpreter shutdown, where the hang happens. It also skips `atexit` handlers and other cleanup, so close files and flush stores first. - When reading local Delta tables directly with deltalake, pass a native filesystem. Resolve diff --git a/src/orcapod/contexts/stored.py b/src/orcapod/contexts/stored.py index adaa7d745..e0ae1a58f 100644 --- a/src/orcapod/contexts/stored.py +++ b/src/orcapod/contexts/stored.py @@ -23,7 +23,7 @@ Example: To list the data contexts a stored table holds, call - ``stored_data_contexts(store, record_path)`` (from ``orcapod.contexts``); it + ``stored_data_contexts(database, record_path)`` (from ``orcapod.contexts``); it returns, for example, ``{"std:v1:default": 3}``. """ @@ -38,7 +38,7 @@ from orcapod.errors import ( InputValidationError, SideEffectLogDataContextError, - StoreContentWarning, + OrcapodDatabaseContentWarning, TableListingNotSupportedError, ) from orcapod.system_constants import ( @@ -77,7 +77,7 @@ # Last path components that mark a table as written before v0.3 wherever it is found: # the older layout names, and the beginnings of the hash components that ended a path -# when a table had no layout component (also as a connector store spells them). +# when a table had no layout component (also as a connector database spells them). _OLD_LAYOUTS: frozenset[str] = frozenset( OLD_PIPELINE_DB_SCHEMA_VERSIONS + OLD_RESULT_DB_SCHEMA_VERSIONS @@ -174,7 +174,7 @@ def report_rows_of_other_data_contexts( """Warn that a read left out stored rows written under another data context. Args: - record_path: Path of the stored table from the root of the store. + record_path: Path of the stored table from the root of the database. skipped: Number of rows of the table left out of the read. """ logger.warning( @@ -187,62 +187,62 @@ def report_rows_of_other_data_contexts( ) -class _DescribedStore(Protocol): - """The part of ``ArrowDatabaseProtocol`` that names a store (no import cycle).""" +class _DescribedDatabase(Protocol): + """The part of ``ArrowDatabaseProtocol`` that names a database (no import cycle).""" @property def base_path(self) -> tuple[str, ...]: - """Path of this store's view from the root of the underlying store.""" + """Path of this database's view from the root of the underlying database.""" ... def to_config(self) -> dict[str, Any]: - """Return the store's configuration, with its location when it has one.""" + """Return the database's configuration, with its location when it has one.""" ... -def describe_store_table( - database: _DescribedStore, record_path: tuple[str, ...] +def describe_database_table( + database: _DescribedDatabase, record_path: tuple[str, ...] ) -> str: - """Name a stored table for a message: the store's location and the table's path. + """Name a stored table for a message: the database's location and the table's path. Args: - database: The store, or a view of it. + database: The database, or a view of it. record_path: Path of the table relative to ``database``. Returns: - For example ``"store at '/data/store' (table 'job/node/...')"``. + For example ``"database at '/data/store' (table 'job/node/...')"``. """ store_config = database.to_config() location = store_config.get("root_uri") - store = ( - f"store at {location!r}" + description = ( + f"database at {location!r}" if location - else f"{store_config.get('type', 'attached')} store" + else f"{store_config.get('type', 'attached')} database" ) table = "/".join(database.base_path + record_path) - return f"{store} (table {table!r})" + return f"{description} (table {table!r})" def report_unexpected_row_count( node: str, location: str, expected: int, found: int ) -> None: - """Warn that a loaded node's store holds another number of rows than its run's. + """Warn that a loaded node's database holds another number of rows than its run's. - The store is the source of truth: the read returns what it holds. The warning says + The database is the source of truth: the read returns what it holds. The warning says that it differs from what the job file recorded (ITL-1043). Args: node: The node, as messages name it (``"FunctionJobNode 'dbl'"``). - location: The store and table, from ``describe_store_table``. + location: The database and table, from ``describe_database_table``. expected: The rows the job file records for the node's run. - found: The rows the store holds under the node's run key. + found: The rows the database holds under the node's run key. """ warnings.warn( - StoreContentWarning( + OrcapodDatabaseContentWarning( f"{node}: the job file records {expected} " f"{'row' if expected == 1 else 'rows'} for its run, but the {location} " - f"holds {found} under the run's key. The read returns what the store " - "holds; the store may not be the one the run wrote, or it may have " + f"holds {found} under the run's key. The read returns what the database " + "holds; the database may not be the one the run wrote, or it may have " "changed since." ), stacklevel=3, @@ -253,7 +253,7 @@ def report_old_layout_rows(record_path: tuple[str, ...], rows: int) -> None: """Warn that a table written at an older table layout was found and is not used. Args: - record_path: Path of the old-layout table from the root of the store. + record_path: Path of the old-layout table from the root of the database. rows: Number of rows the table holds. """ logger.warning( @@ -273,8 +273,8 @@ def report_tables_under_name( """Warn that tables written before v0.3 were found under a function's name. Args: - name_path: Path, from the root of the store, up to the function's name. - record_paths: Paths of the tables found, from the root of the store. + name_path: Path, from the root of the database, up to the function's name. + record_paths: Paths of the tables found, from the root of the database. """ logger.warning( "Found %d %s written by an earlier version of Orcapod under %r (older table " @@ -290,19 +290,19 @@ def report_tables_under_name( def report_unclaimed_tables( - store_path: tuple[str, ...], record_paths: Collection[tuple[str, ...]] + database_path: tuple[str, ...], record_paths: Collection[tuple[str, ...]] ) -> None: """Warn, once for a job, about the pre-v0.3 tables none of its nodes reports. Args: - store_path: Path of the job's tables from the root of the store. - record_paths: Paths of the tables found, from the root of the store. + database_path: Path of the job's tables from the root of the database. + record_paths: Paths of the tables found, from the root of the database. """ logger.warning( - "Store %r holds %d more %s written by an earlier version of Orcapod " + "Database %r holds %d more %s written by an earlier version of Orcapod " "(operator caches, cached sources, or nodes not in this job): %s. They are " "not used and are left unchanged; they can be migrated later.", - "/".join(store_path), + "/".join(database_path), len(record_paths), "table" if len(record_paths) == 1 else "tables", _spelled_out(record_paths), @@ -315,14 +315,14 @@ def report_unlisted_side_effect_tables( """Warn that a side-effecting node could not look for completions logged before v0.3. Args: - name_paths: Paths, from the root of the store, up to the node's function or + name_paths: Paths, from the root of the database, up to the node's function or pod name: where the tables would have been looked for. - reason: Why the store could not list its tables. + reason: Why the database could not list its tables. """ logger.warning( - "The store cannot list its tables (%s), so the node that performs side " + "The database cannot list its tables (%s), so the node that performs side " "effects could not check for completions logged before Orcapod v0.3 under " - "%s. If the store holds such tables, every side effect is delivered again.", + "%s. If the database holds such tables, every side effect is delivered again.", reason, " and ".join(repr("/".join(path)) for path in name_paths), ) @@ -341,7 +341,7 @@ class _RecordStore(Protocol): @property def base_path(self) -> tuple[str, ...]: - """Path of this store's view from the root of the underlying store.""" + """Path of this database's view from the root of the underlying database.""" ... def table_exists(self, record_path: tuple[str, ...]) -> bool: @@ -351,7 +351,7 @@ def table_exists(self, record_path: tuple[str, ...]) -> bool: record_path: Path of the table, relative to ``base_path``. Returns: - True if the store holds a table at that path. + True if the database holds a table at that path. """ ... @@ -375,14 +375,14 @@ class StaleTable(NamedTuple): """Stored rows a node will not use: an old-layout table, or rows of another data context. Attributes: - record_path: Path of the table from the root of the store. + record_path: Path of the table from the root of the database. data_context: Key of the data context that wrote the rows. rows: Number of such rows in the table; ``None`` for a table found by - listing the store, which is not read. + listing the database, which is not read. old_layout: True for a table at an older layout, False for rows of another data context in a table at the current layout. - found_under: For a table found by listing the store, the path (from the - root of the store) under which the tables were listed; ``None`` for a + found_under: For a table found by listing the database, the path (from the + root of the database) under which the tables were listed; ``None`` for a table found at a computed path. """ @@ -397,9 +397,9 @@ class UnlistedTables(NamedTuple): """A place where tables written before v0.3 could not be looked for. Attributes: - name_path: Path, from the root of the store, under which the tables would + name_path: Path, from the root of the database, under which the tables would have been listed. - reason: Why the store could not list its tables. + reason: Why the database could not list its tables. """ name_path: tuple[str, ...] @@ -412,7 +412,7 @@ class StaleTables: Attributes: tables: The tables found, each once. - unlisted: The places that could not be searched because the store cannot + unlisted: The places that could not be searched because the database cannot list its tables. """ @@ -445,7 +445,7 @@ def tables_written_before_v03( every table under it belongs to a node of that name. Args: - database: The store to list. + database: The database to list. name_path: Leading path components, relative to the database's ``base_path``. Returns: @@ -453,7 +453,7 @@ def tables_written_before_v03( ``name_path`` whose last component is not a current layout, sorted. Raises: - TableListingNotSupportedError: If the store cannot list its tables. + TableListingNotSupportedError: If the database cannot list its tables. """ return [ path @@ -465,20 +465,20 @@ def tables_written_before_v03( def marked_tables_written_before_v03(database: object) -> list[tuple[str, ...]]: """List the tables of a whole database view that carry a mark of an older version. - Unlike under the name of one function, a store holds other tables beside node + Unlike under the name of one function, a database holds other tables beside node tables (observer logs, tables of the user), so here a table counts only when its last path component is an older layout (``pdb_v1``, ``rdb_v1``, ``tdb_v1``) or the hash component that ended a path before tables had a layout component (``schema:...``, ``instance:...``, ``node:...``). Args: - database: The store to list, positioned at the tables of one job. + database: The database to list, positioned at the tables of one job. Returns: The paths of those tables, relative to the database's ``base_path``, sorted. Raises: - TableListingNotSupportedError: If the store cannot list its tables. + TableListingNotSupportedError: If the database cannot list its tables. """ return [ path @@ -497,7 +497,7 @@ def unclaimed_tables_written_before_v03( that survives a change of hashes, and tables of nodes the job no longer has. Args: - database: The store, positioned at the tables of one job. + database: The database, positioned at the tables of one job. claimed: The name paths, relative to ``database``, that nodes of the job account for themselves. @@ -506,7 +506,7 @@ def unclaimed_tables_written_before_v03( ``marked_tables_written_before_v03`` finds outside every claimed name path. Raises: - TableListingNotSupportedError: If the store cannot list its tables. + TableListingNotSupportedError: If the database cannot list its tables. """ return [ path @@ -518,10 +518,10 @@ def unclaimed_tables_written_before_v03( def _list_table_paths( database: object, prefix: tuple[str, ...] ) -> list[tuple[str, ...]]: - """Call ``list_table_paths`` of a store that may not have the method. + """Call ``list_table_paths`` of a database that may not have the method. Raises: - TableListingNotSupportedError: If the store has no such method or cannot + TableListingNotSupportedError: If the database has no such method or cannot list its tables. """ list_table_paths = getattr(database, "list_table_paths", None) @@ -544,12 +544,12 @@ def find_stale_tables( - at ``old_paths``, where an earlier version stored the table if no hash in the path changed since (the rows are counted, which reads the table); - - under ``name_path``, by listing the store, for tables of an earlier version + - under ``name_path``, by listing the database, for tables of an earlier version wherever their hashes put them (ITL-954; not read, so not counted); - in the current table, for rows of another data context. Args: - database: The store the paths belong to. + database: The database the paths belong to. old_paths: Paths at which an earlier version of Orcapod stored this table. current: The table's current path and the current data context key, to also look for rows of another data context in the current table (this reads @@ -560,7 +560,7 @@ def find_stale_tables( Returns: The old-layout tables found (those at ``old_paths`` first), then one entry per other data context found in the current table (in key order); and - ``name_path`` under ``unlisted`` if the store could not list its tables. + ``name_path`` under ``unlisted`` if the database could not list its tables. """ found = StaleTables() for path in old_paths: @@ -617,7 +617,7 @@ def account_for_stale_tables( context. It fails before doing anything. Every old-layout table under the node's name counts, so a table of another function or pod with the same name in the job, or of another node of the same function, fails it too: the error - names the tables. Where the store cannot list its tables, the node warns and + names the tables. Where the database cannot list its tables, the node warns and proceeds (developer, November 11, 2026). Args: @@ -665,7 +665,7 @@ def stored_data_contexts( before Orcapod v0.3; all its rows are counted under ``std:v0.1:default``. Args: - database: The store, positioned so that ``record_path`` names the table. + database: The database, positioned so that ``record_path`` names the table. record_path: Path of the stored table, including its layout component. Returns: diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index 2fcce1538..ea1fea3bf 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -78,7 +78,7 @@ def result_database(self) -> ArrowDatabaseProtocol: @property def record_path(self) -> tuple[str, ...]: - """Return the path to the cached records in the result store (versioned).""" + """Return the path to the cached records in the result database (versioned).""" return self._cache.record_path def stale_result_tables(self, include_current: bool = True) -> StaleTables: @@ -402,7 +402,7 @@ async def _async_invoke_with_hooks( def get_all_cached_outputs( self, include_system_columns: bool = False ) -> "pa.Table | None": - """Return all cached records from the result store for this pod.""" + """Return all cached records from the result database for this pod.""" return self._cache.get_all_records( self.data_context_key, include_system_columns=include_system_columns, @@ -486,7 +486,7 @@ def head( ) -> StreamView: """Return the first computed rows, with the columns ``as_table`` gives. - The pod's result store is not looked up: that needs the input rows' hashes, + The pod's result database is not looked up: that needs the input rows' hashes, i.e. reading the input. See ``FunctionPodStream.head``. """ head = super().head(n, columns=columns, all_info=all_info) diff --git a/src/orcapod/core/data_function.py b/src/orcapod/core/data_function.py index 3f085e0ef..83c4a7fd1 100644 --- a/src/orcapod/core/data_function.py +++ b/src/orcapod/core/data_function.py @@ -1135,7 +1135,7 @@ def set_auto_flush(self, on: bool = True) -> None: @property def record_path(self) -> tuple[str, ...]: - """Return the path to the cached records in the result store (versioned).""" + """Return the path to the cached records in the result database (versioned).""" return self._cache.record_path def call( @@ -1266,7 +1266,7 @@ def record_data( def get_all_cached_outputs( self, include_system_columns: bool = False ) -> "pa.Table | None": - """Return all cached records from the result store for this function. + """Return all cached records from the result database for this function. Args: include_system_columns: If True, include system columns diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 8f5e7bf28..752c3bb39 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -28,8 +28,8 @@ StreamView, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER -from orcapod.errors import StoreError, TypeIdentityError -from orcapod.utils.store_errors import keep_store_error, note_callback_failure +from orcapod.errors import OrcapodDatabaseError, TypeIdentityError +from orcapod.utils.database_errors import keep_database_error, note_callback_failure from orcapod.protocols.core_protocols import ( ArgumentGroup, FunctionPodProtocol, @@ -358,18 +358,20 @@ def add_post_run_hook(self, hook: PostRunHook) -> None: def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: """Fire all registered hooks with payload in registration order. - When the invocation failed with a ``StoreError`` or a ``TypeIdentityError`` + When the invocation failed with a ``OrcapodDatabaseError`` or a ``TypeIdentityError`` (both fail the run under every error policy), a fail-loud hook that - raises does not replace it: its failure is logged and noted on the store + raises does not replace it: its failure is logged and noted on the database error, the remaining hooks are skipped as for any fail-loud failure, and the - caller re-raises the store error. + caller re-raises the database error. Args: payload: The post-run payload to pass to each hook. """ error = payload.stats.error - store_error = ( - error if isinstance(error, (StoreError, TypeIdentityError)) else None + database_error = ( + error + if isinstance(error, (OrcapodDatabaseError, TypeIdentityError)) + else None ) for hook in self._post_run_hooks: fn = hook.fn if isinstance(hook, HookConfig) else hook @@ -378,8 +380,10 @@ def _fire_post_run_hooks(self, payload: PostRunPayload) -> None: fn(payload) except Exception as exc: if on_error == "raise": - if store_error is not None: - note_callback_failure(store_error, f"Post-run hook {fn!r}", exc) + if database_error is not None: + note_callback_failure( + database_error, f"Post-run hook {fn!r}", exc + ) return raise logger.warning( @@ -619,7 +623,7 @@ async def async_execute( Raises: Exception: Under ``"fail_fast"``, the exception of the first row whose function raised, inside the ``ExceptionGroup`` of the pod's task group. - StoreError: A failed store read or write, whatever ``error_policy`` is. + OrcapodDatabaseError: A failed database read or write, whatever ``error_policy`` is. """ from orcapod.pipeline.observer import NoOpObserver @@ -642,17 +646,17 @@ async def process_one(tag: TagProtocol, data: DataProtocol) -> None: out_tag, result_data = await self._async_invoke_with_hooks( tag, data, logger=pkt_logger, run_id=run_id ) - except StoreError as exc: - # A failed store read or write fails the run; only rows whose + except OrcapodDatabaseError as exc: + # A failed database read or write fails the run; only rows whose # function raised are skipped. exc.add_context(tag=tag.as_dict()) - with keep_store_error(exc, "Observer on_data_crash"): + with keep_database_error(exc, "Observer on_data_crash"): obs.on_data_crash(pod_label, tag, data, exc) raise except TypeIdentityError as exc: # A stored row that cannot be decoded as its class is never # skipped as a row failure (ITL-881). - with keep_store_error(exc, "Observer on_data_crash"): + with keep_database_error(exc, "Observer on_data_crash"): obs.on_data_crash(pod_label, tag, data, exc) raise except Exception as exc: @@ -1505,7 +1509,7 @@ def head( After a complete computation (``as_table``, or iteration to the end) the rows are the first of the result. After an iteration that stopped part way they are the rows computed so far (``PARTIAL``). Before any computation, or once an input - changed, there are none. A ``CachedFunctionPod``'s result store is not looked + changed, there are none. A ``CachedFunctionPod``'s result database is not looked up: that needs the input rows' hashes, i.e. reading the input. With ``sort_by_tags`` (or ``all_info``) the rows held are sorted by tag and the first ``n`` of those returned: after a complete computation that is diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 5dce998f2..0564bdd4d 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -52,25 +52,25 @@ EphemeralResultMissingError, InputValidationError, PipelineJobRequiredError, - StoreError, + OrcapodDatabaseError, TypeIdentityError, - StoreReadError, - StoreWriteError, + OrcapodDatabaseReadError, + OrcapodDatabaseWriteError, ) from orcapod.contexts.stored import ( StaleTables, keep_rows_of_data_context, require_same_data_context, account_for_stale_tables, - describe_store_table, + describe_database_table, find_stale_tables, report_rows_of_other_data_contexts, report_unexpected_row_count, ) -from orcapod.utils.store_errors import ( +from orcapod.utils.database_errors import ( commit_best_effort, - keep_store_error, - store_error_of, + keep_database_error, + database_error_of, ) from orcapod.protocols.core_protocols import ( FunctionPodProtocol, @@ -785,7 +785,7 @@ def _head_of_pairs( """Return the first ``n`` of ``pairs`` as this node's head. A ``None`` output (a filtered input) is not a row. An ``EmptyData`` placeholder - (a result kept in an ephemeral store that is gone) has no values to show: it is + (a result kept in an ephemeral database that is gone) has no values to show: it is left out and counted in the note. """ rows = [ @@ -809,7 +809,7 @@ def _head_of_pairs( if missing: note = ( f"{missing:,} {'result' if missing == 1 else 'results'} not shown: " - "kept in an ephemeral store that is gone" + "kept in an ephemeral database that is gone" ) return StreamView.of_rows( self, @@ -829,15 +829,15 @@ def __repr__(self) -> str: ) @abstractmethod - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: - """Assign or remove the ephemeral result store for this node. + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: + """Assign or remove the ephemeral result database for this node. ``FunctionJobNode`` overrides this with a real implementation that routes - new computation results to ``store`` instead of the persistent result + new computation results to ``database`` instead of the persistent result database. ``FunctionNode`` (blueprint) provides a no-op override. Args: - store: An ``ArrowDatabaseProtocol`` instance to attach, or ``None`` + database: An ``ArrowDatabaseProtocol`` instance to attach, or ``None`` to detach. """ @@ -879,7 +879,7 @@ def __init__( self._cached_output_table: "pa.Table | None" = None self._cached_content_hash_column: "pa.Array | None" = None - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: """No-op — blueprint nodes carry no database references.""" # ------------------------------------------------------------------ @@ -1096,7 +1096,7 @@ def __init__( @property def record_path(self) -> tuple[str, ...]: - """Path to cached records in the result store (versioned). + """Path to cached records in the result database (versioned). Returns the v1 schema path: ``_record_path + (RESULT_DB_SCHEMA_VERSION,)``. Matches ``CachedFunctionPod.record_path`` semantics so that stub nodes @@ -1121,7 +1121,7 @@ class _JoinedRecords(NamedTuple): derive tag keys in the CACHE_ONLY fallback path. empty_data_tokens: Mapping from ``base_entry_id`` bytes to an ``EmptyData`` token for each ephemeral tag row whose result - was not found in either store (cross-session miss). Empty dict + was not found in either database (cross-session miss). Empty dict when no ephemeral misses occurred. empty_taginfo_rows: Mapping from ``base_entry_id`` bytes to the raw taginfo row dict for each entry in ``empty_data_tokens``. Used @@ -1155,7 +1155,7 @@ class FunctionJobNode(RunHoldingNode, FunctionNodeBase): _stale_tables_checked: bool = False # Set once the rows of the current tables were counted by data context. _current_tables_checked: bool = False - # Set once a store that cannot list its tables was reported. + # Set once a database that cannot list its tables was reported. _unlisted_store_reported: bool = False # Base entry ids of the inputs of the last completed run in this process, each with # its number of inputs; None when no run completed (ITL-1093). A class default so @@ -1203,8 +1203,8 @@ def __init__( self._pipeline_database: ArrowDatabaseProtocol | None = None self._cached_function_pod: CachedFunctionPod | None = None - # Ephemeral result store (None until set_ephemeral_store() is called by the pipeline) - self.ephemeral_result_store: "ArrowDatabaseProtocol | None" = None + # Ephemeral result database (None until set_ephemeral_database() is called by the pipeline) + self.ephemeral_result_database: "ArrowDatabaseProtocol | None" = None self._ephemeral_cached_pod: CachedFunctionPod | None = None # Node-level orchestrator config — applied post-construction via the @@ -1212,8 +1212,8 @@ def __init__( self._node_config: NodeConfig = NodeConfig() # Set by from_descriptor() for a loaded node whose job file records how many - # pipeline records its run had in the store (ITL-1043); checked once per - # store, with a warning only: the store is the source of truth. + # pipeline records its run had in the database (ITL-1043); checked once per + # database, with a warning only: the database is the source of truth. self._expected_row_count: int | None = None self._row_count_checked: bool = False # Batched commits, set only while PipelineJob.run runs the node (ITL-919). @@ -1278,10 +1278,10 @@ def _delivers_side_effects(self) -> bool: def _begin_run(self) -> None: """Make a node that delivers side effects look for older completions again. - Tables written before v0.3 can appear in a store between two runs of one + Tables written before v0.3 can appear in a database between two runs of one node object. Such a node must find them before it delivers anything, so it looks for them at the start of every run, also when every input is already - held in memory; other nodes report once, when they first read the store. + held in memory; other nodes report once, when they first read the database. Raises: SideEffectLogDataContextError: If this node delivers side effects and @@ -1296,14 +1296,14 @@ def _account_for_stale_tables(self) -> None: Done once per node object; a node that delivers side effects does it at the start of every run (``_begin_run``). Its current tables are read and - counted by data context the first time only: the later runs list the store + counted by data context the first time only: the later runs list the database for tables at an older layout, which reads no table. A pipeline table written by an earlier version of Orcapod (``pdb_v1``) holds no ``__data_context`` column: its rows are not read as results and are left unchanged, and their number is reported (ITL-944). Such tables are looked for at this node's own path and, because the hashes in a path can differ between - versions, anywhere under the function's name in the store (ITL-954). A node + versions, anywhere under the function's name in the database (ITL-954). A node that delivers side effects fails instead, and for it the result tables and rows of another data context in the current tables count too: none of them can be matched with current inputs, so every side effect would be delivered @@ -1339,7 +1339,7 @@ def _account_for_stale_tables(self) -> None: ) ) if found.unlisted: - # That the store cannot list its tables is reported once per node + # That the database cannot list its tables is reported once per node # object and attached database, not at every run. if self._unlisted_store_reported: found = StaleTables(tables=found.tables) @@ -1348,7 +1348,7 @@ def _account_for_stale_tables(self) -> None: account_for_stale_tables( found, delivers_side_effects=delivers, current_context=context_key ) - except StoreError as exc: + except OrcapodDatabaseError as exc: exc.add_context(node_label=self.label) raise if delivers: @@ -1393,12 +1393,12 @@ def attach_databases( result_database: Database for cached results. Defaults to ``pipeline_database.at("_result")`` when ``pipeline_database`` is not ``None``. - ephemeral_database: Optional ephemeral store to attach immediately - via ``set_ephemeral_store()``. Equivalent to calling - ``set_ephemeral_store(ephemeral_database)`` after + ephemeral_database: Optional ephemeral database to attach immediately + via ``set_ephemeral_database()``. Equivalent to calling + ``set_ephemeral_database(ephemeral_database)`` after ``attach_databases()``. """ - # Another store has its own tables of other generations to account for. + # Another database has its own tables of other generations to account for. self._stale_tables_checked = False self._current_tables_checked = False self._unlisted_store_reported = False @@ -1425,7 +1425,7 @@ def attach_databases( result_database=result_database, record_path=self._stored_result_record_path, ) - # With a store attached, the stub can serve the results its saved + # With a database attached, the stub can serve the results its saved # run stored, though it still cannot compute. from orcapod.pipeline.serialization import LoadStatus @@ -1440,7 +1440,7 @@ def attach_databases( self._invalidate_pipeline_hash_cache() if ephemeral_database is not None: - self.set_ephemeral_store(ephemeral_database) + self.set_ephemeral_database(ephemeral_database) # ------------------------------------------------------------------ # from_descriptor — read-only stub for loaded jobs @@ -1481,10 +1481,10 @@ def from_descriptor( ``get_all_records()`` to correctly filter DB rows when ``table_scope="pipeline_hash"`` (the default). expected_row_count: How many pipeline records of this node (with its - run's content hash) the store held when the job was saved, from the + run's content hash) the database held when the job was saved, from the job file's optional ``run.expected_rows``; ``None`` when not recorded. - Advisory: a read always returns what the store holds and only warns - (``StoreContentWarning``) when the store holds another number. + Advisory: a read always returns what the database holds and only warns + (``OrcapodDatabaseContentWarning``) when the database holds another number. Returns: A ``FunctionJobNode`` in read-only stub mode @@ -1593,10 +1593,10 @@ def from_descriptor( node._pipeline_database = None node._cached_function_pod = None - # FunctionJobNode — ephemeral result store (same defaults as __init__). + # FunctionJobNode — ephemeral result database (same defaults as __init__). # A loaded node has no function pod, so it never gets an ephemeral pod; # _fetch_joined_records reads it to resolve ephemeral pipeline records. - node.ephemeral_result_store = None + node.ephemeral_result_database = None node._ephemeral_cached_pod = None # Wire databases if provided @@ -1643,9 +1643,9 @@ def _held_pairs(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: After a completed run, the rows are those of that run's inputs: one per input (two equal inputs give two rows, as downstream received them). The in-memory - cache may hold more (results of other inputs, read from the store), which stay + cache may hold more (results of other inputs, read from the database), which stay cached for reuse but are not rows of this run (ITL-1093). Without a completed - run in this process there are no rows, whatever the cache or the store holds + run in this process there are no rows, whatever the cache or the database holds (ITL-1095). Rows come in the order the cache holds them. An input without output (an inactive function) is not a row. """ @@ -1696,7 +1696,7 @@ def _holds_run(self, *, load_only: bool) -> bool: A load-only run's rows are good only for another load-only run. A run that left failed rows is never good: the next run retries them, reading the rest from the - store (ITL-1095). + database (ITL-1095). """ return ( self._run_entry_counts is not None @@ -1717,7 +1717,7 @@ def _stored_entry( ) -> tuple[TagProtocol, DataProtocol | None] | None: """The result held for ``entry_id`` after a lookup, or ``None`` if none is stored. - A placeholder for a result kept in an ephemeral store that is gone is not a + A placeholder for a result kept in an ephemeral database that is gone is not a stored result. """ entry = self._cached_output_datas.get(entry_id) @@ -1743,7 +1743,7 @@ def _execute_load_only( The loaded (tag, output data) rows, excluding ``None`` outputs. Raises: - RuntimeError: If no store is attached (``UNAVAILABLE``). + RuntimeError: If no database is attached (``UNAVAILABLE``). InputValidationError: If the input stream is of another data context. """ from orcapod.pipeline.observer import NoOpObserver @@ -1758,7 +1758,7 @@ def _execute_load_only( not_stored = 0 try: if self.load_status == LoadStatus.UNAVAILABLE: - raise self._no_store_error() + raise self._no_database_error() require_same_data_context( input_stream.data_context_key, self._stored_data_context_key ) @@ -1806,7 +1806,7 @@ async def _async_execute_load_only( try: if self.load_status == LoadStatus.UNAVAILABLE: - raise self._no_store_error() + raise self._no_database_error() require_same_data_context( self._input_stream.data_context_key, self._stored_data_context_key ) @@ -1851,34 +1851,34 @@ async def _async_execute_load_only( self.clear_cache() raise - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: - """Assign or remove the ephemeral result store. + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: + """Assign or remove the ephemeral result database. - When *store* is not ``None``, creates a ``CachedFunctionPod`` backed by - *store* so that ephemeral writes use the same format as persistent writes. - When *store* is ``None``, clears both the store and the ephemeral pod. + When *database* is not ``None``, creates a ``CachedFunctionPod`` backed by + *database* so that ephemeral writes use the same format as persistent writes. + When *database* is ``None``, clears both the database and the ephemeral pod. Note: For deserialized read-only nodes where ``_function_pod`` is ``None``, - ``ephemeral_result_store`` is assigned but ``_ephemeral_cached_pod`` + ``ephemeral_result_database`` is assigned but ``_ephemeral_cached_pod`` remains ``None``. Such nodes cannot compute new results anyway, so ``_process_data_internal`` must guard against this by checking ``_ephemeral_cached_pod is not None`` before attempting ephemeral writes. Args: - store: The ``ArrowDatabaseProtocol`` to use for ephemeral result + database: The ``ArrowDatabaseProtocol`` to use for ephemeral result storage, or ``None`` to detach and revert to persistent-only writes. """ - # Another store has its own tables of other generations to account for, + # Another database has its own tables of other generations to account for, # as in ``attach_databases``. self._stale_tables_checked = False self._current_tables_checked = False self._unlisted_store_reported = False - self.ephemeral_result_store = store - if store is not None and self._function_pod is not None: + self.ephemeral_result_database = database + if database is not None and self._function_pod is not None: self._ephemeral_cached_pod = CachedFunctionPod( self._function_pod, - result_database=store, + result_database=database, ) else: self._ephemeral_cached_pod = None @@ -1887,11 +1887,11 @@ def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: # Internal helpers # ------------------------------------------------------------------ - def _no_store_error(self) -> RuntimeError: - """Return the error for reading a loaded node that has no store attached. + def _no_database_error(self) -> RuntimeError: + """Return the error for reading a loaded node that has no database attached. Only a node loaded by ``PipelineJob.load`` can be ``UNAVAILABLE``: it has no - function pod, so its results can come only from the store. + function pod, so its results can come only from the database. """ return RuntimeError( f"FunctionJobNode {self.label!r} was loaded without a database, so its stored " @@ -1907,12 +1907,12 @@ def _cannot_compute_error(self) -> RuntimeError: """ return RuntimeError( f"FunctionJobNode {self.label!r} was loaded by PipelineJob.load and has no " - "function pod, so it cannot compute; fill it from the store with " + "function pod, so it cannot compute; fill it from the database with " "run(load_only=True), or read its stored results with as_source()." ) def _stored_record_count(self) -> int: - """Return how many pipeline records of this node the attached store holds. + """Return how many pipeline records of this node the attached database holds. A record belongs to this node when its ``_node_content_hash`` is this node's content hash: the rows ``iter_data()`` and ``get_all_records()`` @@ -1936,7 +1936,7 @@ def _expected_row_count_to_save(self, run_finished: bool) -> int | None: A loaded node keeps the count its file recorded, or none. A node of a job whose last run finished (its writes committed) records the pipeline records - the store holds under its run's key; otherwise nothing is certain. + the database holds under its run's key; otherwise nothing is certain. Args: run_finished: Whether the job's last run finished and committed. @@ -1951,18 +1951,18 @@ def _require_readable_stub(self) -> None: """Raise unless this loaded node can read its stored results. Raises: - RuntimeError: If no store is attached. + RuntimeError: If no database is attached. """ from orcapod.pipeline.serialization import LoadStatus if self.load_status == LoadStatus.UNAVAILABLE: - raise self._no_store_error() + raise self._no_database_error() self._check_expected_rows() def _check_expected_rows(self) -> None: - """Warn, once per store, if the store holds another number of rows than recorded. + """Warn, once per database, if the database holds another number of rows than recorded. - The store is the source of truth: the read goes on and returns what it holds + The database is the source of truth: the read goes on and returns what it holds (ITL-1043). Only a job file that records the node's count is checked. """ if self._expected_row_count is None or self._row_count_checked: @@ -1974,7 +1974,7 @@ def _check_expected_rows(self) -> None: if found != self._expected_row_count: report_unexpected_row_count( f"FunctionJobNode {self.label!r}", - describe_store_table( + describe_database_table( self._pipeline_database, self._versioned_pipeline_path ), self._expected_row_count, @@ -2041,23 +2041,23 @@ def _batched_commits(self, rows_per_commit: int) -> Iterator[None]: for pod in pods: pod.set_auto_flush(True) - def _ephemeral_store_in_use(self) -> ArrowDatabaseProtocol | None: - """Return the ephemeral store this node writes results to, if it does.""" + def _ephemeral_database_in_use(self) -> ArrowDatabaseProtocol | None: + """Return the ephemeral database this node writes results to, if it does.""" if self._node_config.is_result_ephemeral and self._ephemeral_cached_pod: - return self.ephemeral_result_store + return self.ephemeral_result_database return None def _commit_result_table(self) -> None: """Commit this node's pending result rows, before its pipeline records. - Every view of a store shares one set of pending writes, and a flush commits + Every view of a database shares one set of pending writes, and a flush commits them table by table in no guaranteed order. Committing the result table on its own first means a crash between the two commits leaves results without pipeline records (found in the result cache on the rerun, so nothing is recomputed), never pipeline records pointing at results that were lost. - Only a store that commits table by table - (``TableCommittingDatabaseProtocol``, the Delta store) needs and offers this; - for another store the pipeline flush that follows commits everything. + Only a database that commits table by table + (``TableCommittingDatabaseProtocol``, the Delta database) needs and offers this; + for another database the pipeline flush that follows commits everything. """ pod = self._cached_function_pod if not isinstance(pod, CachedFunctionPod): @@ -2070,33 +2070,33 @@ def _commit_pending_writes(self) -> None: """Commit this node's pending writes, results first (see ``_batched_commits``). Raises: - StoreReadError: If a stored table could not be read; it names this node. - StoreWriteError: If a commit failed, naming this node and the store + OrcapodDatabaseReadError: If a stored table could not be read; it names this node. + OrcapodDatabaseWriteError: If a commit failed, naming this node and the database (``flush=True``, no row: the batch holds many rows). The failed writes stay pending, and earlier commits are kept. """ pipeline_database = self._require_pipeline_database() - ephemeral = self._ephemeral_store_in_use() + ephemeral = self._ephemeral_database_in_use() try: if ephemeral is not None: try: ephemeral.flush() - except StoreReadError: + except OrcapodDatabaseReadError: raise except Exception as exc: - raise StoreWriteError( + raise OrcapodDatabaseWriteError( ephemeral.base_path, node_label=self.label, flush=True ) from exc try: self._commit_result_table() pipeline_database.flush() - except StoreReadError: + except OrcapodDatabaseReadError: raise except Exception as exc: - raise StoreWriteError( + raise OrcapodDatabaseWriteError( pipeline_database.base_path, node_label=self.label, flush=True ) from exc - except StoreError as exc: + except OrcapodDatabaseError as exc: self._commit_failed = True exc.add_context(node_label=self.label) raise @@ -2118,8 +2118,8 @@ def _commit_at_node_end(self) -> None: def _commit_after_failure(self, run_error: BaseException) -> None: """Commit what succeeded before a failure, never replacing the run's error. - The job flushes its store after a failed run, but not a node's ephemeral - store, a separate database, and not table by table: so the node commits its + The job flushes its database after a failed run, but not a node's ephemeral + database, a separate database, and not table by table: so the node commits its ephemeral results and then its result table here, best-effort (a failure is noted on ``run_error``). Skipped when the failure was this node's own commit: the job's flush is that commit's one retry. @@ -2129,7 +2129,7 @@ def _commit_after_failure(self, run_error: BaseException) -> None: """ if self._commit_every_rows is None or self._commit_failed: return - ephemeral = self._ephemeral_store_in_use() + ephemeral = self._ephemeral_database_in_use() if ephemeral is not None: commit_best_effort(run_error, ephemeral.flush) commit_best_effort(run_error, self._commit_result_table) @@ -2148,8 +2148,8 @@ def _commit_or_end_node( """ try: commit() - except StoreError as exc: - with keep_store_error(exc, "Observer on_node_end"): + except OrcapodDatabaseError as exc: + with keep_database_error(exc, "Observer on_node_end"): ctx_obs.on_node_end(node_label, node_hash) raise @@ -2313,22 +2313,22 @@ def execute( run_id=run_id, base_entry_id=base_entry_id, ) - except StoreError as exc: - # A failed store read or write fails the run whatever the error + except OrcapodDatabaseError as exc: + # A failed database read or write fails the run whatever the error # policy: error_policy only decides about rows whose function # raised. exc.add_context(node_label=node_label, tag=tag.as_dict()) - with keep_store_error(exc, "Observer on_data_crash"): + with keep_database_error(exc, "Observer on_data_crash"): ctx_obs.on_data_crash(node_label, tag, data, exc) - with keep_store_error(exc, "Observer on_node_end"): + with keep_database_error(exc, "Observer on_node_end"): ctx_obs.on_node_end(node_label, node_hash) raise except TypeIdentityError as exc: # A stored row that cannot be decoded as its class is never # skipped as a row failure, whatever the error policy (ITL-881). - with keep_store_error(exc, "Observer on_data_crash"): + with keep_database_error(exc, "Observer on_data_crash"): ctx_obs.on_data_crash(node_label, tag, data, exc) - with keep_store_error(exc, "Observer on_node_end"): + with keep_database_error(exc, "Observer on_node_end"): ctx_obs.on_node_end(node_label, node_hash) raise except Exception as exc: @@ -2402,7 +2402,7 @@ def _process_data_internal( When ``node_config.is_result_ephemeral=True``: - Uses ``_ephemeral_cached_pod`` for both compute and storage. - - Raises ``RuntimeError`` if no ephemeral store has been set. + - Raises ``RuntimeError`` if no ephemeral database has been set. - Calls ``add_pipeline_record`` which computes the next recomputation index (``max_index + 1``) and writes with ``skip_duplicates=True`` so concurrent coroutines are safely serialised. @@ -2453,7 +2453,7 @@ def _process_data_internal( if self._ephemeral_cached_pod is None: raise RuntimeError( f"FunctionJobNode '{self.label}' has is_result_ephemeral=True but no " - "ephemeral store has been assigned. Call set_ephemeral_store() with " + "ephemeral database has been assigned. Call set_ephemeral_database() with " "an ArrowDatabaseProtocol before executing this node." ) tag_out, output_data = self._ephemeral_cached_pod.process_data( @@ -2596,7 +2596,7 @@ async def _async_process_data_internal( if self._ephemeral_cached_pod is None: raise RuntimeError( f"FunctionJobNode '{self.label}' has is_result_ephemeral=True but no " - "ephemeral store has been assigned. Call set_ephemeral_store() with " + "ephemeral database has been assigned. Call set_ephemeral_database() with " "an ArrowDatabaseProtocol before executing this node." ) tag_out, output_data = await self._ephemeral_cached_pod.async_process_data( @@ -2757,7 +2757,7 @@ def add_pipeline_record( - Output data record ID (for joining with result records) - Base entry ID (``_PIPELINE_BASE_ENTRY_ID_COL``) - Recomputation index (``_PIPELINE_RECOMPUTATION_INDEX_COL``) - - Whether the result is stored in the ephemeral store + - Whether the result is stored in the ephemeral database - Input data context key - Whether the result was freshly computed or cached - Output data content hash (``OUTPUT_DATA_HASH_COL``) — used by downstream @@ -2769,7 +2769,7 @@ def add_pipeline_record( data_record_id: UUID of the result record in the result database. computed: Whether the result was freshly computed (``True``) or served from a cache (``False``). - is_ephemeral: Whether the result is stored in the ephemeral store. + is_ephemeral: Whether the result is stored in the ephemeral database. output_data_hash: Content hash of the output produced by processing ``input_data``. When provided, stored as ``OUTPUT_DATA_HASH_COL`` so that downstream ``EmptyData`` flow-through can locate its cached @@ -2803,7 +2803,7 @@ def _add_pipeline_record( base_entry_id: ``compute_base_entry_id(tag, input_data)``. data_record_id: UUID of the result record in the result database. computed: Whether the result was freshly computed. - is_ephemeral: Whether the result is stored in the ephemeral store. + is_ephemeral: Whether the result is stored in the ephemeral database. output_data_hash: Content hash of the output, if any. """ pipeline_database = self._require_pipeline_database() @@ -2892,11 +2892,11 @@ def _add_pipeline_record( combined_record, skip_duplicates=True, ) - except StoreReadError: + except OrcapodDatabaseReadError: # The stored table could not be read: report the read failure itself. raise except Exception as exc: - raise StoreWriteError( + raise OrcapodDatabaseWriteError( self._versioned_pipeline_path, node_label=self.label, tag=tag.as_dict(), @@ -2937,12 +2937,12 @@ def get_all_records( Raises: RuntimeError: If the node was loaded by ``PipelineJob.load`` and no - store is attached. + database is attached. Warns: - StoreContentWarning: Once per store, if the node was loaded from a job file + OrcapodDatabaseContentWarning: Once per database, if the node was loaded from a job file that records another number of pipeline records for its run than the - store holds. The store is the source of truth. + database holds. The database is the source of truth. """ if self._function_pod is None: self._require_readable_stub() @@ -3031,11 +3031,11 @@ def _fetch_joined_records( self, base_entry_ids: list[bytes] | None = None, ) -> _JoinedRecords | None: - """Internal primitive: fetch both DBs and inner-join, supporting two stores. + """Internal primitive: fetch both DBs and inner-join, supporting two databases. Fetches ``taginfo`` from the pipeline database, partitions rows by ``IS_EPHEMERAL_COL`` into persistent and ephemeral groups, performs two - independent inner joins (one per store), merges with persistent priority + independent inner joins (one per database), merges with persistent priority via an anti-join, and returns the combined result. Persistent miss rows (tag entry with no matching result DB row) emit a @@ -3188,7 +3188,7 @@ def _fetch_joined_records( how="anti", ) else: - # No ephemeral store or empty store — all ephemeral rows are misses. + # No ephemeral database or empty database — all ephemeral rows are misses. unmatched_df = ephemeral_taginfo_df for row in unmatched_df.iter_rows(named=True): base_eid = row[_PIPELINE_BASE_ENTRY_ID_COL] @@ -3225,7 +3225,7 @@ def _fetch_joined_records( empty_taginfo_rows[base_eid] = row # ------------------------------------------------------------------ - # Keep both stores' rows until latest-recomputation selection. Persistent + # Keep both databases' rows until latest-recomputation selection. Persistent # rows come last so they retain priority when recomputation indices tie. # ------------------------------------------------------------------ if ephemeral_df.height > 0 and persistent_df.height > 0: @@ -3235,7 +3235,7 @@ def _fetch_joined_records( elif persistent_df.height > 0: merged_df = persistent_df else: - # No results found in either store — return empty table preserving taginfo schema + # No results found in either database — return empty table preserving taginfo schema empty_table = taginfo.slice(0, 0) return _JoinedRecords( table=empty_table, @@ -3478,7 +3478,7 @@ def head( run's inputs, the first of what ``as_table`` returns. A node that has not run in this process, or whose input changed since (stale), shows no rows and says so, pointing to ``run()`` and to ``as_source()`` for its stored results (ITL-1095); - the store is not read. The node has all of its rows, so with ``sort_by_tags`` + the database is not read. The node has all of its rows, so with ``sort_by_tags`` (or ``all_info``) the head is exactly ``as_table(...).slice(0, n)``, whatever order the rows were computed in. See ``StreamBase.head``. """ @@ -3489,7 +3489,7 @@ def head( status = self.load_status if status == LoadStatus.UNAVAILABLE: return self._head_without_rows( - config, ViewStatus.UNAVAILABLE, note="no store attached" + config, ViewStatus.UNAVAILABLE, note="no database attached" ) if self._run_entry_counts is None: return self._head_without_rows( @@ -3521,23 +3521,23 @@ def as_table( ) -> "pa.Table": """Return the rows of this node's current run as a PyArrow table. - Read-only: never computes and never reads the store. Before a run in this + Read-only: never computes and never reads the database. Before a run in this process, or once the node is stale, the table has zero rows and the node's - columns (ITL-1095); ``as_source()`` reads what the store holds. + columns (ITL-1095); ``as_source()`` reads what the database holds. Raises: - RuntimeError: If the node was loaded and no store is attached. + RuntimeError: If the node was loaded and no database is attached. """ from orcapod.pipeline.serialization import LoadStatus if self.load_status == LoadStatus.UNAVAILABLE: - raise self._no_store_error() + raise self._no_database_error() return super().as_table(columns=columns, all_info=all_info) def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Yield the (tag, data) rows of this node's current run. - Strictly read-only: never computes and never reads the store. Callers must + Strictly read-only: never computes and never reads the database. Callers must call ``run()`` (or run the job) first. After a completed run (``execute``, ``async_execute``, ``run()`` or a job run) @@ -3554,7 +3554,7 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: status = self.load_status if status == LoadStatus.UNAVAILABLE: - raise self._no_store_error() + raise self._no_database_error() # A CACHE_ONLY node has no live input to be stale against. if status != LoadStatus.CACHE_ONLY and self.is_stale: self.clear_cache() @@ -3582,7 +3582,7 @@ def run(self, *, load_only: bool = False) -> StreamView: or ``loaded_only`` after a load-only run. Raises: - RuntimeError: If no store is attached (``UNAVAILABLE``); if the node cannot + RuntimeError: If no database is attached (``UNAVAILABLE``); if the node cannot compute (loaded without a function pod, or a placeholder pod) and ``load_only`` is false; or if its input is unavailable (``CACHE_ONLY``): read its stored rows with ``as_source()``. @@ -3591,7 +3591,7 @@ def run(self, *, load_only: bool = False) -> StreamView: status = self._load_status if status == LoadStatus.UNAVAILABLE: - raise self._no_store_error() + raise self._no_database_error() if status == LoadStatus.CACHE_ONLY: raise RuntimeError( f"FunctionJobNode {self.label!r}: its input is unavailable, so it has " @@ -3627,31 +3627,31 @@ def _run_alone( from orcapod.pipeline.node_run import commit_run from orcapod.pipeline.run_summary import RunCountingObserver, RunSummary - from orcapod.utils.store_errors import flush_best_effort + from orcapod.utils.database_errors import flush_best_effort if self.is_stale: # Discard any stale in-memory entries before a fresh computation run # so that rerunning does not mix old cached entries with new results. self.clear_cache() counter = RunCountingObserver() - stores = [ - store - for store in ( + databases = [ + database + for database in ( self._pipeline_database, getattr(self._cached_function_pod, "result_database", None), ) - if store is not None + if database is not None ] try: _ = self.execute(self._input_stream, observer=counter, load_only=load_only) except Exception as run_error: if not load_only: # Commit what succeeded; a load-only run writes nothing (ITL-1095). - flush_best_effort(run_error, *stores) + flush_best_effort(run_error, *databases) self.clear_cache() raise if not load_only: - commit_run(stores, [self]) + commit_run(databases, [self]) node_summary = replace( counter.node_summary(self.label), not_stored=self.last_run_not_stored ) @@ -3701,7 +3701,7 @@ def _run_view(self) -> StreamView: table = self.as_table() if not self._run_load_only: # The rows are the ones held in memory, as head() names them afterwards: - # some may have been read from the result store rather than computed. + # some may have been read from the result database rather than computed. return self._view_of_rows(table, "held") return StreamView.of_rows( self, @@ -3771,7 +3771,7 @@ async def async_execute( return try: self._begin_run() - except StoreError: + except OrcapodDatabaseError: # Nothing was delivered; downstream must not wait for rows. A run that # raised defines no rows. self.clear_cache() @@ -3784,7 +3784,7 @@ async def async_execute( if status == LoadStatus.UNAVAILABLE: await output.close() - raise self._no_store_error() + raise self._no_database_error() if self._function_pod is None: await output.close() @@ -3827,8 +3827,8 @@ async def async_execute( if self._ephemeral_cached_pod is None: raise RuntimeError( f"FunctionJobNode '{self.label}' has is_result_ephemeral=True " - "but no ephemeral store has been assigned. Call " - "set_ephemeral_store() with an ArrowDatabaseProtocol before " + "but no ephemeral database has been assigned. Call " + "set_ephemeral_database() with an ArrowDatabaseProtocol before " "executing this node." ) execution_pod = self._ephemeral_cached_pod @@ -3920,7 +3920,7 @@ async def record_and_forward() -> None: is_ephemeral=is_ephemeral, output_data_hash=output_data.content_hash(), ) - except StoreError as exc: + except OrcapodDatabaseError as exc: # This runs outside the pod's per-row handler: name the # node and row here. exc.add_context( @@ -4014,8 +4014,8 @@ def on_data_crash( clean = tag.drop_meta_columns( _TAG_NODE_INPUT_REF, ignore_missing=True ) - if isinstance(error, StoreError): - # The pod re-raises a failed store read or write after + if isinstance(error, OrcapodDatabaseError): + # The pod re-raises a failed database read or write after # this call; name the node it belongs to. error.add_context(node_label=node_label) else: @@ -4075,7 +4075,7 @@ def create_data_logger(self, tag: Any, data: Any) -> Any: except asyncio.CancelledError as cancelled: # Another node's failure cancelled this one: commit what it wrote, as a # failing node does. The job's flush commits neither a node's ephemeral - # store nor its result table ahead of its pipeline records. + # database nor its result table ahead of its pipeline records. self._commit_after_failure(cancelled) # A cancelled run defines no rows: the node reads as one with no run. self.clear_cache() @@ -4084,9 +4084,9 @@ def create_data_logger(self, tag: Any, data: Any) -> Any: try: if node_open: # The node ends for the observer before it raises, as in execute; - # a failing observer must not replace a store error on its way out. - with keep_store_error( - store_error_of(run_error), "Observer on_node_end" + # a failing observer must not replace a database error on its way out. + with keep_database_error( + database_error_of(run_error), "Observer on_node_end" ): ctx_obs.on_node_end(node_label, node_hash) finally: diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 7bcc6d46d..83b7949c8 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -48,14 +48,14 @@ from orcapod.core.nodes.staleness import RunHoldingNode from orcapod.errors import ( PipelineJobRequiredError, - StoreReadError, - StoreWriteError, + OrcapodDatabaseReadError, + OrcapodDatabaseWriteError, ) from orcapod.contexts.stored import ( keep_rows_of_data_context, require_same_data_context, account_for_stale_tables, - describe_store_table, + describe_database_table, find_stale_tables, report_rows_of_other_data_contexts, report_unexpected_row_count, @@ -445,14 +445,14 @@ def __repr__(self) -> str: ) @abstractmethod - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: - """Assign or remove the ephemeral result store for this node. + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: + """Assign or remove the ephemeral result database for this node. No-op for operator nodes — full ephemeral support for operators is deferred to ITL-509. Args: - store: An ``ArrowDatabaseProtocol`` instance to attach, or ``None`` + database: An ``ArrowDatabaseProtocol`` instance to attach, or ``None`` to detach. Ignored by operator nodes in v1. """ @@ -645,7 +645,7 @@ def from_descriptor( return node - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: """No-op — blueprint operator nodes carry no database references.""" def as_node(self) -> OperatorNode: @@ -718,7 +718,7 @@ class OperatorJobNode(RunHoldingNode, OperatorNodeBase): # Set once a table found at the older layout has been accounted for. _stale_tables_checked: bool = False # The job file's expected row count for a loaded node, and whether it was checked - # against the attached store (ITL-1043). Advisory: a mismatch only warns. + # against the attached database (ITL-1043). Advisory: a mismatch only warns. _expected_row_count: int | None = None _row_count_checked: bool = False @@ -808,7 +808,7 @@ def as_node(self) -> OperatorNode: tracker_manager=self.tracker_manager, ) - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: """No-op — full ephemeral support for operators is deferred to ITL-509.""" # ------------------------------------------------------------------ @@ -843,7 +843,7 @@ def attach_databases( pipeline_database: Database for pipeline records. cache_mode: Caching behaviour (OFF, LOG, or REPLAY). """ - # Another store has its own tables of other generations to account for. + # Another database has its own tables of other generations to account for. self._stale_tables_checked = False self._skipped_rows_reported = False self._row_count_checked = False @@ -888,10 +888,10 @@ def from_descriptor( to database instances. job_content_hash: The content hash of the node in the saved job's run, which its stored rows carry; ``None`` when the file does not record it. - expected_row_count: How many rows the store held under the run's key when + expected_row_count: How many rows the database held under the run's key when the job was saved, from the job file's optional ``run.expected_rows``; - ``None`` when not recorded. Advisory: a read returns what the store - holds and only warns (``StoreContentWarning``) when it differs. + ``None`` when not recorded. Advisory: a read returns what the database + holds and only warns (``OrcapodDatabaseContentWarning``) when it differs. Returns: A new ``OperatorJobNode`` instance. @@ -1029,7 +1029,7 @@ def _expected_row_count_to_save(self, run_finished: bool) -> int | None: """Return the count ``PipelineJob.save`` records for this node, if certain. A loaded node keeps the count its file recorded, or none. A node of a job whose - last run finished (its writes committed) records the rows the store holds under + last run finished (its writes committed) records the rows the database holds under its run's content hash; otherwise nothing is certain. Args: @@ -1043,12 +1043,12 @@ def _expected_row_count_to_save(self, run_finished: bool) -> int | None: return 0 if records is None else records.num_rows def _check_expected_rows(self, found: int) -> None: - """Warn, once per store, if a loaded node finds another number of rows than recorded. + """Warn, once per database, if a loaded node finds another number of rows than recorded. - The store is the source of truth: the read returns what it holds (ITL-1043). + The database is the source of truth: the read returns what it holds (ITL-1043). Args: - found: The rows the store holds under the node's run key. + found: The rows the database holds under the node's run key. """ if self._expected_row_count is None or self._row_count_checked: return @@ -1056,7 +1056,9 @@ def _check_expected_rows(self, found: int) -> None: if found != self._expected_row_count and self._pipeline_database is not None: report_unexpected_row_count( f"OperatorJobNode {self.label!r}", - describe_store_table(self._pipeline_database, self._stored_table_path), + describe_database_table( + self._pipeline_database, self._stored_table_path + ), self._expected_row_count, found, ) @@ -1080,7 +1082,7 @@ def _cannot_compute_error(self) -> RuntimeError: """Describe why this loaded node cannot execute an operator.""" return RuntimeError( f"OperatorJobNode {self.label!r} was loaded by PipelineJob.load and " - "cannot compute; fill it from the store with run(load_only=True), read " + "cannot compute; fill it from the database with run(load_only=True), read " "its stored results with as_source(), or create a fresh PipelineJob to " "run the pipeline again." ) @@ -1196,11 +1198,11 @@ def _store_output_stream(self, stream: StreamProtocol) -> None: record_id_column=self.HASH_COLUMN_NAME, skip_duplicates=True, ) - except StoreReadError: + except OrcapodDatabaseReadError: # The stored table could not be read: report the read failure itself. raise except Exception as exc: - raise StoreWriteError( + raise OrcapodDatabaseWriteError( self._stored_table_path, node_label=self.label ) from exc @@ -1344,7 +1346,7 @@ def _read_run_records(self) -> "pa.Table | None": A loaded node names its run by the content hash its job file records. A file saved before Orcapod recorded it leaves the node only the hash the loaded job - computes; when the store holds no rows under that hash, an empty answer would + computes; when the database holds no rows under that hash, an empty answer would make stored results look dropped, so this raises instead (ITL-1043). Returns: @@ -1353,7 +1355,7 @@ def _read_run_records(self) -> "pa.Table | None": Raises: RuntimeError: If a loaded node's job file does not record its run's - content hash and the store holds no rows under the hash the loaded + content hash and the database holds no rows under the hash the loaded job computes. """ stored = self._read_current_context_records() @@ -1364,7 +1366,7 @@ def _read_run_records(self) -> "pa.Table | None": and (records is None or records.num_rows == 0) ): found = ( - "is absent from the attached store" + "is absent from the attached database" if stored is None else f"holds {stored.num_rows} stored rows, none of them under the " f"content hash the loaded job computes ({self.content_hash().to_string()})" @@ -1400,13 +1402,13 @@ def _load_cached_stream_from_db(self) -> "ArrowTableStream | None": - Returns ``None`` if ``_cache_mode != CacheMode.REPLAY`` (LOG/OFF modes may have stale historical records in the DB). - The store is the source of truth: a loaded node whose job file records its + The database is the source of truth: a loaded node whose job file records its run's content hash reads exactly as a node that runs does, so an absent table or a table with no rows under that hash is an empty result. Raises: RuntimeError: If a loaded REPLAY node's job file does not record its run's - content hash and the store holds no rows under the hash the loaded job + content hash and the database holds no rows under the hash the loaded job computes: the node cannot name its run's rows, so an empty result would make stored results look dropped. """ @@ -1521,7 +1523,7 @@ def execute( ctx_obs.on_node_end(node_label, node_hash) return output except BaseException: - # A run that raised (its store write or an observer included) defines no rows. + # A run that raised (its database write or an observer included) defines no rows. self.clear_cache() raise @@ -1694,7 +1696,7 @@ def _run_view(self, *, computed: bool) -> StreamView: # that reused the kept result computed nothing and is not stamped. self._update_modified_time() # The view names its rows as head() names them afterwards: computed by a live - # computation, plain rows when replayed from the store or kept by execute(). + # computation, plain rows when replayed from the database or kept by execute(). kept = self._cached_output_stream origin = "computed" if kept is None else head_of(kept.head(1), self).origin view = self._view_of_rows(table, origin) @@ -1720,7 +1722,7 @@ def _current_output(self) -> StreamProtocol | None: A node holds a run's output from the end of ``run()``, ``execute()`` or ``async_execute()`` until it is cleared or becomes stale (an input or the operator changed since). Before a run in this process it holds none, whatever - its cache mode and store hold (ITL-1095). + its cache mode and database hold (ITL-1095). """ if self._cached_output_stream is None or self.is_stale: return None @@ -1729,7 +1731,7 @@ def _current_output(self) -> StreamProtocol | None: def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Return an iterator over the (tag, data) rows of the node's current run. - Read-only: never computes and never reads the store. Yields the rows of the + Read-only: never computes and never reads the database. Yields the rows of the latest ``run()``, ``execute()`` or ``async_execute()``; nothing before a run in this process, after ``clear_cache()`` or once the node is stale, in every cache mode (ITL-1095). ``as_source()`` reads what the node stored (``LOG``). @@ -1747,7 +1749,7 @@ def as_table( ) -> "pa.Table": """Return the output of the node's current run as a PyArrow table. - Read-only: never computes and never reads the store. Returns the rows of the + Read-only: never computes and never reads the database. Returns the rows of the latest ``run()``, ``execute()`` or ``async_execute()``; before a run in this process, after ``clear_cache()`` or once the node is stale, a zero-row table with the node's columns, in every cache mode (ITL-1095). @@ -1796,7 +1798,7 @@ def head( After a run (or ``execute``) the rows are the node's output in memory. Before a run in this process the node shows no rows and says so, pointing to ``run()`` and to ``as_source()``, in every cache mode; once stale it says its inputs - changed (ITL-1095). The store is never read. See ``StreamBase.head``. + changed (ITL-1095). The database is never read. See ``StreamBase.head``. """ check_head_size(n) config = ColumnConfig.handle_config(columns, all_info=all_info) diff --git a/src/orcapod/core/result_cache.py b/src/orcapod/core/result_cache.py index 44fcaa28c..fa3ee1de3 100644 --- a/src/orcapod/core/result_cache.py +++ b/src/orcapod/core/result_cache.py @@ -12,7 +12,7 @@ from datetime import datetime, timezone from typing import TYPE_CHECKING, Any -from orcapod.errors import StoreReadError, StoreWriteError +from orcapod.errors import OrcapodDatabaseReadError, OrcapodDatabaseWriteError from orcapod.contexts.stored import ( StaleTables, keep_rows_of_data_context, @@ -56,7 +56,7 @@ class ResultCache: Args: result_database: The database to store/retrieve cached results. record_path: The record path tuple for scoping records in the database. - auto_flush: If True, flush the database after each store operation. + auto_flush: If True, flush the database after each database operation. """ # Meta column indicating whether the result was freshly computed @@ -75,7 +75,7 @@ def __init__( Args: result_database: The store holding the result table. record_path: Path of the result table without its layout component. - auto_flush: Whether to flush the store after each stored result. + auto_flush: Whether to flush the database after each stored result. delivers_side_effects: Whether computing a result is an external action (a function pod that takes the invocation context). Such a cache fails, instead of computing again, when it finds results stored @@ -340,21 +340,21 @@ def store( data_table, skip_duplicates=skip_duplicates, ) - except StoreReadError: + except OrcapodDatabaseReadError: # The stored table could not be read: report the read failure itself. raise except Exception as exc: - raise StoreWriteError(self._versioned_record_path) from exc + raise OrcapodDatabaseWriteError(self._versioned_record_path) from exc if self._auto_flush: try: self._result_database.flush() - except StoreReadError: + except OrcapodDatabaseReadError: # The stored table could not be read: report the read failure itself. raise except Exception as exc: - # Views of one store share their pending writes, so the batch that + # Views of one database share their pending writes, so the batch that # failed may be an earlier row's or another table's. - raise StoreWriteError( + raise OrcapodDatabaseWriteError( self._result_database.base_path, flush=True ) from exc diff --git a/src/orcapod/core/streams/view.py b/src/orcapod/core/streams/view.py index 0e22e43ee..075c10947 100644 --- a/src/orcapod/core/streams/view.py +++ b/src/orcapod/core/streams/view.py @@ -51,7 +51,7 @@ class ViewStatus(str, Enum): the rows are the stored results of its inputs; inputs without one were skipped, so the rows may be fewer than a full run gives (ITL-1095). NOT_COMPUTED: Nothing is computed, or what was computed is out of date. - NOTHING_STORED: A stored view (``node.as_source()``) whose store holds no rows + NOTHING_STORED: A stored view (``node.as_source()``) whose database holds no rows for its node yet. UNAVAILABLE: No rows can exist here: a pipeline blueprint, an input slot with no source bound, or a source that is not available. @@ -191,7 +191,7 @@ class StreamView: total_is_estimate: ``True`` when ``total_rows`` is a recorded estimate. origin: What the rows are: ``"held"`` (in memory), ``"read"`` (a bounded read of a table), ``"computed"`` (cached from an earlier computation), - ``"stored"`` (a job node's store) or ``"delivered"`` (the rows a side-effect + ``"stored"`` (a job node's database) or ``"delivered"`` (the rows a side-effect stream passed on while delivering). note: For a view without rows, the explanation shown instead of the default; for a view with rows, a remark shown after the row count. diff --git a/src/orcapod/databases/delta_arrow_types.py b/src/orcapod/databases/delta_arrow_types.py index 66fc9738a..144bedf8e 100644 --- a/src/orcapod/databases/delta_arrow_types.py +++ b/src/orcapod/databases/delta_arrow_types.py @@ -6,7 +6,7 @@ fails for a value above the signed maximum; every timestamp to microseconds, dropping nanosecond digits; a zone to UTC) and cannot store the others at all. -A Delta store must give back the type it was given: a tag read back with another type +A Delta database must give back the type it was given: a tag read back with another type changes every record id computed from it. So before a write, a column whose type Delta would not keep is recorded in its top-level field metadata (``ARROW_TYPE_KEY``: the column's whole Arrow type, IPC-serialized; Delta keeps top-level field metadata but not @@ -23,7 +23,7 @@ left as it is: its type is restored from its extension metadata. A table written before this existed has no recorded type and reads back with Delta's -types; such a store must be rebuilt. +types; such a database must be rebuilt. """ from __future__ import annotations diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index 32ac55b7a..19dd15fea 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -19,7 +19,7 @@ from orcapod.databases.utils import coerce_record_id from orcapod.errors import ( InputValidationError, - StoreReadError, + OrcapodDatabaseReadError, TableListingNotSupportedError, TypeIdentityError, ) @@ -402,10 +402,10 @@ def _types_conflict(stored: pa.DataType, incoming: pa.DataType) -> bool: return stored != incoming -def _store_error_to_raise(errors: list[Exception]) -> Exception | None: - """Pick the store error a failed flush raises as itself. +def _database_error_to_raise(errors: list[Exception]) -> Exception | None: + """Pick the database error a failed flush raises as itself. - A ``StoreReadError`` is preferred, because it is the error that fails a run + A ``OrcapodDatabaseReadError`` is preferred, because it is the error that fails a run whatever the orchestrator's ``error_policy``; failing that, an ``InputValidationError`` from a refused append. @@ -413,10 +413,10 @@ def _store_error_to_raise(errors: list[Exception]) -> Exception | None: errors: The error of every batch that failed, in flush order. Returns: - The first ``StoreReadError``, else the first ``InputValidationError``, or - ``None`` when no batch failed with a store error. + The first ``OrcapodDatabaseReadError``, else the first ``InputValidationError``, or + ``None`` when no batch failed with a database error. """ - for kind in (StoreReadError, InputValidationError): + for kind in (OrcapodDatabaseReadError, InputValidationError): chosen = next((e for e in errors if isinstance(e, kind)), None) if chosen is not None: return chosen @@ -428,7 +428,7 @@ class _CommittedSnapshot: """What one version of a stored Delta table holds, kept in memory (ITL-919). Per-row lookups are answered from this snapshot instead of re-reading the table. - Each part is read on first use and extended in place after this store's own + Each part is read on first use and extended in place after this database's own commits, so its cost follows the rows written, not the rows stored. Attributes: @@ -654,17 +654,17 @@ def _get_delta_table( self._delta_table_cache.pop(record_key, None) raise self._read_failed(record_path) from e - def _read_failed(self, record_path: tuple[str, ...]) -> StoreReadError: + def _read_failed(self, record_path: tuple[str, ...]) -> OrcapodDatabaseReadError: """Build the error for a table at ``record_path`` that exists but cannot be read. Args: record_path: Path components relative to this database view. Returns: - A ``StoreReadError`` naming the table; the caller raises it from the + A ``OrcapodDatabaseReadError`` naming the table; the caller raises it from the backend's exception. """ - return StoreReadError( + return OrcapodDatabaseReadError( self._get_table_uri(record_path), self._path_prefix + record_path ) @@ -677,11 +677,11 @@ def list_table_paths(self, prefix: tuple[str, ...] = ()) -> list[tuple[str, ...] v0.2 stored it), and never enters ``_delta_log``. No table is opened. Only flushed tables are listed, as with ``table_exists``. - A local store is listed exactly. A cloud store is listed through ``fsspec`` + A local database is listed exactly. A cloud database is listed through ``fsspec`` with one list request per directory: ``s3://`` and ``s3a://`` through ``s3fs`` with the credentials, endpoint and region of this database's storage options; ``gs://``, ``az://`` and ``abfs://`` only when the matching - ``fsspec`` package is installed and reaches the store with its default + ``fsspec`` package is installed and reaches the database with its default credentials. Args: @@ -691,9 +691,9 @@ def list_table_paths(self, prefix: tuple[str, ...] = ()) -> list[tuple[str, ...] The table paths, relative to ``self.base_path``, sorted. Raises: - TableListingNotSupportedError: If the store is on cloud storage and + TableListingNotSupportedError: If the database is on cloud storage and could not be listed; the reason is chained. - StoreReadError: If a local directory could not be listed. + OrcapodDatabaseReadError: If a local directory could not be listed. """ full_prefix = self._path_prefix + prefix if self._is_cloud: @@ -724,10 +724,10 @@ def list_table_paths(self, prefix: tuple[str, ...] = ()) -> list[tuple[str, ...] except Exception as exc: if self._is_cloud: raise TableListingNotSupportedError( - f"Could not list the tables of the Delta store at " + f"Could not list the tables of the Delta database at " f"{self._root_uri!r}: {type(exc).__name__}: {exc}" ) from exc - raise StoreReadError(directory, full_prefix + below) from exc + raise OrcapodDatabaseReadError(directory, full_prefix + below) from exc for name in children: if name == _DELTA_LOG_DIRECTORY: found.append(prefix + below) @@ -744,7 +744,7 @@ def _cloud_directory_lister(self) -> "Callable[[str], list[str]]": Raises: TableListingNotSupportedError: If no ``fsspec`` file system is available - for this store's scheme. + for this database's scheme. """ import fsspec @@ -764,7 +764,7 @@ def _cloud_directory_lister(self) -> "Callable[[str], list[str]]": # deltalake refuses a plain-HTTP endpoint unless AWS_ALLOW_HTTP is set; # the listing follows the same rule and sends no credentials over it. raise TableListingNotSupportedError( - f"The Delta store at {self._root_uri!r} cannot list its tables: its " + f"The Delta database at {self._root_uri!r} cannot list its tables: its " "endpoint uses plain HTTP and AWS_ALLOW_HTTP is not set." ) try: @@ -774,7 +774,7 @@ def _cloud_directory_lister(self) -> "Callable[[str], list[str]]": ) except Exception as exc: raise TableListingNotSupportedError( - f"The Delta store at {self._root_uri!r} cannot list its tables: no " + f"The Delta database at {self._root_uri!r} cannot list its tables: no " f"usable fsspec file system for {scheme!r} " f"({type(exc).__name__}: {exc})." ) from exc @@ -926,7 +926,7 @@ def _stored_ids(self, record_path: tuple[str, ...]) -> set[bytes]: """Return the record ids stored at ``record_path`` (in the loaded version). Raises: - StoreReadError: If the table exists but its ids could not be read. + OrcapodDatabaseReadError: If the table exists but its ids could not be read. """ delta_table = self._get_delta_table(record_path) if delta_table is None: @@ -946,7 +946,7 @@ def _stored_rows(self, record_path: tuple[str, ...]) -> KeyIndex | None: """Return the indexed rows stored at ``record_path``, or ``None`` if no table. Raises: - StoreReadError: If the table exists but could not be read. + OrcapodDatabaseReadError: If the table exists but could not be read. """ delta_table = self._get_delta_table(record_path) if delta_table is None: @@ -1273,7 +1273,7 @@ def _without_stored_ids( ``when_not_matched_insert_all``), so a pending row with a stored id is never returned by a read: the stored row is. The stored ids are read even when ``pending`` is empty, so a read with rows pending raises - ``StoreReadError`` whenever the stored ids cannot be read. + ``OrcapodDatabaseReadError`` whenever the stored ids cannot be read. Args: record_path: Path components relative to this database view. @@ -1388,7 +1388,7 @@ def get_all_records( Combined Arrow table with all records, or None if no records exist Raises: - StoreReadError: If the Delta table exists but could not be read. + OrcapodDatabaseReadError: If the Delta table exists but could not be read. """ tables_to_combine = [] @@ -1570,7 +1570,7 @@ def get_records_by_ids( if pending is not None: id_type = pending.table.schema.field(self.RECORD_ID_COLUMN).type # Reading the stored ids drops the pending rows a flush would not - # insert; it raises StoreReadError when the stored table cannot be read. + # insert; it raises OrcapodDatabaseReadError when the stored table cannot be read. pending_matches = self._without_stored_ids( record_path, self._rows_with_ids(pending, record_ids_list, id_type) ) @@ -1628,10 +1628,10 @@ def flush(self) -> None: stays queued, so a later flush tries it again, and each failure is logged. Raises: - StoreReadError: If a batch's existing table could not be loaded. It is + OrcapodDatabaseReadError: If a batch's existing table could not be loaded. It is raised as itself; every other batch that failed in this flush is added to it as a note. - InputValidationError: If no batch failed with ``StoreReadError`` but a + InputValidationError: If no batch failed with ``OrcapodDatabaseReadError`` but a batch's column types differ from its stored table's. Other failed batches are added to it as notes. RuntimeError: If batches failed with other errors only, naming the @@ -1645,16 +1645,16 @@ def flush(self) -> None: logger.error(f"Error flushing batch for {record_key}: {e}") errors.append((record_key, e)) if errors: - store_error = _store_error_to_raise([e for _, e in errors]) - if store_error is not None: + database_error = _database_error_to_raise([e for _, e in errors]) + if database_error is not None: # Keep every other failure visible on the error that is raised. for record_key, other in errors: - if other is not store_error: - store_error.add_note( + if other is not database_error: + database_error.add_note( f"Another batch also failed to flush ({record_key}): " f"{type(other).__name__}: {other}" ) - raise store_error + raise database_error failed_keys = [k for k, _ in errors] raise RuntimeError( f"Failed to flush {len(errors)} batch(es): {failed_keys}" @@ -1727,7 +1727,9 @@ def _flush_by_key(self, record_key: str) -> None: except TableNotFoundError: delta_table = None except Exception as load_error: - raise StoreReadError(table_uri, abs_path_components) from load_error + raise OrcapodDatabaseReadError( + table_uri, abs_path_components + ) from load_error if delta_table is None: # TODO: reconsider mode="overwrite" here @@ -1772,7 +1774,7 @@ def _flush_by_key(self, record_key: str) -> None: "the new rows have " f"{original_table.schema.field(overflowing).type} values it " "cannot hold. The stored rows are unchanged and nothing was " - "written; rebuild the store." + "written; rebuild the database." ) base = self._snapshot_for_commit( record_key, delta_table, table_uri, abs_path_components @@ -1927,7 +1929,7 @@ def _snapshot_for_commit( A snapshot of the loaded version with its ``ids`` read. Raises: - StoreReadError: If the stored ids could not be read. + OrcapodDatabaseReadError: If the stored ids could not be read. """ cached = self._snapshots.get(record_key) if ( @@ -1939,7 +1941,7 @@ def _snapshot_for_commit( try: ids = self._scan_ids(delta_table) except Exception as e: - raise StoreReadError(table_uri, abs_path_components) from e + raise OrcapodDatabaseReadError(table_uri, abs_path_components) from e return _CommittedSnapshot(delta_table.version(), ids=ids) def _extend_snapshot( diff --git a/src/orcapod/databases/extension_aware_database.py b/src/orcapod/databases/extension_aware_database.py index 276f969fc..01e4fa8af 100644 --- a/src/orcapod/databases/extension_aware_database.py +++ b/src/orcapod/databases/extension_aware_database.py @@ -213,7 +213,7 @@ def list_table_paths(self, prefix: tuple[str, ...] = ()) -> list[tuple[str, ...] The table paths, relative to ``self.base_path``, sorted. Raises: - TableListingNotSupportedError: If the wrapped store cannot enumerate + TableListingNotSupportedError: If the wrapped database cannot enumerate its tables. """ return self._db.list_table_paths(prefix) diff --git a/src/orcapod/databases/in_memory_databases.py b/src/orcapod/databases/in_memory_databases.py index 699c5fac5..a4686da94 100644 --- a/src/orcapod/databases/in_memory_databases.py +++ b/src/orcapod/databases/in_memory_databases.py @@ -29,7 +29,7 @@ class InMemoryArrowDatabase: Supports the same pending-batch semantics as DeltaTableDatabase: records are buffered in a pending batch and become part of the committed - store only after flush() is called (or flush=True is passed to a write method). + database only after flush() is called (or flush=True is passed to a write method). Lookups by record ID or by a binary or string column value use hash indexes over the committed rows, and each flush appends without copying the stored rows, so the @@ -176,7 +176,7 @@ def _committed_id_index(self, record_key: str) -> dict[Any, list[int]]: def _filter_existing_records( self, record_key: str, table: "pa.Table" ) -> "pa.Table": - """Filter out records whose IDs are already in pending or committed store.""" + """Filter out records whose IDs are already in pending or committed database.""" input_ids = set(table[self.RECORD_ID_COLUMN].to_pylist()) pending_ids = self._pending_record_ids[record_key] committed_ids = self._committed_id_index(record_key) @@ -460,7 +460,7 @@ def get_record_by_id( if filtered.num_rows > 0: return self._handle_record_id_column(filtered, record_id_column) - # Check committed store + # Check committed database committed = self._tables.get(record_key) if committed is None: return None diff --git a/src/orcapod/databases/storage_utils.py b/src/orcapod/databases/storage_utils.py index 69410c432..87257263e 100644 --- a/src/orcapod/databases/storage_utils.py +++ b/src/orcapod/databases/storage_utils.py @@ -62,7 +62,7 @@ def delta_to_pyarrow_dataset( A local table is therefore read through a native pyarrow filesystem rooted at the table, which returns the same schema and values without Python file objects. Any other - URI, a cloud store or a ``filesystem`` passed by the caller, keeps deltalake's reader. + URI, a cloud database or a ``filesystem`` passed by the caller, keeps deltalake's reader. ``table_uri`` is percent-encoded: deltalake 1.0.2 and 1.2.1 return a bare path such as ``/data/a%20b/``, and 1.6.6 returns ``file:///data/a%20b/``. pyarrow's own URI parser diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 034181529..e3d1fef1e 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -363,14 +363,14 @@ class SourceCheckFailedWarning(UserWarning): """ -class StoreContentWarning(UserWarning): - """A loaded node's store holds another number of rows than the saved run recorded. +class OrcapodDatabaseContentWarning(UserWarning): + """A loaded node's database holds another number of rows than the saved run recorded. A job file may record, in its optional ``run.expected_rows`` section, how many rows - the store held under each node's run key when the job was saved. Reading a loaded - node always returns what the attached store holds, which is the source of truth; + the database held under each node's run key when the job was saved. Reading a loaded + node always returns what the attached database holds, which is the source of truth; when that differs from the recorded number, this warning is issued once per node. - The store may not be the one the run wrote, or it may have changed since. A file + The database may not be the one the run wrote, or it may have changed since. A file without the section, or without a node's entry, is not checked. """ @@ -488,22 +488,22 @@ def __init__( class TableListingNotSupportedError(NotImplementedError): - """Raised by ``list_table_paths`` when a store cannot enumerate its tables. + """Raised by ``list_table_paths`` when a database cannot enumerate its tables. - Some stores can read and write a table at a known path but have no way to list - the tables they hold (for example a Delta store on cloud storage for which no + Some databases can read and write a table at a known path but have no way to list + the tables they hold (for example a Delta database on cloud storage for which no file system package is installed). Orcapod then cannot look for tables written by an earlier version; it says so in a warning and carries on. """ -class StoreError(RuntimeError): - """Base of the errors raised when the store could not be read or written. +class OrcapodDatabaseError(RuntimeError): + """Base of the errors raised when the database could not be read or written. - A store error always fails the run, whatever the orchestrator's + A database error always fails the run, whatever the orchestrator's ``error_policy``: ``error_policy="continue"`` skips rows whose *function* raised, never rows whose stored data could not be read or written. Catch - ``StoreError`` to handle both ``StoreReadError`` and ``StoreWriteError``. + ``OrcapodDatabaseError`` to handle both ``OrcapodDatabaseReadError`` and ``OrcapodDatabaseWriteError``. The backend's own exception is chained as ``__cause__``. @@ -568,13 +568,13 @@ def add_context( self.tag = dict(tag) -class StoreWriteError(StoreError): - """Raised when a node's results or records could not be written to the store. +class OrcapodDatabaseWriteError(OrcapodDatabaseError): + """Raised when a node's results or records could not be written to the database. The run stops at the first failed write. Rows stored before the failure are kept, and re-running the job does not recompute stored function results. A write that fails because the stored table cannot be read raises - ``StoreReadError`` instead. + ``OrcapodDatabaseReadError`` instead. Attributes: record_path: Path of the table the write was for, relative to the @@ -585,7 +585,7 @@ class StoreWriteError(StoreError): tag: Tag values of the row being stored, or ``None`` when the write covered many rows (an operator's output), no row, or was a flush. flush: ``True`` when committing pending writes failed. Every view of a - store shares one set of pending writes, so the write that failed may + database shares one set of pending writes, so the write that failed may belong to an earlier row or another table than the one being stored. """ @@ -643,17 +643,18 @@ def __str__(self) -> str: if self.flush: node = f" for node {self.node_label!r}" if self.node_label else "" return ( - f"Could not commit pending writes to the store{node} " - f"(store {'/'.join(self.record_path)!r}){reason}. The failing write " - "may belong to an earlier row or another table of this store. " + retry + f"Could not commit pending writes to the database{node} " + f"(database {'/'.join(self.record_path)!r}){reason}. The failing write " + "may belong to an earlier row or another table of this database. " + + retry ) return ( - f"Could not write to the store for {where}{row} " + f"Could not write to the database for {where}{row} " f"(table {'/'.join(self.record_path)!r}){reason}. " + retry ) -class StoreReadError(StoreError): +class OrcapodDatabaseReadError(OrcapodDatabaseError): """Raised when a stored table exists but could not be read. A table that does not exist reads as empty. A table that exists but cannot be @@ -696,7 +697,7 @@ def __str__(self) -> str: ) -class SideEffectLogDataContextError(StoreError): +class SideEffectLogDataContextError(OrcapodDatabaseError): """Raised when a side-effecting node finds completions logged under another data context. A node whose run is an external action finds what it already did by keys hashed @@ -717,14 +718,14 @@ class SideEffectLogDataContextError(StoreError): with the same name in the same job, or of another node of the same function, therefore raises this error too; ``record_paths`` names every table found. - A fresh store is not a safe way around this error: it holds no completions, so on - it the node performs every side effect again. Run the node on a fresh store only + A fresh database is not a safe way around this error: it holds no completions, so on + it the node performs every side effect again. Run the node on a fresh database only when a repeated delivery is acceptable; otherwise keep using the Orcapod version that wrote the tables. Migrating stored rows to the current data context is planned. Attributes: - record_paths: Paths, from the root of the store, of the tables holding such + record_paths: Paths, from the root of the database, of the tables holding such completions. record_path: The first of ``record_paths``. found_context: Data context key of the completions that do not match. @@ -768,7 +769,7 @@ def __str__(self) -> str: "current inputs, so running the node would perform every side effect " "again; nothing was delivered or written. Keep using the Orcapod " "version that wrote these tables; they are left in place and can be " - "migrated to the current data context later. A fresh store holds no " + "migrated to the current data context later. A fresh database holds no " "completions, so on it the node performs every side effect again: use " "one only when a repeated delivery is acceptable." ) diff --git a/src/orcapod/hashing/pandas_value_hash.py b/src/orcapod/hashing/pandas_value_hash.py index e2e2a1068..8bbdb9f75 100644 --- a/src/orcapod/hashing/pandas_value_hash.py +++ b/src/orcapod/hashing/pandas_value_hash.py @@ -15,7 +15,7 @@ The hash is strict (developer's decisions, November 11 and 12, 2026): everything about a value that can be changed is part of its identity, ``attrs`` and ``-0.0`` versus ``0.0`` included. The one exception is a property the stored form does not restore, which never -counts, so that a value read back from a store hashes like the value that was written: +counts, so that a value read back from a database hashes like the value that was written: ``flags``, the kind of a ``RangeIndex``, an index's ``freq``, unused MultiIndex levels, the storage of a string dtype, and the ``object`` dtype of a column that reads back as another dtype. ``PANDAS_IDENTITY_PROPERTIES`` lists every property and whether it is identity; it diff --git a/src/orcapod/logical_types/numpy_type.py b/src/orcapod/logical_types/numpy_type.py index 9686eebae..ae7fa4b99 100644 --- a/src/orcapod/logical_types/numpy_type.py +++ b/src/orcapod/logical_types/numpy_type.py @@ -12,7 +12,7 @@ ``writeable`` flag, dtype metadata, the ``align`` flag of a structured dtype (its offsets are kept) and padding bytes between structured fields (NumPy does not define their content; they are written as zero, so memory the array happened -to sit on never reaches a store). +to sit on never reaches a database). Value hash ``D`` (label ``ndarray_v1``), in two steps. With ``lp(x) = u64le(len(utf8(x))) || utf8(x)``: diff --git a/src/orcapod/pipeline/async_orchestrator.py b/src/orcapod/pipeline/async_orchestrator.py index 437b3ccd9..55cd52d5e 100644 --- a/src/orcapod/pipeline/async_orchestrator.py +++ b/src/orcapod/pipeline/async_orchestrator.py @@ -19,9 +19,13 @@ from typing import TYPE_CHECKING, Any, Literal, TypeVar from orcapod.channels import BroadcastChannel, Channel -from orcapod.errors import StoreError, StoreWriteError, TypeIdentityError +from orcapod.errors import ( + OrcapodDatabaseError, + OrcapodDatabaseWriteError, + TypeIdentityError, +) from orcapod.pipeline.result import OrchestratorResult -from orcapod.utils.store_errors import keep_store_error +from orcapod.utils.database_errors import keep_database_error from orcapod.protocols.node_protocols import ( is_function_node, is_operator_node, @@ -92,7 +96,7 @@ def run( cancels the run, waits for the functions still running to return, and is re-raised. A second interrupt during that wait returns at once, with a ``RuntimeWarning`` naming the nodes whose functions have not returned; when they - do, nothing more is written to the store. + do, nothing more is written to the database. Args: graph: A ``GraphProtocol`` DAG with GraphNode objects as vertices. @@ -208,7 +212,7 @@ async def _run_async( effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) topo_order: list[Any] = [] - store_error: StoreError | TypeIdentityError | None = None + database_error: OrcapodDatabaseError | TypeIdentityError | None = None # id of each exception a node raised -> that node's label. failed_nodes: dict[int, str] = {} try: @@ -354,26 +358,26 @@ async def _run_async( node_outputs=collectors if materialize_results else {} ) except ExceptionGroup as group: - # A failed store read or write stops the run. Raise it on its own, as the + # A failed database read or write stops the run. Raise it on its own, as the # sync orchestrator does, rather than inside the TaskGroup's exception # group. - store_error = _single_store_error(group, topo_order) - if store_error is not None: - raise store_error + database_error = _single_database_error(group, topo_order) + if database_error is not None: + raise database_error # A stored row that cannot be decoded as its class stops the run too, # raised on its own as the sync orchestrator raises it (ITL-881). identity_errors = [ exc for exc in _leaves(group) if isinstance(exc, TypeIdentityError) ] if identity_errors: - store_error = identity_errors[0] - raise store_error + database_error = identity_errors[0] + raise database_error # Any other failure is raised on its own too, as the sync orchestrator # raises it: a row's error under fail_fast, a side effect's delivery error. raise _single_failure(group, topo_order, failed_nodes) finally: - # A failing on_run_end must not replace a failed store read or write. - with keep_store_error(store_error, "Observer on_run_end"): + # A failing on_run_end must not replace a failed database read or write. + with keep_database_error(database_error, "Observer on_run_end"): effective_observer.on_run_end(run_id) @@ -520,7 +524,7 @@ def _warn_function_still_running( which = f"a function in {_nodes(labels)}" if labels else "a function" message = ( f"The run was cancelled, but {which} has not returned yet. Nothing more " - "will be written to the store when it does. To stop it, restart the kernel " + "will be written to the database when it does. To stop it, restart the kernel " "(or end the process)." ) # Shown every time, whatever filters hide repeated warnings. @@ -622,7 +626,7 @@ def on_data_end( output_data: Any, cached: bool, ) -> None: - """A row's function returned (or its result came from the store).""" + """A row's function returned (or its result came from the database).""" self._change(node_label, -1) def on_data_crash( @@ -664,7 +668,7 @@ def _start_node( ) -> None: """Start a node's coroutine in ``tg``, recording which node each failure came from. - A store error raised on its own or inside an exception group (a node that runs + A database error raised on its own or inside an exception group (a node that runs its own task group) gets the node's label when it has none. Every exception the node raises, each one inside a group included, is recorded in ``failed_nodes`` under the node's label, so ``_single_failure`` can order the run's failures as @@ -680,17 +684,17 @@ def _start_node( async def run() -> None: try: await work - except StoreError as exc: + except OrcapodDatabaseError as exc: # Name the node the failure belongs to, if the raising site did not, - # so _single_store_error can order it like the sync orchestrator. + # so _single_database_error can order it like the sync orchestrator. exc.add_context(node_label=node.label) failed_nodes[id(exc)] = node.label raise except BaseExceptionGroup as group: - # A node that runs its own task group raises its store errors inside + # A node that runs its own task group raises its database errors inside # a group; name the node on each and re-raise the group unchanged. - for store_error in StoreError.in_group(group): - store_error.add_context(node_label=node.label) + for database_error in OrcapodDatabaseError.in_group(group): + database_error.add_context(node_label=node.label) for leaf in _leaves(group): failed_nodes[id(leaf)] = node.label raise @@ -704,7 +708,7 @@ async def run() -> None: def _single_failure( group: BaseExceptionGroup, topo_order: list[Any], failed_nodes: dict[int, str] ) -> BaseException: - """Pick the one exception an async run raises when no store error is among its failures. + """Pick the one exception an async run raises when no database error is among its failures. The caller gets one exception, as from the sync orchestrator, never the task group's exception group: the one whose node comes first in topological order (the @@ -735,18 +739,18 @@ def _single_failure( return chosen -def _single_store_error( +def _single_database_error( group: ExceptionGroup, topo_order: list[Any] -) -> StoreError | None: - """Pick the one ``StoreError`` an async run raises from its exception group. +) -> OrcapodDatabaseError | None: + """Pick the one ``OrcapodDatabaseError`` an async run raises from its exception group. Concurrent node tasks can fail together, so the ``TaskGroup``'s group may hold - several store errors, read and write ones alike, and other exceptions. The - caller gets a single ``StoreError``, as from the sync orchestrator: the one + several database errors, read and write ones alike, and other exceptions. The + caller gets a single ``OrcapodDatabaseError``, as from the sync orchestrator: the one whose node comes first in topological order (the one the sync orchestrator would meet first, when both failures happen), with ties and node-less errors broken by their order in the group. Every other exception stays attached for - diagnosis: each is noted on the chosen error, other store errors are also + diagnosis: each is noted on the chosen error, other database errors are also logged at WARNING, and the whole group is the chosen error's ``__context__``. Its ``__cause__`` stays the backend error. @@ -755,25 +759,25 @@ def _single_store_error( topo_order: The run's nodes in topological order. Returns: - The store error to raise, or ``None`` when the group holds none. + The database error to raise, or ``None`` when the group holds none. """ - store_errors = StoreError.in_group(group) - if not store_errors: + database_errors = OrcapodDatabaseError.in_group(group) + if not database_errors: return None rank = {getattr(node, "label", None): i for i, node in enumerate(topo_order)} - chosen = min(store_errors, key=lambda err: rank.get(err.node_label, len(rank))) + chosen = min(database_errors, key=lambda err: rank.get(err.node_label, len(rank))) for other in _leaves(group): if other is chosen: continue - if isinstance(other, StoreError): - kind = "write" if isinstance(other, StoreWriteError) else "read" + if isinstance(other, OrcapodDatabaseError): + kind = "write" if isinstance(other, OrcapodDatabaseWriteError) else "read" logger.warning( - "Another store %s failed in the same run: %s", + "Another database %s failed in the same run: %s", kind, other, exc_info=other, ) - chosen.add_note(f"Another store {kind} failed in the same run: {other}") + chosen.add_note(f"Another database {kind} failed in the same run: {other}") else: chosen.add_note( f"Another node failed in the same run: {type(other).__name__}: {other}" diff --git a/src/orcapod/pipeline/base.py b/src/orcapod/pipeline/base.py index 8fa2173de..3236c1c2c 100644 --- a/src/orcapod/pipeline/base.py +++ b/src/orcapod/pipeline/base.py @@ -139,25 +139,25 @@ def operator_pods(self) -> dict[str, Any]: """Copy of compiled nodes that are operator-pod nodes (label → node).""" return {k: v for k, v in self._nodes.items() if v.node_type == "operator"} - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: - """Assign or remove the ephemeral result store for all nodes. + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: + """Assign or remove the ephemeral result database for all nodes. - Propagates ``store`` to every function-pod and operator-pod node in the - pipeline by calling ``node.set_ephemeral_store(store)`` on each. Source + Propagates ``database`` to every function-pod and operator-pod node in the + pipeline by calling ``node.set_ephemeral_database(database)`` on each. Source nodes are skipped — they produce, not consume, data. Nodes that do not support ephemeral storage (e.g. operator nodes in v1) treat this as a no-op. - Pass ``None`` to detach the ephemeral store from all nodes, reverting + Pass ``None`` to detach the ephemeral database from all nodes, reverting them to persistent-only writes for subsequent runs. Args: - store: An ``ArrowDatabaseProtocol`` instance to attach, or ``None`` + database: An ``ArrowDatabaseProtocol`` instance to attach, or ``None`` to detach. """ for node in self._nodes.values(): - if node.node_type != "source": - node.set_ephemeral_store(store) + if node.node_type not in ("source", "side_effect"): + node.set_ephemeral_database(database) @property def dag(self) -> OrcaDAG[NodeT]: diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index a4554b23a..605dbf415 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -22,9 +22,9 @@ unclaimed_tables_written_before_v03, ) from orcapod.errors import ( - StoreError, - StoreReadError, - StoreWriteError, + OrcapodDatabaseError, + OrcapodDatabaseReadError, + OrcapodDatabaseWriteError, TableListingNotSupportedError, ) from orcapod.pipeline.base import AbstractPipelineBase @@ -44,7 +44,7 @@ from orcapod.protocols import core_protocols as cp from orcapod.databases.utils import as_table_committing from orcapod.types import CacheMode, NodeConfig -from orcapod.utils.store_errors import flush_best_effort +from orcapod.utils.database_errors import flush_best_effort if TYPE_CHECKING: from pathlib import Path @@ -59,7 +59,7 @@ logger = logging.getLogger(__name__) _PENDING_BEFORE_RUN_NOTE = ( - "These writes were left pending in the store before this run started (written " + "These writes were left pending in the database before this run started (written " "without a flush, or by a run that failed). The run stopped before computing " "anything." ) @@ -73,8 +73,8 @@ class _PreparedRun: exec_dag: The nodes to run (unbound sources and their dependents left out). run_id: The run's identifier. pipeline_uri: ``@``, passed to the observer. - pipeline_db: The job's view of the store. - result_db: The job's function-result view of the store. + pipeline_db: The job's view of the database. + result_db: The job's function-result view of the database. plan: What the run runs and holds (``plan_run``, ITL-1095). unchecked_sources: The sources the start-of-run check could not check (``check_sources``, ITL-1102). @@ -165,7 +165,7 @@ def __init__( self._sources_by_hash: dict[str, cp.StreamProtocol] = {} self._has_run: bool = False # True only while the last run started has finished and committed its writes: - # what the store then holds for each node is what that run left (ITL-1043). + # what the database then holds for each node is what that run left (ITL-1043). self._last_run_finished: bool = False self._run_id: str | None = None # The summary of the last run this object completed (ITL-997). @@ -234,7 +234,7 @@ def compile(self) -> None: # Invalidate the lazy compiled_pipeline cache. self._compiled_pipeline = None - # Wire databases if a store is already set. + # Wire databases if a database is already set. if self._database is not None: self._distribute_databases() @@ -247,7 +247,7 @@ def last_run(self) -> RunSummary | None: """The summary of the last run this job completed, or ``None`` before one. The same object ``run`` and ``run_async`` returned: per function node, the rows - computed, read from the store and failed, and each node's first error. Use + computed, read from the database and failed, and each node's first error. Use ``job.last_run.print_traceback("")`` to see a failed row's full traceback. A run that raised leaves it unchanged. A loaded job has none. """ @@ -479,7 +479,7 @@ def bind( def _distribute_databases(self) -> None: """Wire live DB references to all FunctionJobNode and OperatorJobNode objects. - Called by ``compile()`` when a store is already set, and by ``bind()`` + Called by ``compile()`` when a database is already set, and by ``bind()`` when *database* is changed. Raises: @@ -514,7 +514,7 @@ def _report_unclaimed_tables_written_before_v03( ) -> None: """Report, once per run, the pre-v0.3 tables no node of this job reports itself. - A function or side-effect node looks under its own name in the store and + A function or side-effect node looks under its own name in the database and reports, or refuses, the tables an earlier version of Orcapod wrote there. Operator caches and cached sources have no name in their path that survives a change of hashes, and a node that left the job has no node to report it. @@ -543,7 +543,7 @@ def _report_unclaimed_tables_written_before_v03( unclaimed = unclaimed_tables_written_before_v03(pipeline_db, claimed) except TableListingNotSupportedError as exc: logger.info( - "The store cannot list its tables (%s), so tables written by an " + "The database cannot list its tables (%s), so tables written by an " "earlier version of Orcapod are reported only where a node finds " "them at its own path.", exc, @@ -814,10 +814,10 @@ def unbound_sources(self) -> list[str]: return result def is_complete(self) -> bool: - """Return ``True`` when all source nodes are bound and a store is set. + """Return ``True`` when all source nodes are bound and a database is set. Returns: - ``True`` if all SourceNode slots are bound and a store is set. + ``True`` if all SourceNode slots are bound and a database is set. """ return self._database is not None and not self.unbound_sources @@ -924,17 +924,17 @@ def run( ``None`` when no matching records exist; records from an earlier LOG run remain readable after switching to OFF. commit_every_rows: How many rows a function node processes between - commits of its results and pipeline records to the store + commits of its results and pipeline records to the database (default 100). Each node also commits when it finishes, before a failed run raises, and the job commits at its end. On a durable - store (``DeltaTableDatabase``) a hard crash (power loss, killed + database (``DeltaTableDatabase``) a hard crash (power loss, killed process) loses at most the rows of the running node since its last commit; re-running the job recomputes only those. A larger value commits less often (faster, fewer files); ``1`` commits after every row. A node run on its own, outside ``run()``, - commits after every row. It must be smaller than the store's - ``batch_size`` when the store has one (``DeltaTableDatabase``, - default 1000): the store commits every pending table by itself + commits after every row. It must be smaller than the database's + ``batch_size`` when the database has one (``DeltaTableDatabase``, + default 1000): the database commits every pending table by itself once one table's pending rows reach that size, which would break the node's results-before-records commit order. load_only: Fill the nodes from what is already stored and compute @@ -944,7 +944,7 @@ def run( the inputs without one, which the summary counts (``not_stored``); a function node forwards only the rows it loaded. No user function is called, no side effect is delivered (a side-effect node passes its - rows on) and nothing is written to the store, a ``LOG`` operator's + rows on) and nothing is written to the database, a ``LOG`` operator's cache included. Afterwards each node shows the rows it loaded. Rows whose function raised (with the default ``error_policy="continue"``) are @@ -954,24 +954,24 @@ def run( Returns: The run's ``RunSummary``: per function node, the rows computed, read from - the store and failed, with each node's first error. Shown at the end of a + the database and failed, with each node's first error. Shown at the end of a notebook cell; also kept as ``job.last_run``. Results are read from ``job.nodes``. Raises: ValueError: If no database is set, or ``commit_every_rows`` is not a - positive integer smaller than the store's ``batch_size``. + positive integer smaller than the database's ``batch_size``. RuntimeError: If no pipeline has been compiled, or the job was built by ``PipelineJob.load`` (a loaded job is read-only). - StoreWriteError: If a node's results or records could not be written - to the store, whatever the orchestrator's ``error_policy``. The run + OrcapodDatabaseWriteError: If a node's results or records could not be written + to the database, whatever the orchestrator's ``error_policy``. The run stops at the first failed write. Writes that succeeded before it are flushed first, so rows stored before the failure are kept, and re-running the job does not recompute stored function results. A failed commit of a node's batch names the node but no row (``flush=True``): the batch's writes stay pending and are retried once by the flush that follows the failure. - StoreReadError: If a stored table exists but could not be read, + OrcapodDatabaseReadError: If a stored table exists but could not be read, whatever the orchestrator's ``error_policy``, also when the read was needed to write a node's results. Its rows are not recomputed and the table is not overwritten. @@ -1037,8 +1037,8 @@ async def run_async( run a ``SyncPipelineOrchestrator`` with ``job.run``. ValueError: As for ``run``. RuntimeError: As for ``run``. - StoreWriteError: As for ``run``. - StoreReadError: As for ``run``. + OrcapodDatabaseWriteError: As for ``run``. + OrcapodDatabaseReadError: As for ``run``. """ from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator @@ -1085,12 +1085,12 @@ def _prepare_run( load_only: As for ``run``: a loaded job can run only this way. Returns: - The execution DAG, run id, pipeline URI and the store views of the run. + The execution DAG, run id, pipeline URI and the database views of the run. Raises: ValueError: As for ``run``. RuntimeError: As for ``run``. - StoreError: If writes still pending from before the run cannot be committed. + OrcapodDatabaseError: If writes still pending from before the run cannot be committed. """ import hashlib import uuid @@ -1111,17 +1111,17 @@ def _prepare_run( ) # A loaded job never computes: function pods are not saved with it, and even # an operator-only loaded job, whose operators are rebuilt, stays read-only. - # A load-only run fills it from the store (ITL-1095). + # A load-only run fills it from the database (ITL-1095). if self._loaded and not load_only: raise RuntimeError( "PipelineJob.run() cannot execute a loaded job: its function pods are " "not saved with it, so it cannot compute. Fill its nodes from the " - "store with run(load_only=True) (bind in-memory sources first), or " + "database with run(load_only=True) (bind in-memory sources first), or " "read a node's stored results with node.as_source(). To run a new " "computation, create a fresh PipelineJob from the original pipeline " "definition." ) - # Until this run finishes, what the store holds for a node is not certain. + # Until this run finishes, what the database holds for a node is not certain. self._last_run_finished = False table_committing = as_table_committing(self._database) @@ -1131,10 +1131,10 @@ def _prepare_run( ): raise ValueError( f"commit_every_rows={commit_every_rows} must be smaller than the " - f"store's batch_size={table_committing.batch_size}: the store commits a " + f"database's batch_size={table_committing.batch_size}: the database commits a " "table by itself once it holds batch_size pending rows, which could " "commit pipeline records before their results. Use a smaller " - "commit_every_rows, or create the store with a larger batch_size." + "commit_every_rows, or create the database with a larger batch_size." ) if self._database is None: raise ValueError( @@ -1152,7 +1152,7 @@ def _prepare_run( from orcapod.pipeline.serialization import LoadStatus # A loaded function node has no function pod, whatever its load status - # (READ_ONLY once a store is attached, so it can read its stored results). + # (READ_ONLY once a database is attached, so it can read its stored results). unavailable_labels = [ node.label for node in self._persistent_node_map.values() @@ -1263,24 +1263,26 @@ def _prepare_run( plan = plan_run(exec_dag, load_only=load_only) # Pre-v0.3 tables are reported for a run that runs a node or leaves one out # (ITL-954): an excluded node reports nothing itself, so the job's total counts - # its tables. Only a run that runs nothing and leaves nothing out lists no store. + # its tables. Only a run that runs nothing and leaves nothing out lists no database. if plan.to_run or excluded_hashes: self._report_unclaimed_tables_written_before_v03(excluded_hashes) - # A store that commits a table by itself at batch_size: commit what it already + # A database that commits a table by itself at batch_size: commit what it already # holds pending (writes made without a flush, or a failed run's batches) before # this run writes. Those rows would otherwise count towards a table's # batch_size, which could then be reached mid-run and commit this run's pipeline # records ahead of their results (ITL-919). - # A load-only run writes nothing, and a run that runs no node touches no store, + # A load-only run writes nothing, and a run that runs no node touches no database, # so neither commits anything, not even pending writes of earlier operations. if table_committing is not None and not load_only and plan.to_run: try: pipeline_db.flush() - except StoreError as exc: + except OrcapodDatabaseError as exc: exc.add_note(_PENDING_BEFORE_RUN_NOTE) raise except Exception as exc: - flush_error = StoreWriteError(pipeline_db.base_path, flush=True) + flush_error = OrcapodDatabaseWriteError( + pipeline_db.base_path, flush=True + ) flush_error.add_note(_PENDING_BEFORE_RUN_NOTE) raise flush_error from exc @@ -1358,12 +1360,12 @@ def _finish_run( The run's summary, also kept as ``last_run``. Raises: - StoreReadError: As for ``run``. - StoreWriteError: As for ``run``. + OrcapodDatabaseReadError: As for ``run``. + OrcapodDatabaseWriteError: As for ``run``. """ pipeline_db = prepared.pipeline_db result_db = prepared.result_db - # Flush databases. Every view of the store shares one set of pending + # Flush databases. Every view of the database shares one set of pending # batches, so a failed batch can leave the ones after it unwritten. pending_note = ( "Writes that were still pending when the flush failed, such as a " @@ -1373,17 +1375,17 @@ def _finish_run( try: if not load_only and plan.to_run: # A load-only run writes nothing and commits nothing, and a run that - # ran no node touches no store (ITL-1095). + # ran no node touches no database (ITL-1095). pipeline_db.flush() result_db.flush() - except StoreReadError as exc: + except OrcapodDatabaseReadError as exc: # A stored table could not be read: report the read failure itself. exc.add_note(pending_note) # A run whose commit failed did not complete: no node holds its rows. clear_run_rows(plan.to_run) raise except Exception as exc: - flush_error = StoreWriteError(pipeline_db.base_path, flush=True) + flush_error = OrcapodDatabaseWriteError(pipeline_db.base_path, flush=True) flush_error.add_note(pending_note) clear_run_rows(plan.to_run) raise flush_error from exc @@ -1488,10 +1490,10 @@ def save(self, path: "str | Path") -> None: # stubs whose content_hash() matches the _node_content_hash column in the # DB — required for _filter_by_content_hash() in get_all_records(). node_job_content_hashes: dict[str, str] = {} - # Advisory only (ITL-1043): how many rows the store holds under each node's + # Advisory only (ITL-1043): how many rows the database holds under each node's # run key, written only when certain (the last run finished and committed, or - # a loaded node's own record). A loaded node compares it with its store and - # warns on a difference; the store stays the source of truth. + # a loaded node's own record). A loaded node compares it with its database and + # warns on a difference; the database stays the source of truth. expected_rows: dict[str, int] = {} for node in (self._persistent_node_map or {}).values(): if isinstance(node, (FunctionJobNode, OperatorJobNode)): @@ -1631,7 +1633,7 @@ def load( logger.info( "Source %r (%s) keeps its data in memory and is not saved with the " "job; the loaded job has no data for it. Stored function-node " - "results are still readable from the store. To read source node " + "results are still readable from the database. To read source node " "%r itself, bind a source: job.bind(sources={%r: }).", spec_name, source_type, @@ -1641,7 +1643,7 @@ def load( except Exception as exc: logger.warning( "Could not reconstruct source %r (%s) from its saved config: %s. " - "Stored function-node results are still readable from the store. " + "Stored function-node results are still readable from the database. " "To read source node %r itself, bind a source: " "job.bind(sources={%r: }).", spec_name, @@ -1724,8 +1726,8 @@ def load( "node_job_content_hashes", {} ) # The optional, advisory section (ITL-1043): a node with an entry warns when - # its store holds another number of rows. Without it nothing is checked. The - # ITL-904 flag (node_has_stored_records) of older files is not used: the store + # its database holds another number of rows. Without it nothing is checked. The + # ITL-904 flag (node_has_stored_records) of older files is not used: the database # is the source of truth. expected_rows: dict[str, int] = run_block.get("expected_rows", {}) diff --git a/src/orcapod/pipeline/node_run.py b/src/orcapod/pipeline/node_run.py index dcb69a537..37ffe2a65 100644 --- a/src/orcapod/pipeline/node_run.py +++ b/src/orcapod/pipeline/node_run.py @@ -18,13 +18,13 @@ from dataclasses import dataclass from typing import TYPE_CHECKING, Any -from orcapod.errors import StoreError, StoreWriteError +from orcapod.errors import OrcapodDatabaseError, OrcapodDatabaseWriteError from orcapod.pipeline.run_summary import ( RunCountingObserver, RunSummary, run_node_summaries, ) -from orcapod.utils.store_errors import flush_best_effort +from orcapod.utils.database_errors import flush_best_effort if TYPE_CHECKING: from orcapod.protocols.database_protocols import ArrowDatabaseProtocol @@ -126,8 +126,8 @@ def refuse_unloadable_inputs(node: Any) -> None: ) -def _stores(nodes: list[Any]) -> list[ArrowDatabaseProtocol]: - """The distinct stores the given nodes write to: pipeline and result databases.""" +def _databases(nodes: list[Any]) -> list[ArrowDatabaseProtocol]: + """The distinct databases the given nodes write to: pipeline and result databases.""" found: list[ArrowDatabaseProtocol] = [] for node in nodes: pipeline_database = getattr(node, "_pipeline_database", None) @@ -354,25 +354,25 @@ def is_source(node: Any) -> bool: return RunPlan(dag, held_rows, to_run, held_counts) -def commit_run(stores: Sequence[ArrowDatabaseProtocol], ran: Sequence[Any]) -> None: +def commit_run(databases: Sequence[ArrowDatabaseProtocol], ran: Sequence[Any]) -> None: """Commit a run's writes at its end; a failed commit clears the nodes that ran. A run whose commit failed did not complete, so no node it ran holds its rows (ITL-1095); nodes it held keep the rows of their earlier run. Raises: - StoreError: The commit's own store error. - StoreWriteError: If committing failed for another reason. + OrcapodDatabaseError: The commit's own database error. + OrcapodDatabaseWriteError: If committing failed for another reason. """ - for store in stores: + for database in databases: try: - store.flush() - except StoreError: + database.flush() + except OrcapodDatabaseError: clear_run_rows(ran) raise except Exception as exc: clear_run_rows(ran) - raise StoreWriteError(store.base_path, flush=True) from exc + raise OrcapodDatabaseWriteError(database.base_path, flush=True) from exc except BaseException: # An interruption during the commit leaves its outcome unknown too. clear_run_rows(ran) @@ -388,7 +388,7 @@ def run_with_dependencies( one ``job.run()`` makes over the job: a dependency that holds a current run is not run and its rows feed the nodes after it, a stream outside the job is read, every other node runs once, in dependency order, on the ``SyncPipelineOrchestrator`` - (error policy ``"continue"``). The run ends with a commit of every store it wrote + (error policy ``"continue"``). The run ends with a commit of every database it wrote to, also, best-effort, when it fails; a load-only run commits nothing (it writes nothing, and pending writes of others stay pending). A run that raises leaves no node it ran holding rows. @@ -403,7 +403,7 @@ def run_with_dependencies( The run's summary, one entry per function node of the graph. Raises: - StoreWriteError: If committing the run's writes failed. + OrcapodDatabaseWriteError: If committing the run's writes failed. ValueError: For a load-only run, if a node it would run reads an input that computes or delivers when read (``refuse_unloadable_inputs``); nothing has run then. """ @@ -432,7 +432,7 @@ def visit(node: Any) -> None: visit(target) plan = plan_run(graph, load_only=load_only, outside=outside) - stores = _stores(plan.to_run) + databases = _databases(plan.to_run) counter = RunCountingObserver() try: _ = SyncPipelineOrchestrator().run( @@ -442,13 +442,13 @@ def visit(node: Any) -> None: if isinstance(run_error, Exception) and not load_only: # Commit the writes that succeeded before the failure, as job.run() does; # a load-only run writes nothing, so it commits nothing (ITL-1095). - flush_best_effort(run_error, *stores) + flush_best_effort(run_error, *databases) # A run that raised leaves no node it ran holding rows: whether their writes # were committed is unknown. clear_run_rows(plan.to_run) raise if not load_only: - commit_run(stores, plan.to_run) + commit_run(databases, plan.to_run) return RunSummary( _job_name(target), run_node_summaries(graph, counter, held=plan.held_counts), diff --git a/src/orcapod/pipeline/run_summary.py b/src/orcapod/pipeline/run_summary.py index d09462cc3..34c781252 100644 --- a/src/orcapod/pipeline/run_summary.py +++ b/src/orcapod/pipeline/run_summary.py @@ -2,7 +2,7 @@ With the default ``error_policy="continue"``, a row whose function raises is skipped and the run goes on. ``RunSummary`` tells the user, per function node, how many rows were -computed, how many were read from the store, and how many failed with which first error. +computed, how many were read from the database, and how many failed with which first error. """ from __future__ import annotations @@ -34,7 +34,7 @@ class NodeRunSummary: Attributes: label: The node's label, its key in ``job.nodes``. computed: Rows whose function ran and returned. - reused: Rows whose result was read from the store instead. + reused: Rows whose result was read from the database instead. failed: Rows whose function raised; the run skipped them. first_error: The exception of the first failed row, with its traceback, or ``None`` when no row failed. @@ -412,7 +412,7 @@ def on_data_end( output_data: DataProtocol | None, cached: bool, ) -> None: - """Count the row as reused (from the pipeline or result store) or computed.""" + """Count the row as reused (from the pipeline or result database) or computed.""" from_store = cached or ( output_data is not None and output_data.get_meta_value(ResultCache.RESULT_COMPUTED_FLAG, True) diff --git a/src/orcapod/pipeline/sync_orchestrator.py b/src/orcapod/pipeline/sync_orchestrator.py index a02723c57..bf9708a39 100644 --- a/src/orcapod/pipeline/sync_orchestrator.py +++ b/src/orcapod/pipeline/sync_orchestrator.py @@ -10,9 +10,9 @@ import uuid from typing import TYPE_CHECKING, Any, Literal -from orcapod.errors import StoreError, TypeIdentityError +from orcapod.errors import OrcapodDatabaseError, TypeIdentityError from orcapod.pipeline.result import OrchestratorResult -from orcapod.utils.store_errors import keep_store_error +from orcapod.utils.database_errors import keep_database_error from orcapod.protocols.node_protocols import ( is_function_node, is_operator_node, @@ -90,7 +90,7 @@ def run( effective_observer = observer if observer is not None else NoOpObserver() effective_observer.on_run_start(run_id, pipeline_uri=pipeline_uri) - store_error: StoreError | TypeIdentityError | None = None + database_error: OrcapodDatabaseError | TypeIdentityError | None = None current: Any = None try: topo_order = list(graph.topological_sort()) @@ -153,20 +153,20 @@ def run( buffers.clear() return OrchestratorResult(node_outputs=buffers) - except StoreError as exc: + except OrcapodDatabaseError as exc: # Name the node the failure belongs to, if the raising site did not. if current is not None: exc.add_context(node_label=current.label) - store_error = exc + database_error = exc raise except TypeIdentityError as exc: # A stored row that cannot be decoded as its class stops the run as a - # store error does; on_run_end must not replace it (ITL-881). - store_error = exc + # database error does; on_run_end must not replace it (ITL-881). + database_error = exc raise finally: - # A failing on_run_end must not replace a failed store write. - with keep_store_error(store_error, "Observer on_run_end"): + # A failing on_run_end must not replace a failed database write. + with keep_database_error(database_error, "Observer on_run_end"): effective_observer.on_run_end(run_id) @staticmethod diff --git a/src/orcapod/protocols/database_protocols.py b/src/orcapod/protocols/database_protocols.py index 74c5254dd..fb585495c 100644 --- a/src/orcapod/protocols/database_protocols.py +++ b/src/orcapod/protocols/database_protocols.py @@ -89,7 +89,7 @@ def at(self, *path_components: str) -> "ArrowDatabaseProtocol": For backends with shared pending state (InMemoryArrowDatabase, ConnectorArrowDatabase), calling flush() on any view drains the entire shared pending queue — not just the caller's prefix. - This is intentional: all views share the same underlying store. + This is intentional: all views share the same underlying database. Args: *path_components: Zero or more path components to append. @@ -136,7 +136,7 @@ def list_table_paths(self, prefix: tuple[str, ...] = ()) -> list[tuple[str, ...] - ``DeltaTableDatabase`` on ``s3://`` or ``s3a://``: the same walk through ``s3fs``, one list request per directory. On ``gs://``, ``az://`` and ``abfs://`` only when the matching ``fsspec`` package is installed and - can reach the store with its default credentials. A cloud store that + can reach the database with its default credentials. A cloud database that cannot be listed raises ``TableListingNotSupportedError``. - ``ConnectorArrowDatabase`` (SQLite, PostgreSQL, SpiralDB): the tables whose name in the backend starts with the name ``prefix`` maps to. A @@ -156,9 +156,9 @@ def list_table_paths(self, prefix: tuple[str, ...] = ()) -> list[tuple[str, ...] The table paths, sorted. Empty when there is none. Raises: - TableListingNotSupportedError: If this store cannot enumerate its + TableListingNotSupportedError: If this database cannot enumerate its tables. - StoreReadError: If the store can list its tables but the listing + OrcapodDatabaseReadError: If the database can list its tables but the listing failed. """ ... diff --git a/src/orcapod/protocols/node_protocols.py b/src/orcapod/protocols/node_protocols.py index 0f7954a8c..499c74b53 100644 --- a/src/orcapod/protocols/node_protocols.py +++ b/src/orcapod/protocols/node_protocols.py @@ -102,10 +102,10 @@ async def async_execute( load_only: bool = False, ) -> None: ... - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: - """Assign or remove the ephemeral result store for this node. + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: + """Assign or remove the ephemeral result database for this node. - Pass an ``ArrowDatabaseProtocol`` to attach the store. + Pass an ``ArrowDatabaseProtocol`` to attach the database. Pass ``None`` to detach it — the node falls back to persistent-only behaviour for subsequent writes. No-op for node types that do not support ephemeral result storage (e.g. blueprint ``FunctionNode``). @@ -140,8 +140,8 @@ async def async_execute( load_only: bool = False, ) -> None: ... - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: - """Assign or remove the ephemeral result store for this node. + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: + """Assign or remove the ephemeral result database for this node. No-op for operator nodes in v1 — full ephemeral support for operators is deferred to ITL-509. diff --git a/src/orcapod/protocols/pipeline_protocols.py b/src/orcapod/protocols/pipeline_protocols.py index 1845a55b5..4d39cc75b 100644 --- a/src/orcapod/protocols/pipeline_protocols.py +++ b/src/orcapod/protocols/pipeline_protocols.py @@ -62,16 +62,16 @@ def dag(self) -> "GraphProtocol[NodeT]": """Node-object DAG for topology traversal and introspection.""" ... - def set_ephemeral_store(self, store: "ArrowDatabaseProtocol | None") -> None: - """Assign or remove the ephemeral result store for all nodes. + def set_ephemeral_database(self, database: "ArrowDatabaseProtocol | None") -> None: + """Assign or remove the ephemeral result database for all nodes. - Propagates ``store`` to every function-pod and operator-pod node in - the pipeline by calling ``node.set_ephemeral_store(store)`` on each. + Propagates ``database`` to every function-pod and operator-pod node in + the pipeline by calling ``node.set_ephemeral_database(database)`` on each. Source nodes are skipped — they produce data, not consume it. Nodes that do not support ephemeral storage (e.g. operator nodes in v1) treat this as a no-op. - Pass ``None`` to detach the ephemeral store from all nodes, + Pass ``None`` to detach the ephemeral database from all nodes, reverting them to persistent-only writes for subsequent runs. """ ... diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index d8def4977..fad8d7f1a 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -34,9 +34,9 @@ ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import ( - StoreError, - StoreReadError, - StoreWriteError, + OrcapodDatabaseError, + OrcapodDatabaseReadError, + OrcapodDatabaseWriteError, ) from orcapod.contexts.stored import ( StaleTables, @@ -56,7 +56,7 @@ ) from orcapod.types import ColumnConfig from orcapod.utils import arrow_utils -from orcapod.utils.store_errors import flush_best_effort +from orcapod.utils.database_errors import flush_best_effort from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -419,7 +419,7 @@ def _execute_side_effect_row( ): try: prior = pipeline_database.get_record_by_id(table_path, record_id) - except StoreError as exc: + except OrcapodDatabaseError as exc: # A failed lookup fails the run like a failed write: name pod and row. exc.add_context(node_label=pod_name, tag=tag.as_dict()) raise @@ -455,8 +455,8 @@ def _execute_side_effect_row( data_context_key=data_context_key, ) return (tag, data) - except StoreError as exc: - # A failed store read or write fails the run, even under on_error="log". + except OrcapodDatabaseError as exc: + # A failed database read or write fails the run, even under on_error="log". exc.add_context(node_label=pod_name, tag=tag.as_dict()) raise except Exception as exc: @@ -514,11 +514,11 @@ def _write_invocation_row( with_data_context(record, data_context_key), skip_duplicates=True, ) - except StoreReadError: + except OrcapodDatabaseReadError: # The stored table could not be read: report the read failure itself. raise except Exception as exc: - raise StoreWriteError(table_path) from exc + raise OrcapodDatabaseWriteError(table_path) from exc # --------------------------------------------------------------------------- @@ -1070,10 +1070,10 @@ def _refuse_log_of_another_data_context(self) -> None: older table layout (written by Orcapod up to v0.2), and rows of another data context in the current log. An older log is looked for at this node's own path and, because the hash in the path can differ between versions, anywhere - under the pod's name in the store (ITL-954). Checked at the start of every + under the pod's name in the database (ITL-954). Checked at the start of every run, so a log that appears between two runs of one node object is found too. The current log is read and counted by data context the first time only; - the later runs list the store, which reads no table. + the later runs list the database, which reads no table. Raises: SideEffectLogDataContextError: If such deliveries are found; nothing has @@ -1093,7 +1093,7 @@ def _refuse_log_of_another_data_context(self) -> None: name_path=self.node_uri, ) if found.unlisted: - # The check runs before every run; that the store cannot list its tables + # The check runs before every run; that the database cannot list its tables # is reported once per node object and attached database. if self._unlisted_store_reported: found = StaleTables(tables=found.tables) @@ -1102,7 +1102,7 @@ def _refuse_log_of_another_data_context(self) -> None: account_for_stale_tables( found, delivers_side_effects=True, current_context=current ) - except StoreError as exc: + except OrcapodDatabaseError as exc: exc.add_context(node_label=self.label) raise self._current_log_checked = True @@ -1327,7 +1327,7 @@ def run(self, *, load_only: bool = False) -> StreamView: Raises: SideEffectLogDataContextError: As for ``execute``. - StoreWriteError: If committing the invocation log fails. + OrcapodDatabaseWriteError: If committing the invocation log fails. """ import sys @@ -1353,14 +1353,14 @@ def _run_alone(self, *, load_only: bool) -> None: # Committed as at the end of a job run (PipelineJob._finish_run). try: database.flush() - except StoreReadError: + except OrcapodDatabaseReadError: # A stored table could not be read: report the read failure itself. # The run raised, so it leaves the node no rows. self.clear_cache() raise except Exception as exc: self.clear_cache() - raise StoreWriteError(database.base_path, flush=True) from exc + raise OrcapodDatabaseWriteError(database.base_path, flush=True) from exc except BaseException: # An interruption leaves the commit's outcome unknown: the node holds no # run, so the next run runs and commits again (ITL-1121). diff --git a/src/orcapod/types.py b/src/orcapod/types.py index 965abbfa2..b720a1578 100644 --- a/src/orcapod/types.py +++ b/src/orcapod/types.py @@ -357,10 +357,10 @@ class NodeConfig: Attributes: is_result_ephemeral: ``None`` inherits the default (``False``). ``True`` writes new computation results to the pipeline-scoped - ephemeral store instead of the persistent result database. + ephemeral database instead of the persistent result database. Persistent cache hits are still served when available. Raises ``RuntimeError`` at execution time if ``True`` but no ephemeral - store has been injected via ``set_ephemeral_store()``. + database has been injected via ``set_ephemeral_database()``. """ is_result_ephemeral: bool | None = None diff --git a/src/orcapod/utils/store_errors.py b/src/orcapod/utils/database_errors.py similarity index 66% rename from src/orcapod/utils/store_errors.py rename to src/orcapod/utils/database_errors.py index d297f024d..040bb5400 100644 --- a/src/orcapod/utils/store_errors.py +++ b/src/orcapod/utils/database_errors.py @@ -1,6 +1,6 @@ -"""Helpers that keep a ``StoreError`` intact on its way out of a run. +"""Helpers that keep a ``OrcapodDatabaseError`` intact on its way out of a run. -A failed store read or write always fails the run (ITL-878, ITL-884), and so does a +A failed database read or write always fails the run (ITL-878, ITL-884), and so does a stored row that cannot be decoded as its class (``TypeIdentityError``, ITL-881). User callbacks that run while the error propagates (a post-run hook, an observer callback) may fail too; their failure is logged and noted on the error, never raised in its place. @@ -13,7 +13,7 @@ from contextlib import contextmanager from typing import TYPE_CHECKING -from orcapod.errors import StoreError, TypeIdentityError +from orcapod.errors import OrcapodDatabaseError, TypeIdentityError if TYPE_CHECKING: from orcapod.protocols.database_protocols import ArrowDatabaseProtocol @@ -22,75 +22,79 @@ def note_callback_failure( - store_error: StoreError | TypeIdentityError, during: str, failure: Exception + database_error: OrcapodDatabaseError | TypeIdentityError, + during: str, + failure: Exception, ) -> None: - """Log a callback failure and record it on the propagating store error. + """Log a callback failure and record it on the propagating database error. Args: - store_error: The store error (or identity error) on its way out of the run. + database_error: The database error (or identity error) on its way out of the run. during: What was running, for the log and the note, such as ``"Observer on_data_crash"``. failure: The exception the callback raised. """ logger.warning( - "%s raised while a store failure was propagating, which is raised " + "%s raised while a database failure was propagating, which is raised " + "instead: %s", during, failure, exc_info=failure, ) - store_error.add_note( + database_error.add_note( f"{during} also raised while this error propagated: " + f"{type(failure).__name__}: {failure}" ) -def store_error_of(error: BaseException) -> StoreError | TypeIdentityError | None: - """Return the store error a failure carries: itself, or the first inside its group. +def database_error_of( + error: BaseException, +) -> OrcapodDatabaseError | TypeIdentityError | None: + """Return the database error a failure carries: itself, or the first inside its group. Args: error: A run's failure, possibly an exception group from a task group. Returns: - The ``StoreError`` or ``TypeIdentityError`` that is ``error`` or is the first + The ``OrcapodDatabaseError`` or ``TypeIdentityError`` that is ``error`` or is the first one found inside it, or ``None`` when it carries none. """ - if isinstance(error, (StoreError, TypeIdentityError)): + if isinstance(error, (OrcapodDatabaseError, TypeIdentityError)): return error if isinstance(error, BaseExceptionGroup): for inner in error.exceptions: - found = store_error_of(inner) + found = database_error_of(inner) if found is not None: return found return None @contextmanager -def keep_store_error( - store_error: StoreError | TypeIdentityError | None, during: str +def keep_database_error( + database_error: OrcapodDatabaseError | TypeIdentityError | None, during: str ) -> Iterator[None]: - """Run a callback so that it cannot replace a propagating store error. + """Run a callback so that it cannot replace a propagating database error. - With ``store_error`` set, an ``Exception`` the block raises is logged and noted on - ``store_error`` (see ``note_callback_failure``) and not re-raised, so the caller's - ``raise`` propagates the store error. With ``store_error=None`` the block runs + With ``database_error`` set, an ``Exception`` the block raises is logged and noted on + ``database_error`` (see ``note_callback_failure``) and not re-raised, so the caller's + ``raise`` propagates the database error. With ``database_error=None`` the block runs unchanged and its exceptions propagate. Args: - store_error: The store error (or identity error) on its way out, or ``None`` + database_error: The database error (or identity error) on its way out, or ``None`` when none is. during: What the block runs, for the log and the note. Yields: Nothing; the block runs inside the context. """ - if store_error is None: + if database_error is None: yield return try: yield except Exception as failure: - note_callback_failure(store_error, during, failure) + note_callback_failure(database_error, during, failure) def commit_best_effort(run_error: BaseException, commit: Callable[[], None]) -> None: diff --git a/superpowers/plans/ITL-1087-database-renames.md b/superpowers/plans/ITL-1087-database-renames.md new file mode 100644 index 000000000..965eb53fc --- /dev/null +++ b/superpowers/plans/ITL-1087-database-renames.md @@ -0,0 +1,16 @@ +# ITL-1087 plan: rename remaining public "store" names to "database" + +Mechanical rename, no behaviour change (skip-brainstorming: names decided by developer Nov 13-14; coordinator brief). + +Renames: set_ephemeral_store->set_ephemeral_database (param store->database; ephemeral_result_store attr and +_ephemeral_store_in_use follow); StoreError->OrcapodDatabaseError; StoreWriteError->OrcapodDatabaseWriteError; +StoreReadError->OrcapodDatabaseReadError; StoreContentWarning->OrcapodDatabaseContentWarning; +describe_store_table->describe_database_table; store_path->database_path; utils/store_errors.py->utils/database_errors.py +(store_error_of, keep_store_error, note_callback_failure param store_error follow). + +Steps: 1 git mv + perl rename over src/tests/docs/README/CLAUDE.md/.zed/rules/design_issues (not superpowers, CHANGELOG.md, +old changelog.d). 2 re-grep, list kept 'store' identifiers. 3 changelog.d/ITL-1087.md. 4 targeted tests, ruff, basedpyright, +golden. 5 merge origin/main, push, draft PR, CI, one signed @coderabbitai review. + +Notes: (progress notes below) +Progress: rename applied (scripted), wording pass done; next tests, changelog, PR diff --git a/superpowers/plans/itl-1147-ephemeral-database-side-effect-nodes.md b/superpowers/plans/itl-1147-ephemeral-database-side-effect-nodes.md new file mode 100644 index 000000000..af8d82f5d --- /dev/null +++ b/superpowers/plans/itl-1147-ephemeral-database-side-effect-nodes.md @@ -0,0 +1,13 @@ +# ITL-1147: PipelineJob.set_ephemeral_database with side-effect nodes + +Bug: `PipelineBase.set_ephemeral_database` calls the setter on every non-source node; +`SideEffectJobNode` (live and loaded) has none -> AttributeError. + +Fix (smallest): pipeline setter skips `source` and `side_effect` node types; no ephemeral +storage for side-effect nodes. Function and operator nodes still receive the database. + +Tests (public interface, tests/test_pipeline/test_ephemeral_database_side_effects.py): +live job and loaded job with a side-effect node, set to a db and to None; function and +operator nodes receive it (read `ephemeral_result_database` where public). + +Also: `### Fixed` bullet in changelog.d/ITL-1087.md; `Fixes ITL-1147` in PR body. diff --git a/tests/test_arrays/data/make_v02_array_store.py b/tests/test_arrays/data/make_v02_array_store.py index 782ad74c9..783dfd59d 100644 --- a/tests/test_arrays/data/make_v02_array_store.py +++ b/tests/test_arrays/data/make_v02_array_store.py @@ -1,6 +1,6 @@ """Regenerate ``v02_array_store.tar.gz`` with Orcapod v0.2 (ITL-888). -The store is kept as an archive because Delta table paths contain ``:``, which +The database is kept as an archive because Delta table paths contain ``:``, which some file systems refuse. Run from the repository root with a v0.2 source tree first on the path: diff --git a/tests/test_arrays/data/make_v02_full_store.py b/tests/test_arrays/data/make_v02_full_store.py index 3f1a0dc53..e21f02e22 100644 --- a/tests/test_arrays/data/make_v02_full_store.py +++ b/tests/test_arrays/data/make_v02_full_store.py @@ -1,6 +1,6 @@ """Regenerate ``v02_full_store.tar.gz`` with Orcapod v0.2.0 (ITL-954). -The store is kept as an archive because Delta table paths contain ``:``, which +The database is kept as an archive because Delta table paths contain ``:``, which some file systems refuse. Run from the repository root in an environment that has Orcapod v0.2.0 installed diff --git a/tests/test_arrays/test_array_pipeline_stores.py b/tests/test_arrays/test_array_pipeline_stores.py index fc311a2fe..1060fa197 100644 --- a/tests/test_arrays/test_array_pipeline_stores.py +++ b/tests/test_arrays/test_array_pipeline_stores.py @@ -112,7 +112,7 @@ def split(x: np.ndarray) -> list[np.ndarray]: # pyright: ignore[reportMissingTy ], ) def test_pod_output_list_of_arrays(tmp_path: Path, store: str, runner: str) -> None: - """A pod output ``list[np.ndarray]`` is stored and read back on every store and runner.""" + """A pod output ``list[np.ndarray]`` is stored and read back on every database and runner.""" source = DictSource( data=[{"k": k, "x": ARRAYS[k]} for k in NUMERIC], tag_columns=["k"], @@ -142,7 +142,7 @@ def first_elements(x: list[np.ndarray]) -> list[float]: # pyright: ignore[repor def test_batch_and_group_by_over_an_array_column( tmp_path: Path, operator: str, store: str, runner: str ) -> None: - """``Batch`` and ``GroupBy`` over an array column feed a pod on every store and runner.""" + """``Batch`` and ``GroupBy`` over an array column feed a pod on every database and runner.""" source = DictSource( data=[ {"g": "a" if i < 2 else "b", "k": f"r{i}", "x": np.arange(3.0) + i} diff --git a/tests/test_arrays/test_array_v02_store.py b/tests/test_arrays/test_array_v02_store.py index 8706caa41..acdafb168 100644 --- a/tests/test_arrays/test_array_v02_store.py +++ b/tests/test_arrays/test_array_v02_store.py @@ -3,7 +3,7 @@ The array format changed in v0.3 without a migration (ITL-888), and so did every Arrow-derived hash (starfix hash format 2, label ``arrow_v1``, ITL-910). v0.3 stores every table at a new layout (``pdb_v2``, ``rdb_v2``), with the data context of each row in ``__data_context`` -(ITL-944). The tables a v0.2 store holds are at the old layout: v0.3 does not refuse the +(ITL-944). The tables a v0.2 database holds are at the old layout: v0.3 does not refuse the store and does not read those tables as results. It computes every input again into its own tables and leaves what v0.2 wrote byte-for-byte in place for a later migration (ITL-913, ITL-914). diff --git a/tests/test_arrays/test_array_value_format.py b/tests/test_arrays/test_array_value_format.py index a79ff5a28..2cc177a1c 100644 --- a/tests/test_arrays/test_array_value_format.py +++ b/tests/test_arrays/test_array_value_format.py @@ -568,7 +568,7 @@ def test_a_cell_with_trailing_bytes_is_refused() -> None: def test_padding_bytes_are_stored_as_zero_and_do_not_change_the_hash() -> None: - """Garbage in structured padding never reaches the store, and is not part of the hash.""" + """Garbage in structured padding never reaches the database, and is not part of the hash.""" aligned = np.dtype([("a", "i1"), ("b", " Any: def write_store(store_root: Path) -> None: - """Run the job and fill the cached source on a Delta store at ``store_root``.""" + """Run the job and fill the cached source on a Delta database at ``store_root``.""" from orcapod.databases import DeltaTableDatabase store = DeltaTableDatabase(base_path=store_root) diff --git a/tests/test_arrays/v02_store_pods.py b/tests/test_arrays/v02_store_pods.py index bcfe0381e..560b2eae9 100644 --- a/tests/test_arrays/v02_store_pods.py +++ b/tests/test_arrays/v02_store_pods.py @@ -24,7 +24,7 @@ def build_job(store_root: Path) -> Any: - """Build the fixture's pipeline on a Delta store at ``store_root``.""" + """Build the fixture's pipeline on a Delta database at ``store_root``.""" from orcapod import PipelineJob, function_pod from orcapod.databases import DeltaTableDatabase from orcapod.operators import DropDataColumns diff --git a/tests/test_core/data_function/test_cached_data_function.py b/tests/test_core/data_function/test_cached_data_function.py index 7fb2f689d..ae5cf6942 100644 --- a/tests/test_core/data_function/test_cached_data_function.py +++ b/tests/test_core/data_function/test_cached_data_function.py @@ -521,7 +521,7 @@ def test_most_recent_wins(self, inner_pf, input_data): time.sleep(0.01) # ensure distinct timestamps cpf.call(input_data, skip_cache_lookup=True) - # Two records in the store + # Two records in the database all_records = db.get_all_records(cpf.record_path) assert all_records is not None assert all_records.num_rows == 2 diff --git a/tests/test_core/function_pod/test_cached_function_pod_job.py b/tests/test_core/function_pod/test_cached_function_pod_job.py index e14541b98..533f33fab 100644 --- a/tests/test_core/function_pod/test_cached_function_pod_job.py +++ b/tests/test_core/function_pod/test_cached_function_pod_job.py @@ -164,7 +164,7 @@ def test_row_is_computed_once_and_stored_in_both_caches( assert pod_rows is not None and pod_rows.num_rows == 2 assert sorted(counter.calls) == [2, 5] - # A second run over the same store computes nothing. + # A second run over the same database computes nothing. run_job(cached_pod, store, orchestrator) assert sorted(counter.calls) == [2, 5] @@ -227,7 +227,7 @@ def test_saved_job_equals_the_uncached_pods_and_loads( plain_saved = saved(plain, tmp_path / "plain.json") cached_saved = saved(job, tmp_path / "cached.json") - # Fields that differ between any two saves: the run id and the store's location. + # Fields that differ between any two saves: the run id and the database's location. for doc in (plain_saved, cached_saved): doc["run"].pop("run_id") doc["bindings"].pop("database") diff --git a/tests/test_core/function_pod/test_ephemeral_result.py b/tests/test_core/function_pod/test_ephemeral_result.py index b37fd416a..a5135b852 100644 --- a/tests/test_core/function_pod/test_ephemeral_result.py +++ b/tests/test_core/function_pod/test_ephemeral_result.py @@ -1,4 +1,4 @@ -"""Tests for FunctionJobNode ephemeral result store — ITL-507.""" +"""Tests for FunctionJobNode ephemeral result database — ITL-507.""" from __future__ import annotations @@ -81,24 +81,24 @@ def _make_node( # --------------------------------------------------------------------------- -# Task 4 test: no-op set_ephemeral_store on blueprint node classes +# Task 4 test: no-op set_ephemeral_database on blueprint node classes # --------------------------------------------------------------------------- class TestNoOpSetEphemeralStore: - def test_function_node_has_set_ephemeral_store(self): - """FunctionNode (blueprint) must have set_ephemeral_store as a no-op.""" + def test_function_node_has_set_ephemeral_database(self): + """FunctionNode (blueprint) must have set_ephemeral_database as a no-op.""" stream = _make_stream([{"id": 0, "x": 10}]) pf = PythonDataFunction(double, output_keys="result") pod = FunctionPod(pf) node = FunctionNode(function_pod=pod, input_stream=stream) store = InMemoryArrowDatabase() # Must not raise - node.set_ephemeral_store(store) - node.set_ephemeral_store(None) + node.set_ephemeral_database(store) + node.set_ephemeral_database(None) - def test_operator_node_has_set_ephemeral_store(self): - """OperatorNode (blueprint) must have set_ephemeral_store as a no-op.""" + def test_operator_node_has_set_ephemeral_database(self): + """OperatorNode (blueprint) must have set_ephemeral_database as a no-op.""" from orcapod.core.nodes import OperatorNode from orcapod.core.operators import Join @@ -108,32 +108,32 @@ def test_operator_node_has_set_ephemeral_store(self): node = OperatorNode(operator=op, input_streams=(stream_a, stream_b)) store = InMemoryArrowDatabase() # Must not raise - node.set_ephemeral_store(store) - node.set_ephemeral_store(None) + node.set_ephemeral_database(store) + node.set_ephemeral_database(None) # --------------------------------------------------------------------------- -# Task 5 test: FunctionJobNode set_ephemeral_store real override +# Task 5 test: FunctionJobNode set_ephemeral_database real override # --------------------------------------------------------------------------- class TestSetEphemeralStore: - def test_set_ephemeral_store_assigns_store(self): - """set_ephemeral_store(store) assigns the ephemeral_result_store attribute.""" + def test_set_ephemeral_database_assigns_store(self): + """set_ephemeral_database(store) assigns the ephemeral_result_database attribute.""" stream = _make_stream([{"id": 0, "x": 10}]) node, _ = _make_node(stream) store = InMemoryArrowDatabase() - node.set_ephemeral_store(store) - assert node.ephemeral_result_store is store + node.set_ephemeral_database(store) + assert node.ephemeral_result_database is store - def test_set_ephemeral_store_none_detaches(self): - """set_ephemeral_store(None) removes the ephemeral store.""" + def test_set_ephemeral_database_none_detaches(self): + """set_ephemeral_database(None) removes the ephemeral database.""" stream = _make_stream([{"id": 0, "x": 10}]) node, _ = _make_node(stream) store = InMemoryArrowDatabase() - node.set_ephemeral_store(store) - node.set_ephemeral_store(None) - assert node.ephemeral_result_store is None + node.set_ephemeral_database(store) + node.set_ephemeral_database(None) + assert node.ephemeral_result_database is None # --------------------------------------------------------------------------- @@ -165,7 +165,7 @@ def test_is_ephemeral_true_written_to_pipeline_db(self): pipeline_db = InMemoryArrowDatabase() ephemeral_store = InMemoryArrowDatabase() node, _ = _make_node(stream, pipeline_db=pipeline_db, is_result_ephemeral=True) - node.set_ephemeral_store(ephemeral_store) + node.set_ephemeral_database(ephemeral_store) results = node.execute(stream) assert len(results) == 1 @@ -193,7 +193,7 @@ def test_ephemeral_false_unchanged(self): assert vals == {0: 20, 1: 40} def test_ephemeral_result_written_to_memory_not_persistent_db(self): - """With is_result_ephemeral=True, persistent DB has no result rows; ephemeral store does.""" + """With is_result_ephemeral=True, persistent DB has no result rows; ephemeral database does.""" stream = _make_stream([{"id": 0, "x": 10}]) pipeline_db = InMemoryArrowDatabase() result_db = InMemoryArrowDatabase() @@ -206,7 +206,7 @@ def test_ephemeral_result_written_to_memory_not_persistent_db(self): result_database=result_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(ephemeral_store) + node.set_ephemeral_database(ephemeral_store) results = node.execute(stream) assert len(results) == 1 @@ -221,7 +221,7 @@ def test_ephemeral_result_written_to_memory_not_persistent_db(self): assert result_db.get_all_records(node._cached_function_pod.record_path) is None def test_within_session_ephemeral_hit(self): - """Same node called twice: second call hits ephemeral store — no recomputation.""" + """Same node called twice: second call hits ephemeral database — no recomputation.""" call_count = {"n": 0} def counting_double(x: int) -> int: @@ -239,7 +239,7 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(ephemeral_store) + node.set_ephemeral_database(ephemeral_store) node.execute(stream) assert call_count["n"] == 1 @@ -260,7 +260,7 @@ def counting_double(x: int) -> int: stream = _make_stream([{"id": 0, "x": 10}]) pipeline_db = InMemoryArrowDatabase() - # Session 1: execute with ephemeral store + # Session 1: execute with ephemeral database pf = PythonDataFunction(counting_double, output_keys="result") pod = FunctionPod(pf) node = FunctionJobNode( @@ -269,11 +269,11 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(InMemoryArrowDatabase()) + node.set_ephemeral_database(InMemoryArrowDatabase()) node.execute(stream) assert call_count["n"] == 1 - # Session 2: fresh in-memory node with a fresh ephemeral store + # Session 2: fresh in-memory node with a fresh ephemeral database pf2 = PythonDataFunction(counting_double, output_keys="result") pod2 = FunctionPod(pf2) node2 = FunctionJobNode( @@ -282,12 +282,12 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node2.node_config = NodeConfig(is_result_ephemeral=True) - node2.set_ephemeral_store(InMemoryArrowDatabase()) # fresh store + node2.set_ephemeral_database(InMemoryArrowDatabase()) # fresh database node2.execute(stream) assert call_count["n"] == 2 # recomputed def test_persistent_hit_served_when_ephemeral_true(self): - """A persistent result is still served from cache when ephemeral store is also set.""" + """A persistent result is still served from cache when ephemeral database is also set.""" call_count = {"n": 0} def counting_double(x: int) -> int: @@ -303,8 +303,8 @@ def counting_double(x: int) -> int: input_stream=stream, pipeline_database=pipeline_db, ) - # Attach an ephemeral store — must NOT break persistent reads - node.set_ephemeral_store(InMemoryArrowDatabase()) + # Attach an ephemeral database — must NOT break persistent reads + node.set_ephemeral_database(InMemoryArrowDatabase()) # Run 1: writes to persistent DB (is_result_ephemeral=False) results = node.execute(stream) @@ -349,7 +349,7 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node_e.node_config = NodeConfig(is_result_ephemeral=True) - node_e.set_ephemeral_store(ephemeral_store) + node_e.set_ephemeral_database(ephemeral_store) node_e.execute(stream_b) # Verify id=1 result is in ephemeral_store (not persistent) @@ -367,7 +367,7 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node_both.node_config = NodeConfig(is_result_ephemeral=True) - node_both.set_ephemeral_store(ephemeral_store) + node_both.set_ephemeral_database(ephemeral_store) node_both._cached_output_datas.clear() results = node_both.execute(stream_both) @@ -411,7 +411,7 @@ def counting_double(x: int) -> int: input_stream=stream, pipeline_database=pipeline_db, ) - node.set_ephemeral_store(ephemeral_store) + node.set_ephemeral_database(ephemeral_store) # Step 1: Write the persistent result (IS_EPHEMERAL=False) node.execute(stream) @@ -492,7 +492,7 @@ def counting_double(x: int) -> int: class TestEphemeralWritePath: def test_store_not_assigned_raises(self): - """is_result_ephemeral=True but ephemeral_result_store=None raises RuntimeError.""" + """is_result_ephemeral=True but ephemeral_result_database=None raises RuntimeError.""" stream = _make_stream([{"id": 0, "x": 10}]) pod = _make_pod() pipeline_db = InMemoryArrowDatabase() @@ -502,7 +502,7 @@ def test_store_not_assigned_raises(self): pipeline_database=pipeline_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - # set_ephemeral_store never called → ephemeral_result_store is None + # set_ephemeral_database never called → ephemeral_result_database is None with pytest.raises(RuntimeError, match="is_result_ephemeral=True"): node.execute(stream, error_policy="fail_fast") @@ -519,7 +519,7 @@ def test_ephemeral_only_node(self): pipeline_database=pipeline_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(ephemeral_store) + node.set_ephemeral_database(ephemeral_store) results = node.execute(stream) assert len(results) == 1 assert results[0][1].as_dict()["result"] == 20 @@ -544,11 +544,11 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node1.node_config = NodeConfig(is_result_ephemeral=True) - node1.set_ephemeral_store(InMemoryArrowDatabase()) + node1.set_ephemeral_database(InMemoryArrowDatabase()) node1.execute(stream) assert call_count["n"] == 1 - # Session 2: fresh store → miss → recompute + # Session 2: fresh database → miss → recompute pf2 = PythonDataFunction(counting_double, output_keys="result") pod2 = FunctionPod(pf2) ephemeral2 = InMemoryArrowDatabase() @@ -558,11 +558,11 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node2.node_config = NodeConfig(is_result_ephemeral=True) - node2.set_ephemeral_store(ephemeral2) + node2.set_ephemeral_database(ephemeral2) node2.execute(stream) assert call_count["n"] == 2 - # Session 3: same ephemeral store as session 2 → hit → no recompute + # Session 3: same ephemeral database as session 2 → hit → no recompute pf3 = PythonDataFunction(counting_double, output_keys="result") pod3 = FunctionPod(pf3) node3 = FunctionJobNode( @@ -571,7 +571,7 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node3.node_config = NodeConfig(is_result_ephemeral=True) - node3.set_ephemeral_store(ephemeral2) # reuse session 2's store + node3.set_ephemeral_database(ephemeral2) # reuse session 2's store node3._cached_output_datas.clear() node3.execute(stream) assert call_count["n"] == 2 # NOT recomputed @@ -594,7 +594,7 @@ def counting_double(x: int) -> int: stream = _make_stream([{"id": 0, "x": 10}]) pipeline_db = InMemoryArrowDatabase() - # Session 1: compute with ephemeral store 1 → pipeline DB gets index-0 record + # Session 1: compute with ephemeral database 1 → pipeline DB gets index-0 record pf = PythonDataFunction(counting_double, output_keys="result") pod = FunctionPod(pf) node1 = FunctionJobNode( @@ -603,11 +603,11 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node1.node_config = NodeConfig(is_result_ephemeral=True) - node1.set_ephemeral_store(InMemoryArrowDatabase()) + node1.set_ephemeral_database(InMemoryArrowDatabase()) node1.execute(stream) assert call_count["n"] == 1 - # Session 2: fresh ephemeral store → cross-session miss → recompute → index-1 record written + # Session 2: fresh ephemeral database → cross-session miss → recompute → index-1 record written pf2 = PythonDataFunction(counting_double, output_keys="result") pod2 = FunctionPod(pf2) ephemeral2 = InMemoryArrowDatabase() @@ -617,7 +617,7 @@ def counting_double(x: int) -> int: pipeline_database=pipeline_db, ) node2.node_config = NodeConfig(is_result_ephemeral=True) - node2.set_ephemeral_store(ephemeral2) + node2.set_ephemeral_database(ephemeral2) node2.execute(stream) assert call_count["n"] == 2 # recomputed once due to cross-session miss @@ -647,8 +647,8 @@ def counting_double(x: int) -> int: class TestPipelineInjectsStore: - def test_pipeline_job_set_ephemeral_store_propagates(self): - """PipelineJob.set_ephemeral_store propagates to all compiled nodes.""" + def test_pipeline_job_set_ephemeral_database_propagates(self): + """PipelineJob.set_ephemeral_database propagates to all compiled nodes.""" from orcapod.pipeline.job import PipelineJob # A job's input must be a source (ITL-1126: a raw stream has no system tags). @@ -663,16 +663,16 @@ def test_pipeline_job_set_ephemeral_store_propagates(self): pod(stream) ephemeral_store = InMemoryArrowDatabase() - job.set_ephemeral_store(ephemeral_store) + job.set_ephemeral_database(ephemeral_store) - # Every function node in the compiled pipeline should now have the store + # Every function node in the compiled pipeline should now have the database for label, node in job.function_pods.items(): - assert node.ephemeral_result_store is ephemeral_store, ( - f"Node '{label}' did not receive the ephemeral store" + assert node.ephemeral_result_database is ephemeral_store, ( + f"Node '{label}' did not receive the ephemeral database" ) - def test_pipeline_job_set_ephemeral_store_none_detaches(self): - """PipelineJob.set_ephemeral_store(None) detaches the store from all nodes.""" + def test_pipeline_job_set_ephemeral_database_none_detaches(self): + """PipelineJob.set_ephemeral_database(None) detaches the database from all nodes.""" from orcapod.pipeline.job import PipelineJob # A job's input must be a source (ITL-1126: a raw stream has no system tags). @@ -686,12 +686,12 @@ def test_pipeline_job_set_ephemeral_store_none_detaches(self): pod(stream) store = InMemoryArrowDatabase() - job.set_ephemeral_store(store) - job.set_ephemeral_store(None) + job.set_ephemeral_database(store) + job.set_ephemeral_database(None) for label, node in job.function_pods.items(): - assert node.ephemeral_result_store is None, ( - f"Node '{label}' ephemeral store was not detached" + assert node.ephemeral_result_database is None, ( + f"Node '{label}' ephemeral database was not detached" ) @@ -810,7 +810,7 @@ def test_ephemeral_only_node_no_persistent_db(self): pipeline_database=pipeline_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(ephemeral_store) + node.set_ephemeral_database(ephemeral_store) results = node.execute(stream) assert len(results) == 2 @@ -826,7 +826,7 @@ def test_ephemeral_only_node_no_persistent_db(self): class TestAsyncEphemeralExecution: @pytest.mark.asyncio async def test_async_execute_ephemeral_happy_path(self): - """async_execute with is_result_ephemeral=True writes to ephemeral store and emits results.""" + """async_execute with is_result_ephemeral=True writes to ephemeral database and emits results.""" stream = _make_stream([{"id": 0, "x": 5}, {"id": 1, "x": 10}]) pipeline_db = InMemoryArrowDatabase() ephemeral_store = InMemoryArrowDatabase() @@ -839,7 +839,7 @@ async def test_async_execute_ephemeral_happy_path(self): pipeline_database=pipeline_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(ephemeral_store) + node.set_ephemeral_database(ephemeral_store) input_ch = Channel(buffer_size=16) output_ch = Channel(buffer_size=16) @@ -855,7 +855,7 @@ async def test_async_execute_ephemeral_happy_path(self): values = sorted(data.as_dict()["result"] for _, data in results) assert values == [10, 20] - # Result records must be in the ephemeral store, not the persistent store + # Result records must be in the ephemeral database, not the persistent database eph_records = ephemeral_store.get_all_records( node._ephemeral_cached_pod.record_path, ) @@ -864,7 +864,7 @@ async def test_async_execute_ephemeral_happy_path(self): @pytest.mark.asyncio async def test_async_process_data_internal_raises_when_no_store(self): - """_async_process_data_internal raises RuntimeError when is_result_ephemeral=True but no store.""" + """_async_process_data_internal raises RuntimeError when is_result_ephemeral=True but no database.""" stream = _make_stream([{"id": 0, "x": 5}]) pipeline_db = InMemoryArrowDatabase() @@ -876,7 +876,7 @@ async def test_async_process_data_internal_raises_when_no_store(self): pipeline_database=pipeline_db, ) node.node_config = NodeConfig(is_result_ephemeral=True) - # set_ephemeral_store never called → _ephemeral_cached_pod is None + # set_ephemeral_database never called → _ephemeral_cached_pod is None tag, data = next(iter(stream.iter_data())) with pytest.raises(RuntimeError, match="is_result_ephemeral=True"): diff --git a/tests/test_core/function_pod/test_function_node_arrow_types.py b/tests/test_core/function_pod/test_function_node_arrow_types.py index e2253d91f..1cd7bf732 100644 --- a/tests/test_core/function_pod/test_function_node_arrow_types.py +++ b/tests/test_core/function_pod/test_function_node_arrow_types.py @@ -1,7 +1,7 @@ """A function node in a job keeps the Arrow type of every tag column (ITL-928). A function pod never touches tags, so a function node's tag columns must have the -input's exact Arrow types (large-normalized) whatever the store, the orchestrator and +input's exact Arrow types (large-normalized) whatever the database, the orchestrator and the read: ``as_table()``, ``iter_data()``, ``get_all_records()``, a fresh job on the same store, and a re-run. Before ITL-928 a node's ``as_table()`` rebuilt its table from Python values typed by the node's Python output schema, so an ``int8`` tag came back ``int64`` @@ -115,7 +115,7 @@ def _expected(name: str) -> pa.DataType: def _new_store(kind: str, tmp_path: pathlib.Path): - """A store of the given kind (``memory`` or ``delta``); a Delta store under ``tmp_path``.""" + """A store of the given kind (``memory`` or ``delta``); a Delta database under ``tmp_path``.""" if kind == "delta": return DeltaTableDatabase(tmp_path / "store") return InMemoryArrowDatabase() @@ -206,7 +206,7 @@ def test_join_after_a_function_node_on_a_typed_tag(name: str) -> None: def test_as_table_mixing_stored_and_new_rows_keeps_tag_type() -> None: - """Rows served from the store and rows computed now make one correctly typed table.""" + """Rows served from the database and rows computed now make one correctly typed table.""" store = InMemoryArrowDatabase() first = _build(store, _source_table("uint32").slice(0, 2)) first.run() @@ -221,7 +221,7 @@ def test_as_table_mixing_stored_and_new_rows_keeps_tag_type() -> None: # --------------------------------------------------------------------------- -# Reads from the store: get_all_records() and a fresh job (part B on Delta) +# Reads from the database: get_all_records() and a fresh job (part B on Delta) # --------------------------------------------------------------------------- @@ -230,7 +230,7 @@ def test_as_table_mixing_stored_and_new_rows_keeps_tag_type() -> None: def test_store_read_back_keeps_tag_type( name: str, store: str, tmp_path: pathlib.Path ) -> None: - """Reads from the store (``get_all_records``, a fresh job's run) keep the tag type.""" + """Reads from the database (``get_all_records``, a fresh job's run) keep the tag type.""" database = _new_store(store, tmp_path) table = _source_table(name) _build(database, table).run() @@ -244,7 +244,7 @@ def test_store_read_back_keeps_tag_type( records = fresh.get_all_records() assert records is not None assert records.schema.field("k").type == _expected(name) - # The node shows rows once its run has read them from the store (ITL-1095). + # The node shows rows once its run has read them from the database (ITL-1095). fresh_job.run() tag, _ = next(iter(fresh.iter_data())) assert tag.as_table().schema.field("k").type == _expected(name) @@ -282,9 +282,9 @@ def test_new_rows_over_a_store_written_before_the_fix_raise_naming_the_column( ) -> None: """Rows of one node never make a column of a type none of them has. - A Delta store written before ITL-928 gives a ``uint32`` tag back as ``int32``: + A Delta database written before ITL-928 gives a ``uint32`` tag back as ``int32``: rows served from it and rows computed now disagree, and the node says so instead - of picking a type. Such a store must be rebuilt. + of picking a type. Such a database must be rebuilt. """ _build(DeltaTableDatabase(tmp_path), _source_table("uint32").slice(0, 2)).run() _as_written_before_the_fix(tmp_path) @@ -303,7 +303,7 @@ def test_new_rows_over_a_store_written_before_the_fix_raise_naming_the_column( def test_new_rows_over_a_store_written_after_the_fix_keep_the_tag_type( tmp_path: pathlib.Path, ) -> None: - """The same run over a store this build wrote: one table, typed as the input.""" + """The same run over a database this build wrote: one table, typed as the input.""" _build(DeltaTableDatabase(tmp_path), _source_table("uint32").slice(0, 2)).run() job = _build(DeltaTableDatabase(tmp_path), _source_table("uint32")) job.run() diff --git a/tests/test_core/function_pod/test_output_conformance.py b/tests/test_core/function_pod/test_output_conformance.py index aa7d90043..4c0fcc168 100644 --- a/tests/test_core/function_pod/test_output_conformance.py +++ b/tests/test_core/function_pod/test_output_conformance.py @@ -272,7 +272,7 @@ def _half_on_even(x: int) -> int: def _store(kind: str, tmp_path: Path) -> Any: - """A Delta store under ``tmp_path`` for ``"delta"``, else an in-memory store.""" + """A Delta database under ``tmp_path`` for ``"delta"``, else an in-memory store.""" return ( DeltaTableDatabase(tmp_path / "db") if kind == "delta" @@ -530,7 +530,7 @@ def test_output_as_dict_matches_as_table( class TestRejectionInPipelineJob: - """Every decided rejection is a row failure on both orchestrators and both stores.""" + """Every decided rejection is a row failure on both orchestrators and both databases.""" @pytest.mark.parametrize("orchestrator_cls", ORCHESTRATORS) @pytest.mark.parametrize("store_kind", STORES) diff --git a/tests/test_core/function_pod/test_pandas_dataframe_order.py b/tests/test_core/function_pod/test_pandas_dataframe_order.py index 01e68ddb0..c642ae1b8 100644 --- a/tests/test_core/function_pod/test_pandas_dataframe_order.py +++ b/tests/test_core/function_pod/test_pandas_dataframe_order.py @@ -134,7 +134,7 @@ def _run( @pytest.mark.parametrize("store_name", list(_STORES)) @pytest.mark.parametrize("orchestrator_name", list(_ORCHESTRATORS)) class TestPandasColumnOrder: - """Column order of a frame in a pipeline, on every store and orchestrator.""" + """Column order of a frame in a pipeline, on every database and orchestrator.""" def test_frames_stored_with_their_order( self, store_name: str, orchestrator_name: str, tmp_path: Path @@ -166,7 +166,7 @@ def test_cached_pod_computes_for_each_order( def test_warm_cache_does_not_serve_another_order( self, store_name: str, orchestrator_name: str, tmp_path: Path ) -> None: - """One order per run on one store, so the second run meets a warm cache.""" + """One order per run on one database, so the second run meets a warm cache.""" store = _STORES[store_name](tmp_path) got: dict[str, str] = {} for key, reverse in (("fwd", False), ("rev", True)): @@ -240,9 +240,9 @@ def test_frame_read_back_from_the_store_keeps_its_cache_key( ) -> None: """A frame that storage reads back in another form is still a cache hit. - The store holds the frame in the form storage restores (``object`` ints as + The database holds the frame in the form storage restores (``object`` ints as ``int64``, ``string[pyarrow]`` as ``string[python]``, ``attrs`` as JSON). A second - run on the same store must not compute the consumer again. The hash-level + run on the same database must not compute the consumer again. The hash-level guarantee, ``hash(read(write(v))) == hash(v)``, is tested in ``tests/test_hashing/test_pandas_value_identity.py``. """ diff --git a/tests/test_core/function_pod/test_pod_stream_interleaved_reads.py b/tests/test_core/function_pod/test_pod_stream_interleaved_reads.py index 468530091..98d633ea2 100644 --- a/tests/test_core/function_pod/test_pod_stream_interleaved_reads.py +++ b/tests/test_core/function_pod/test_pod_stream_interleaved_reads.py @@ -103,7 +103,7 @@ def _other() -> ArrowTableSource: def _pod( function: object, output: str, executor: str, cached: bool ) -> FunctionPod | CachedFunctionPod: - """The pod of ``function``; when ``cached``, storing its results in a fresh store.""" + """The pod of ``function``; when ``cached``, storing its results in a fresh database.""" pod = function_pod( output_keys=output, executor=_ConcurrentExecutor() if executor == "concurrent" else None, diff --git a/tests/test_core/function_pod/test_pod_stream_staleness.py b/tests/test_core/function_pod/test_pod_stream_staleness.py index a82c3d2f7..478183eda 100644 --- a/tests/test_core/function_pod/test_pod_stream_staleness.py +++ b/tests/test_core/function_pod/test_pod_stream_staleness.py @@ -4,7 +4,7 @@ before it ends its own iterator, the same rule as a job node (ITL-1042). An upstream in a chain finishes before its downstream sees the end, so after a full read every stream is at least as new as its upstreams: a second read (``iter_data``, ``as_table`` or ``head``) calls -no function and reads no store. A real change upstream (``clear_cache()``) still makes the +no function and reads no database. A real change upstream (``clear_cache()``) still makes the downstream stale, and its next read recomputes. """ @@ -33,7 +33,7 @@ # Rows of the source, not in tag order. IDS = [3, 1, 4, 0, 2] -# The read methods of the store protocol, counted on every view of an in-memory store. +# The read methods of the database protocol, counted on every view of an in-memory store. STORE_READS = ( "get_all_records", "get_records_with_column_value", diff --git a/tests/test_core/function_pod/test_polars_dataframe_values.py b/tests/test_core/function_pod/test_polars_dataframe_values.py index c18a38515..17e5da0ac 100644 --- a/tests/test_core/function_pod/test_polars_dataframe_values.py +++ b/tests/test_core/function_pod/test_polars_dataframe_values.py @@ -381,7 +381,7 @@ class TestPipelineStores: def test_frames_stored_and_read_back( self, store_name: str, orchestrator_name: str, tmp_path: Path ) -> None: - """Frames written by a job read back exactly, in each store and orchestrator.""" + """Frames written by a job read back exactly, in each database and orchestrator.""" job = PipelineJob(name="frames", database=_STORES[store_name](tmp_path)) with job: frames = make_frame.pod(_int_source(), label="make") @@ -449,8 +449,8 @@ def test_null_struct_and_struct_of_nulls_are_distinct_cache_keys( def test_saved_job_loaded_back_reads_the_frames(self, tmp_path: Path) -> None: """A saved job, loaded in place of the original, reads its stored frames exactly.""" - store_path = tmp_path / "delta" - job = PipelineJob(name="frames", database=DeltaTableDatabase(store_path)) + database_path = tmp_path / "delta" + job = PipelineJob(name="frames", database=DeltaTableDatabase(database_path)) with job: frames = make_frame.pod(_int_source(), label="make") frame_height.pod(frames, label="height") @@ -458,7 +458,9 @@ def test_saved_job_loaded_back_reads_the_frames(self, tmp_path: Path) -> None: saved = tmp_path / "job.json" job.save(str(saved)) - loaded = PipelineJob.load(str(saved), database=DeltaTableDatabase(store_path)) + loaded = PipelineJob.load( + str(saved), database=DeltaTableDatabase(database_path) + ) node = loaded.nodes["make"] _, data_schema = node.output_schema() assert data_schema["df"] is pl.DataFrame @@ -503,7 +505,7 @@ def test_sqlite_store(self, tmp_path: Path) -> None: _assert_exact(got["b"], _frame(3)) -# Runs one pipeline over a Delta store and prints how many times each pod body ran, plus +# Runs one pipeline over a Delta database and prints how many times each pod body ran, plus # the frames read back. The pod bodies append to a marker file so a cache hit (body not # run) is observable from the parent process. _CROSS_PROCESS_SCRIPT = textwrap.dedent( @@ -515,7 +517,7 @@ def test_sqlite_store(self, tmp_path: Path) -> None: from orcapod.databases import DeltaTableDatabase from orcapod.sources import DictSource - store_path, marker = Path(sys.argv[1]), Path(sys.argv[2]) + database_path, marker = Path(sys.argv[1]), Path(sys.argv[2]) @function_pod(output_keys="df") def make_frame(n: int) -> pl.DataFrame: @@ -533,7 +535,7 @@ def frame_height(df: pl.DataFrame) -> int: return df.height source = DictSource([{"k": "a", "n": 1}, {"k": "b", "n": 3}], tag_columns=["k"]) - job = PipelineJob(name="frames", database=DeltaTableDatabase(store_path)) + job = PipelineJob(name="frames", database=DeltaTableDatabase(database_path)) with job: frames = make_frame.pod(source, label="make") frame_height.pod(frames, label="height") @@ -548,10 +550,10 @@ def frame_height(df: pl.DataFrame) -> int: ) -def _run_pipeline_process(store_path: Path, marker: Path) -> dict[str, Any]: +def _run_pipeline_process(database_path: Path, marker: Path) -> dict[str, Any]: """Run the pipeline script in a new process and return its printed JSON.""" result = subprocess.run( - [sys.executable, "-c", _CROSS_PROCESS_SCRIPT, str(store_path), str(marker)], + [sys.executable, "-c", _CROSS_PROCESS_SCRIPT, str(database_path), str(marker)], capture_output=True, text=True, check=False, @@ -561,14 +563,14 @@ def _run_pipeline_process(store_path: Path, marker: Path) -> dict[str, Any]: def test_cached_rerun_in_new_process_hits_the_cache(tmp_path: Path) -> None: - """A second process over the same Delta store recomputes nothing and reads equal frames.""" - store_path, marker = tmp_path / "store", tmp_path / "calls.txt" + """A second process over the same Delta database recomputes nothing and reads equal frames.""" + database_path, marker = tmp_path / "store", tmp_path / "calls.txt" - first = _run_pipeline_process(store_path, marker) + first = _run_pipeline_process(database_path, marker) calls_after_first = marker.read_text().splitlines() assert sorted(calls_after_first) == ["height", "height", "make", "make"] - second = _run_pipeline_process(store_path, marker) + second = _run_pipeline_process(database_path, marker) assert marker.read_text().splitlines() == calls_after_first # nothing re-ran assert second == first assert second["heights"] == {"a": 1, "b": 3} diff --git a/tests/test_core/function_pod/test_readback_datagram_id.py b/tests/test_core/function_pod/test_readback_datagram_id.py index 9b922b8ae..46ae521a3 100644 --- a/tests/test_core/function_pod/test_readback_datagram_id.py +++ b/tests/test_core/function_pod/test_readback_datagram_id.py @@ -1,4 +1,4 @@ -"""A result row read back from the store carries its stored ``__data_id`` (ITL-1060).""" +"""A result row read back from the database carries its stored ``__data_id`` (ITL-1060).""" from __future__ import annotations @@ -91,7 +91,7 @@ def test_a_fresh_job_on_the_store_reads_the_stored_ids() -> None: fresh = build(store) # A node shows rows only after a run (ITL-1095): this run computes nothing, every - # row comes from the store. + # row comes from the database. _ = fresh.run(orchestrator=SyncPipelineOrchestrator()) assert read_back_ids(fresh) == stored diff --git a/tests/test_core/nodes/test_function_job_node_async_execute.py b/tests/test_core/nodes/test_function_job_node_async_execute.py index 72fdd4e13..23ea63be8 100644 --- a/tests/test_core/nodes/test_function_job_node_async_execute.py +++ b/tests/test_core/nodes/test_function_job_node_async_execute.py @@ -436,7 +436,7 @@ def double(value: int) -> int: pod, src, pipeline_database=pipeline_db, result_database=result_db ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(ephemeral_db) + node.set_ephemeral_database(ephemeral_db) results = await _run_node(node) assert len(results) == 2 @@ -450,7 +450,7 @@ def double(value: int) -> int: @pytest.mark.asyncio async def test_ephemeral_missing_store_raises_runtime_error(self): - """Raises RuntimeError when is_result_ephemeral=True but no store set.""" + """Raises RuntimeError when is_result_ephemeral=True but no database set.""" from orcapod.types import NodeConfig def double(value: int) -> int: @@ -464,7 +464,7 @@ def double(value: int) -> int: pod, src, pipeline_database=pipeline_db, result_database=result_db ) node.node_config = NodeConfig(is_result_ephemeral=True) - # Deliberately do NOT call set_ephemeral_store() + # Deliberately do NOT call set_ephemeral_database() with pytest.raises(RuntimeError, match="ephemeral"): await _run_node(node) diff --git a/tests/test_core/nodes/test_function_node_empty_data.py b/tests/test_core/nodes/test_function_node_empty_data.py index b6ca7555b..a6d30559a 100644 --- a/tests/test_core/nodes/test_function_node_empty_data.py +++ b/tests/test_core/nodes/test_function_node_empty_data.py @@ -73,7 +73,7 @@ def test_stored_hash_matches_input_content_hash(self, persistent_node): @pytest.fixture def ephemeral_node(): - """FunctionJobNode with is_result_ephemeral=True and an ephemeral store.""" + """FunctionJobNode with is_result_ephemeral=True and an ephemeral database.""" table = pa.table( { "key": pa.array(["a"], type=pa.large_string()), @@ -89,7 +89,7 @@ def ephemeral_node(): result_database=None, ) node.node_config = NodeConfig(is_result_ephemeral=True) - node.set_ephemeral_store(InMemoryArrowDatabase()) + node.set_ephemeral_database(InMemoryArrowDatabase()) return node @@ -106,13 +106,13 @@ def test_returns_empty_data_tokens_field(self, ephemeral_node): assert hasattr(result, "empty_taginfo_rows") def test_ephemeral_miss_produces_empty_data_token(self, ephemeral_node): - """When the ephemeral store is cleared, _fetch_joined_records emits EmptyData.""" + """When the ephemeral database is cleared, _fetch_joined_records emits EmptyData.""" node = ephemeral_node for tag, data in node._input_stream.iter_data(): node.execute_data(tag, data) - # Simulate ephemeral result expiry by replacing the ephemeral store with empty one - node.set_ephemeral_store(InMemoryArrowDatabase()) + # Simulate ephemeral result expiry by replacing the ephemeral database with empty one + node.set_ephemeral_database(InMemoryArrowDatabase()) result = node._fetch_joined_records() assert result is not None @@ -134,8 +134,8 @@ def test_empty_data_token_hash_matches_output(self, ephemeral_node): tag0, data0 = input_pairs[0] _, output0 = node.execute_data(tag0, data0) - # Clear ephemeral store - node.set_ephemeral_store(InMemoryArrowDatabase()) + # Clear ephemeral database + node.set_ephemeral_database(InMemoryArrowDatabase()) result = node._fetch_joined_records() assert result is not None @@ -153,8 +153,8 @@ def test_empty_data_token_appears_in_loaded_entries(self, ephemeral_node): for tag, data in node._input_stream.iter_data(): node.execute_data(tag, data) - # Simulate expiry by replacing ephemeral store - node.set_ephemeral_store(InMemoryArrowDatabase()) + # Simulate expiry by replacing ephemeral database + node.set_ephemeral_database(InMemoryArrowDatabase()) loaded = node._load_cached_entries() assert len(loaded) == 1 @@ -180,7 +180,7 @@ def _entries(node: FunctionJobNode) -> list[tuple[TagProtocol, DataProtocol]]: These tests drive ``execute_data`` (one row, not a run), so the node shows no rows of a run (ITL-1095); ``get_cached_results`` gives the entries it holds or has - stored, an ``EmptyData`` placeholder for a result whose ephemeral store is gone. + stored, an ``EmptyData`` placeholder for a result whose ephemeral database is gone. """ entry_ids = [ node.compute_base_entry_id(tag, data) @@ -193,7 +193,7 @@ class TestProcessDataInternalEmptyData: def test_cache_hit_returns_cached_result(self, ephemeral_node): """Downstream cache hit on EmptyData input returns the cached result.""" node = ephemeral_node - # Step 1: Run upstream to populate ephemeral store and pipeline DB + # Step 1: Run upstream to populate ephemeral database and pipeline DB for tag, data in node._input_stream.iter_data(): node.execute_data(tag, data) @@ -222,7 +222,7 @@ def triple_value(result: int) -> int: downstream.execute_data(tag, data) # Step 4: Simulate upstream ephemeral expiry - node.set_ephemeral_store(InMemoryArrowDatabase()) + node.set_ephemeral_database(InMemoryArrowDatabase()) node._cached_output_datas.clear() # Step 5: Upstream now emits EmptyData @@ -244,7 +244,7 @@ def test_cache_miss_raises_ephemeral_result_missing_error(self, ephemeral_node): node.execute_data(tag, data) # Simulate expiry - node.set_ephemeral_store(InMemoryArrowDatabase()) + node.set_ephemeral_database(InMemoryArrowDatabase()) node._cached_output_datas.clear() # Build a downstream node that has NEVER run before @@ -294,7 +294,7 @@ async def test_async_empty_data_cache_hit_returns_cached_result( finds a match in ``_cached_function_pod.lookup_cached_data()``. """ node = ephemeral_node - # Run upstream to populate ephemeral store and pipeline DB + # Run upstream to populate ephemeral database and pipeline DB for tag, data in node._input_stream.iter_data(): node.execute_data(tag, data) @@ -315,7 +315,7 @@ def triple_value(result: int) -> int: downstream.execute_data(tag, data) # Simulate ephemeral expiry - node.set_ephemeral_store(InMemoryArrowDatabase()) + node.set_ephemeral_database(InMemoryArrowDatabase()) node._cached_output_datas.clear() # Upstream now emits EmptyData @@ -341,7 +341,7 @@ async def test_async_empty_data_cache_miss_raises(self, ephemeral_node): node.execute_data(tag, data) # Simulate expiry - node.set_ephemeral_store(InMemoryArrowDatabase()) + node.set_ephemeral_database(InMemoryArrowDatabase()) node._cached_output_datas.clear() # Build downstream that has NEVER run — empty result cache diff --git a/tests/test_core/operators/test_materialize_source_info.py b/tests/test_core/operators/test_materialize_source_info.py index 057291375..d5f7a48cd 100644 --- a/tests/test_core/operators/test_materialize_source_info.py +++ b/tests/test_core/operators/test_materialize_source_info.py @@ -645,7 +645,7 @@ def test_operator_node_identity_and_row_zero_record_hash(self): class TestStoreWrittenBeforeFix: - """What a user sees when a store written by pre-fix ``main`` is re-run. + """What a user sees when a database written by pre-fix ``main`` is re-run. Operator-node tables written by ``main`` from c398e7d4 (ITL-539) up to the fix hold row 0's source-info tokens on every row. The fixed code gives the corrected diff --git a/tests/test_core/side_effect_pod/test_side_effect_completion_lookup_errors.py b/tests/test_core/side_effect_pod/test_side_effect_completion_lookup_errors.py index 8a7b7ef9e..7a18bd61e 100644 --- a/tests/test_core/side_effect_pod/test_side_effect_completion_lookup_errors.py +++ b/tests/test_core/side_effect_pod/test_side_effect_completion_lookup_errors.py @@ -1,7 +1,7 @@ """A failed prior-completion lookup names the side-effect node and the row (ITL-919). A side-effect pod that tracks completion looks up each row's invocation record before -delivering it. When that lookup cannot read the store, the ``StoreReadError`` names the +delivering it. When that lookup cannot read the database, the ``OrcapodDatabaseReadError`` names the pod and the row's tag, as a failed write of the same record does (CodeRabbit on #290). """ @@ -12,7 +12,7 @@ import pytest from orcapod.databases import InMemoryArrowDatabase -from orcapod.errors import StoreReadError +from orcapod.errors import OrcapodDatabaseReadError from orcapod.pipeline.job import PipelineJob from orcapod.side_effects import SideEffectPod, SideEffectPodConfig from orcapod.sources import DictSource @@ -31,7 +31,9 @@ def at(self, *path_components: str) -> UnreadableCompletions: def get_record_by_id(self, record_path: tuple[str, ...], *args: Any, **kwargs: Any): """Fail as an unreadable stored table would.""" - raise StoreReadError("/store/" + "/".join(record_path), tuple(record_path)) + raise OrcapodDatabaseReadError( + "/store/" + "/".join(record_path), tuple(record_path) + ) def __getattr__(self, name: str) -> Any: """Delegate everything else.""" @@ -39,7 +41,7 @@ def __getattr__(self, name: str) -> Any: def test_completion_lookup_failure_names_the_node_and_row() -> None: - """The lookup's ``StoreReadError`` carries the pod's label and the row's tag.""" + """The lookup's ``OrcapodDatabaseReadError`` carries the pod's label and the row's tag.""" delivered: list[int] = [] def deliver(value: int, ctx: Any) -> None: @@ -51,7 +53,7 @@ def deliver(value: int, ctx: Any) -> None: with job: pod.process(DictSource([{"id": 0, "value": 10}], tag_columns=["id"])) - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: job.run() assert info.value.tag == {"id": 0} assert info.value.node_label is not None diff --git a/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py b/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py index b72f5ff7f..6897f0cca 100644 --- a/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py +++ b/tests/test_core/side_effect_pod/test_side_effect_job_node_reads.py @@ -24,7 +24,7 @@ from orcapod.core.sources import DictSource from orcapod.core.streams import ArrowTableStream from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase -from orcapod.errors import StoreWriteError +from orcapod.errors import OrcapodDatabaseWriteError from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator from orcapod.pipeline.job import PipelineJob from orcapod.protocols.core_protocols import StreamProtocol @@ -305,7 +305,7 @@ def test_a_run_that_raises_leaves_no_rows_to_read(orchestrator: Orchestrator) -> @pytest.mark.parametrize("track_completion", [True, False]) def test_a_read_before_any_run_delivers_nothing(track_completion: bool) -> None: - """A fresh job over a store with deliveries recorded returns no rows and delivers nothing.""" + """A fresh job over a database with deliveries recorded returns no rows and delivers nothing.""" database = InMemoryArrowDatabase() first, first_sent = _job(track_completion=track_completion, database=database) _ = first.run() @@ -353,7 +353,7 @@ def test_node_run_commits_the_invocation_log(tmp_path: Path) -> None: job, sent = _job(database=DeltaTableDatabase(tmp_path / "store")) _ = job.nodes["delivered"].run() assert len(sent) == ROWS - # A new database object reads only what was committed to the store. + # A new database object reads only what was committed to the database. again, again_sent = _job(database=DeltaTableDatabase(tmp_path / "store")) _ = again.run() assert again_sent == [] @@ -411,8 +411,8 @@ class _FailingFlushDatabase(InMemoryArrowDatabase): @override def flush(self) -> None: - """Fail, as a store that cannot be written does.""" - raise OSError("the store cannot be written") + """Fail, as a database that cannot be written does.""" + raise OSError("the database cannot be written") def test_node_run_whose_log_commit_fails_leaves_no_rows() -> None: @@ -422,7 +422,7 @@ def test_node_run_whose_log_commit_fails_leaves_no_rows() -> None: # The job hands its nodes scoped views of its database; attach the failing one # to the node itself. node.attach_databases(pipeline_database=_FailingFlushDatabase()) - with pytest.raises(StoreWriteError): + with pytest.raises(OrcapodDatabaseWriteError): _ = node.run() assert len(sent) == ROWS assert list(node.iter_data()) == [] diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 3cd815c22..6ea561158 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -555,13 +555,13 @@ def test_cached_function_pod_stream_does_not_read_its_result_store() -> None: calls: list[int] = [] store = InMemoryArrowDatabase() pod = CachedFunctionPod(_counting_pod(calls).pod, result_database=store) - _ = pod.process(_source()).as_table() # fills the result store + _ = pod.process(_source()).as_table() # fills the result database assert len(calls) == 25 stream = pod.process(_source()) head = stream.head() assert head.status is ViewStatus.NOT_COMPUTED - table = stream.as_table() # served from the result store + table = stream.as_table() # served from the result database assert len(calls) == 25 after = stream.head(5) assert after.status is ViewStatus.ROWS diff --git a/tests/test_core/streams/test_run.py b/tests/test_core/streams/test_run.py index 2414e7db1..14d11b9e6 100644 --- a/tests/test_core/streams/test_run.py +++ b/tests/test_core/streams/test_run.py @@ -148,7 +148,7 @@ def _function_pod_stream() -> Any: def _cached_function_pod_stream() -> Any: - """The stream of a ``CachedFunctionPod`` (in-memory result store) over ``_source()``.""" + """The stream of a ``CachedFunctionPod`` (in-memory result database) over ``_source()``.""" pod = CachedFunctionPod( _counting_pod([]).pod, result_database=InMemoryArrowDatabase() ) diff --git a/tests/test_databases/arrow_work_counter.py b/tests/test_databases/arrow_work_counter.py index 4ebcb03ca..84e154920 100644 --- a/tests/test_databases/arrow_work_counter.py +++ b/tests/test_databases/arrow_work_counter.py @@ -1,6 +1,6 @@ """A database wrapper that counts the Arrow memory its calls allocate (ITL-919). -The scaling tests use the bytes a store's calls allocate from the Arrow memory pool as a +The scaling tests use the bytes a database's calls allocate from the Arrow memory pool as a deterministic measure of the work those calls do: a call that filters, copies or concatenates the rows already stored allocates in proportion to them, while a call that touches only the rows it reads or writes does not. Unlike wall-clock time, the count does diff --git a/tests/test_databases/fake_pyspiral.py b/tests/test_databases/fake_pyspiral.py index e9b3b6130..887baff8c 100644 --- a/tests/test_databases/fake_pyspiral.py +++ b/tests/test_databases/fake_pyspiral.py @@ -40,7 +40,7 @@ top-level column declared non-null is refused on write (Arrow validates the outgoing batch on the client). A null in a struct field declared non-null is accepted and stored as the type's zero value (a null written to an ``int32`` field read back as ``0``). Switch it off - only to model a store that keeps nullability. + only to model a database that keeps nullability. * ``reject_struct_nulls`` (documented by pyspiral ``Table.write``): a null struct value is refused unless written with ``push_down_nulls=True``, which turns it into a struct of null fields (observed live, ITL-936; top-level struct columns only in the fake). @@ -235,7 +235,7 @@ class _Store: """Directory-backed table store shared by every fake object.""" def __init__(self, root: Path) -> None: - """Open the store under ``root`` and load its storage behaviour.""" + """Open the database under ``root`` and load its storage behaviour.""" self.root = root self.root.mkdir(parents=True, exist_ok=True) bfile = self.root / _BEHAVIOUR_FILE @@ -733,7 +733,7 @@ class FakeSpiral: def __init__( self, *, config: Any = None, overrides: dict[str, str] | None = None ) -> None: - """Open the store under ``FAKE_SPIRAL_ROOT`` and record ``overrides``.""" + """Open the database under ``FAKE_SPIRAL_ROOT`` and record ``overrides``.""" root = os.environ.get(_ROOT_ENV) if not root: raise RuntimeError(f"{_ROOT_ENV} is not set; call fake_pyspiral.install()") diff --git a/tests/test_databases/test_delta_arrow_type_roundtrip.py b/tests/test_databases/test_delta_arrow_type_roundtrip.py index d87183636..e0d1ee6da 100644 --- a/tests/test_databases/test_delta_arrow_type_roundtrip.py +++ b/tests/test_databases/test_delta_arrow_type_roundtrip.py @@ -1,4 +1,4 @@ -"""The Delta store gives back the Arrow types it was given (ITL-928, part B). +"""The Delta database gives back the Arrow types it was given (ITL-928, part B). Delta Lake has no unsigned integer, nanosecond, second or millisecond timestamp, or zoned (other than UTC) timestamp type, and the Delta writer converts such columns. A @@ -67,7 +67,7 @@ def _written_and_read(tmp_path: pathlib.Path, column: pa.Array[Any]) -> pa.Table: - """Write one three-row column to a Delta store and read it back from a new store object.""" + """Write one three-row column to a Delta database and read it back from a new store object.""" DeltaTableDatabase(tmp_path).add_records( ("t",), pa.table({"id": pa.array([b"a", b"b", b"c"], pa.large_binary()), "c": column}), @@ -182,7 +182,7 @@ def _source_table(k: pa.Array[Any]) -> pa.Table: def _chained_job(root: pathlib.Path, table: pa.Table) -> PipelineJob: - """A job on a Delta store at ``root``: source -> ``double`` -> ``increment``.""" + """A job on a Delta database at ``root``: source -> ``double`` -> ``increment``.""" with PipelineJob(name="j", database=DeltaTableDatabase(root)) as job: src = ArrowTableSource(table, tag_columns=["k"], source_id="src") increment.pod(double.pod(src, label="double"), label="increment") @@ -195,7 +195,7 @@ def test_a_rerun_on_the_same_store_writes_no_pipeline_rows( name: str, mode: str, tmp_path: pathlib.Path ) -> None: """The ITL-990 case: source -> pod -> pod, run twice; a fresh job's run shows 3 rows - and the store holds 3 records per node (a node shows rows only after a run, + and the database holds 3 records per node (a node shows rows only after a run, ITL-1095).""" from orcapod.pipeline import AsyncPipelineOrchestrator @@ -220,7 +220,7 @@ def test_a_rerun_on_the_same_store_writes_no_pipeline_rows( @pytest.mark.parametrize("name", list(TAGS)) def test_a_fresh_job_reads_the_tag_type(name: str, tmp_path: pathlib.Path) -> None: - """A new job on the same store, run, reads the tag type through every read of a + """A new job on the same database, run, reads the tag type through every read of a node (a node shows rows only after a run, ITL-1095).""" table = _source_table(TAGS[name]) _chained_job(tmp_path, table).run() @@ -258,7 +258,7 @@ def test_a_loaded_job_reads_the_tag_type(name: str, tmp_path: pathlib.Path) -> N """A job saved after its run and loaded in a new process reads the tag type. The load runs in a child process, so nothing of the run (pods, sources, store - objects) can stand in for what the saved job and the store hold. + objects) can stand in for what the saved job and the database hold. """ job = _chained_job(tmp_path / "store", _source_table(TAGS[name])) job.run() @@ -319,7 +319,7 @@ def test_a_cached_source_reads_the_tag_type(name: str, tmp_path: pathlib.Path) - table = _source_table(TAGS[name]) def cached() -> CachedSource: - """A ``CachedSource`` of the table, cached in a new Delta store object.""" + """A ``CachedSource`` of the table, cached in a new Delta database object.""" return CachedSource( ArrowTableSource(table, tag_columns=["k"], source_id="src"), cache_database=DeltaTableDatabase(tmp_path), @@ -376,7 +376,7 @@ def test_nanoseconds_cannot_be_appended_to_a_table_written_before_the_fix( """A build before ITL-928 stored ``timestamp[ns]`` as ``timestamp[us]``. The integer form this build writes differs from that stored type, so the append - is refused (nothing is written) and the store must be rebuilt. + is refused (nothing is written) and the database must be rebuilt. """ from deltalake import write_deltalake @@ -432,7 +432,7 @@ def test_unsigned_rows_append_to_a_signed_column_without_a_recorded_type( ) -> None: """A column with no recorded type may hold an unsigned column a build before ITL-928 wrote (Delta's ``int32`` for ``uint32``): appending to it is not refused, - as before. Such a store must be rebuilt (developer, November 12, 2026).""" + as before. Such a database must be rebuilt (developer, November 12, 2026).""" store = DeltaTableDatabase(tmp_path) store.add_records( ("t",), diff --git a/tests/test_databases/test_delta_committed_snapshot.py b/tests/test_databases/test_delta_committed_snapshot.py index 00151a2f8..99982b657 100644 --- a/tests/test_databases/test_delta_committed_snapshot.py +++ b/tests/test_databases/test_delta_committed_snapshot.py @@ -3,7 +3,7 @@ A ``DeltaTableDatabase`` family answers lookups from the rows of the table version it loaded, read once and extended after its own commits. These tests pin the boundary: -- A cold handle (nothing loaded) on an unreadable table raises ``StoreReadError`` for +- A cold handle (nothing loaded) on an unreadable table raises ``OrcapodDatabaseReadError`` for every read, including lookups of ids or values that are not stored. - A warm family keeps answering from the version it loaded, as before (it reloads a table only at its own commit). At its own commit it notices another writer's commit, @@ -26,7 +26,7 @@ import pytest from orcapod.databases import DeltaTableDatabase -from orcapod.errors import StoreReadError +from orcapod.errors import OrcapodDatabaseReadError PATH = ("table",) @@ -82,7 +82,7 @@ def test_cold_handle_on_an_unreadable_table_raises( ) cold = DeltaTableDatabase(tmp_path) monkeypatch.setattr(deltalake.DeltaTable, "to_pyarrow_dataset", _storage_down) - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: _ = READS[method](cold) assert isinstance(info.value.__cause__, OSError) @@ -105,7 +105,7 @@ def test_warm_family_reloads_at_its_own_commit_after_another_writer( db.add_records(PATH, _rows(["c"], [3]), flush=True) with monkeypatch.context() as patch: patch.setattr(deltalake.DeltaTable, "to_pyarrow_dataset", _storage_down) - with pytest.raises(StoreReadError): + with pytest.raises(OrcapodDatabaseReadError): _ = READS[method](db) assert _values(db.get_records_with_column_value(PATH, {"g": b"g"})) == [1, 2, 3] assert _values(db.get_records_by_ids(PATH, [b"a", b"b", b"c"])) == [1, 2, 3] @@ -131,7 +131,7 @@ def fail_first_scan(self: deltalake.DeltaTable, **kwargs: Any) -> Any: return dataset monkeypatch.setattr(deltalake.DeltaTable, "to_pyarrow_dataset", fail_first_scan) - with pytest.raises(StoreReadError): + with pytest.raises(OrcapodDatabaseReadError): _ = db.get_records_with_column_value(PATH, {"g": b"g"}) assert _values(db.get_records_with_column_value(PATH, {"g": b"g"})) == [1, 2] assert _values(db.get_record_by_id(PATH, b"b")) == [2] diff --git a/tests/test_databases/test_delta_database_lookup_equivalence.py b/tests/test_databases/test_delta_database_lookup_equivalence.py index c9861cc9a..06520e9b7 100644 --- a/tests/test_databases/test_delta_database_lookup_equivalence.py +++ b/tests/test_databases/test_delta_database_lookup_equivalence.py @@ -160,7 +160,7 @@ def test_scoped_views_share_records(tmp_path: Path) -> None: def test_another_writer_is_seen_after_the_next_commit(tmp_path: Path) -> None: - """Rows another writer committed are read after this store's next commit.""" + """Rows another writer committed are read after this database's next commit.""" db = DeltaTableDatabase(tmp_path) _write(db, 0, 10, 5) _assert_all_lookups(db) diff --git a/tests/test_databases/test_delta_exit.py b/tests/test_databases/test_delta_exit.py index 73a65268d..a2081e3c6 100644 --- a/tests/test_databases/test_delta_exit.py +++ b/tests/test_databases/test_delta_exit.py @@ -1,4 +1,4 @@ -"""A process that read a local Delta store must exit cleanly (ITL-892). +"""A process that read a local Delta database must exit cleanly (ITL-892). deltalake's default pyarrow reader goes through a filesystem implemented in Python. Arrow finishes tearing down a scan on a thread-pool worker after the scan returns. If that worker @@ -99,7 +99,7 @@ def _run_once(script: str, store: Path) -> str | None: def test_process_exits_after_reading_a_local_delta_store( tmp_path: Path, script: str ) -> None: - """Every one of ``RUNS`` fresh interpreters reads a local Delta store and exits cleanly. + """Every one of ``RUNS`` fresh interpreters reads a local Delta database and exits cleanly. Each run must finish within ``DEADLINE_S`` with exit code 0. A run that misses the deadline or aborts is the ITL-892 exit hang. @@ -111,6 +111,6 @@ def test_process_exits_after_reading_a_local_delta_store( ] assert not failures, ( f"{len(failures)} of {RUNS} processes did not exit cleanly after reading a " - f"local Delta store (ITL-892: Arrow thread-pool teardown at interpreter exit). " + f"local Delta database (ITL-892: Arrow thread-pool teardown at interpreter exit). " f"First failures: {failures[:3]}" ) diff --git a/tests/test_databases/test_delta_native_fs_filtered_reads.py b/tests/test_databases/test_delta_native_fs_filtered_reads.py index a82eb2af6..b30b55982 100644 --- a/tests/test_databases/test_delta_native_fs_filtered_reads.py +++ b/tests/test_databases/test_delta_native_fs_filtered_reads.py @@ -1,12 +1,12 @@ -"""Filtered Delta reads on the native reader keep the store's read contract (ITL-892). +"""Filtered Delta reads on the native reader keep the database's read contract (ITL-892). Orcapod scans local Delta tables through a native pyarrow filesystem (ITL-892). These tests pin how that reader combines with the read contract of ITL-884 and ITL-864, for every read method, including the filtered ones: -- A stored table that cannot be read raises ``StoreReadError``, never "no rows". +- A stored table that cannot be read raises ``OrcapodDatabaseReadError``, never "no rows". - A bad filter (an unknown column, a value that does not compare with the column) is the - caller's error: it is raised as itself, not as ``StoreReadError``, and the same error is + caller's error: it is raised as itself, not as ``OrcapodDatabaseReadError``, and the same error is raised before and after a flush. - A filtered read returns the stored rows plus the pending rows a flush would insert, exactly the rows the unfiltered read returns, filtered in memory. @@ -23,7 +23,7 @@ import pytest from orcapod.databases import DeltaTableDatabase -from orcapod.errors import StoreError, StoreReadError +from orcapod.errors import OrcapodDatabaseError, OrcapodDatabaseReadError PATH = ("table",) ID = DeltaTableDatabase.RECORD_ID_COLUMN @@ -117,7 +117,7 @@ def test_filtered_read_equals_the_unfiltered_read_filtered_in_memory( def _damage(store: Path, how: str) -> None: - """Make every data file of the store's tables unreadable. + """Make every data file of the database's tables unreadable. Args: store: Root directory of the ``DeltaTableDatabase``. @@ -138,7 +138,7 @@ def _damage(store: Path, how: str) -> None: def test_filtered_read_of_an_unreadable_table_raises_store_read_error( tmp_path: Path, method: str, pending: bool, how: str ) -> None: - """Every read of a table whose data files cannot be read raises ``StoreReadError``.""" + """Every read of a table whose data files cannot be read raises ``OrcapodDatabaseReadError``.""" writer = DeltaTableDatabase(tmp_path) writer.add_records(PATH, _rows(["a", "c"], [1, 3], [1, 0]), flush=True) _damage(tmp_path, how) @@ -151,7 +151,7 @@ def test_filtered_read_of_an_unreadable_table_raises_store_read_error( else: read = FILTERED[method][0] - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: _ = read(db) assert info.value.__cause__ is not None @@ -166,7 +166,7 @@ def test_filtered_read_of_an_unreadable_table_raises_store_read_error( def test_bad_filter_raises_its_own_error_before_and_after_flush( tmp_path: Path, bad: str ) -> None: - """A bad filter is the caller's error: never ``StoreReadError``, the same at every stage.""" + """A bad filter is the caller's error: never ``OrcapodDatabaseReadError``, the same at every stage.""" db = _stored_and_pending(tmp_path) column_values = BAD_FILTERS[bad] @@ -178,6 +178,6 @@ def test_bad_filter_raises_its_own_error_before_and_after_flush( db.flush() for when, error in errors.items(): - assert not isinstance(error, StoreError), f"{when}: {error!r}" + assert not isinstance(error, OrcapodDatabaseError), f"{when}: {error!r}" assert isinstance(error, pa.ArrowException), f"{when}: {error!r}" assert type(errors["before flush"]) is type(errors["after flush"]) diff --git a/tests/test_databases/test_delta_native_fs_read.py b/tests/test_databases/test_delta_native_fs_read.py index 81f52cec6..1746cbe65 100644 --- a/tests/test_databases/test_delta_native_fs_read.py +++ b/tests/test_databases/test_delta_native_fs_read.py @@ -1,4 +1,4 @@ -"""Reading a local Delta store returns exactly what deltalake's default reader returns (ITL-892). +"""Reading a local Delta database returns exactly what deltalake's default reader returns (ITL-892). Orcapod scans local Delta tables through a native pyarrow filesystem instead of deltalake's default ``PyFileSystem(DeltaStorageHandler)``, to avoid a hang at interpreter exit. Orcapod is @@ -91,7 +91,7 @@ def test_pipeline_store_reads_match_default_reader_for_logical_types( @op.function_pod(output_keys=["path"]) def same_path(f: Path) -> Path: - """Return the input path, so the store holds a ``Path`` column.""" + """Return the input path, so the database holds a ``Path`` column.""" return f @op.function_pod(output_keys=["m"]) @@ -295,7 +295,7 @@ def test_database_reads_a_store_under_a_special_character_root( (table_dir,) = _delta_table_dirs(store_dir) expected = _default_reader(table_dir) assert expected.num_rows == 3 - assert actual is not None, f"the store under {root_name!r} read as empty" + assert actual is not None, f"the database under {root_name!r} read as empty" _assert_identical(actual, expected, sort_by=DeltaTableDatabase.RECORD_ID_COLUMN) diff --git a/tests/test_databases/test_delta_read_failures.py b/tests/test_databases/test_delta_read_failures.py index 728e490da..7e9dc3694 100644 --- a/tests/test_databases/test_delta_read_failures.py +++ b/tests/test_databases/test_delta_read_failures.py @@ -1,10 +1,10 @@ """Regression tests for ITL-884: a Delta table that cannot be read is never "no rows". Covers: -- A table that exists but cannot be read raises ``StoreReadError`` on every read +- A table that exists but cannot be read raises ``OrcapodDatabaseReadError`` on every read path, naming the table; a table that does not exist still reads as ``None``. - A table whose log cannot be loaded is never overwritten by a flush. -- ``flush`` raises ``StoreReadError`` and the append refusal's +- ``flush`` raises ``OrcapodDatabaseReadError`` and the append refusal's ``InputValidationError`` themselves, not wrapped in ``RuntimeError``, and notes every other batch that failed in the same flush. - An append whose column types differ from the stored table is refused before @@ -12,7 +12,7 @@ (same types in another layout, new or missing columns, unchanged extension storage) still succeed. - A readable table whose stored user-class shape differs from the registered class - raises ``TypeIdentityError``, not ``StoreReadError`` (ITL-881). + raises ``TypeIdentityError``, not ``OrcapodDatabaseReadError`` (ITL-881). """ from __future__ import annotations @@ -30,7 +30,11 @@ from deltalake.exceptions import DeltaProtocolError from orcapod.databases import DeltaTableDatabase -from orcapod.errors import InputValidationError, StoreReadError, TypeIdentityError +from orcapod.errors import ( + InputValidationError, + OrcapodDatabaseReadError, + TypeIdentityError, +) from orcapod.sources import DictSource PATH = ("results", "t") @@ -81,7 +85,7 @@ def ext_array(values: list[Any], storage: pa.DataType) -> pa.ExtensionArray[Any] @pytest.fixture def base(tmp_path: Path) -> Path: - """Return the store root, inside the test's temporary directory.""" + """Return the database root, inside the test's temporary directory.""" return tmp_path / "store" @@ -123,7 +127,7 @@ def unsupported_reader_feature(base: Path) -> None: log.write_text("\n".join(json.dumps(a) for a in actions)) -def assert_names_table(exc: StoreReadError, base: Path) -> None: +def assert_names_table(exc: OrcapodDatabaseReadError, base: Path) -> None: """Assert that ``exc`` names the table at ``PATH`` and chains its cause.""" assert Path(exc.table) == table_dir(base) assert exc.table in str(exc) @@ -149,15 +153,15 @@ class TestUnreadableTable: @pytest.mark.parametrize("read", READS.values(), ids=list(READS)) def test_read_raises_store_read_error(self, base, unreadable, read): - """Each read of the unreadable table raises ``StoreReadError`` naming it.""" - with pytest.raises(StoreReadError) as info: + """Each read of the unreadable table raises ``OrcapodDatabaseReadError`` naming it.""" + with pytest.raises(OrcapodDatabaseReadError) as info: read(unreadable) assert_names_table(info.value, base) def test_skip_duplicates_append_raises_store_read_error(self, base, unreadable): """The existing-ids read for ``skip_duplicates`` raises, never assumes none.""" # skip_duplicates reads the stored record ids before buffering the rows. - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: unreadable.add_records( PATH, rows([b"c"], pa.array([3], pa.int64())), @@ -172,7 +176,7 @@ def test_table_still_exists(self, unreadable): def test_bad_filter_is_not_reported_as_unreadable_table(stored): - """A bad filter column is the caller's error, not a ``StoreReadError``.""" + """A bad filter column is the caller's error, not a ``OrcapodDatabaseReadError``.""" # The table reads fine; the filter names a column it does not have. with pytest.raises(pa.ArrowInvalid): stored.get_records_with_column_value(PATH, {"no_such_column": 1}) @@ -203,9 +207,9 @@ def test_unsupported_reader_feature(self, base, stored): before = snapshot(table_dir(base)) db = DeltaTableDatabase(base_path=base) - with pytest.raises(StoreReadError): + with pytest.raises(OrcapodDatabaseReadError): db.get_all_records(PATH) - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: db.add_records( PATH, rows([b"c"], pa.array([3], pa.int64())), @@ -235,7 +239,7 @@ def failing_load(table_uri, *args, **kwargs): before = snapshot(target) db = DeltaTableDatabase(base_path=base) monkeypatch.setattr(deltalake, "DeltaTable", failing_load) - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: db.add_records( PATH, rows([b"c"], pa.array([3], pa.int64())), @@ -252,12 +256,12 @@ def failing_load(table_uri, *args, **kwargs): class TestFlushRaisesStoreErrorsThemselves: - """``flush`` raises store errors as themselves, with the other failures noted.""" + """``flush`` raises database errors as themselves, with the other failures noted.""" def test_store_read_error_is_raised_with_a_note_per_other_failed_batch( self, base, stored ): - """The ``StoreReadError`` is raised and the refused batch is noted on it.""" + """The ``OrcapodDatabaseReadError`` is raised and the refused batch is noted on it.""" other = ("results", "u") stored.add_records( other, @@ -274,10 +278,10 @@ def test_store_read_error_is_raised_with_a_note_per_other_failed_batch( PATH, rows([b"c"], pa.array([3], pa.int64())), record_id_column="rid" ) - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: db.flush() - # The StoreReadError wins over the earlier refusal, because it is the one + # The OrcapodDatabaseReadError wins over the earlier refusal, because it is the one # that fails a run whatever the error policy; the refusal is noted on it. assert_names_table(info.value, base) notes = getattr(info.value, "__notes__", []) @@ -285,7 +289,7 @@ def test_store_read_error_is_raised_with_a_note_per_other_failed_batch( assert "results/u" in notes[0] assert "InputValidationError" in notes[0] # Both failed batches stay queued, so the next flush fails the same way. - with pytest.raises(StoreReadError): + with pytest.raises(OrcapodDatabaseReadError): db.flush() def test_other_flush_failures_still_raise_runtime_error(self, base, monkeypatch): @@ -554,9 +558,9 @@ def other_shape(base: Path) -> DeltaTableDatabase: def test_class_mismatch_is_type_identity_error_not_store_read_error(other_shape, read): """A stored shape that differs from the registered class raises ``TypeIdentityError``. - ``StoreReadError`` means the table cannot be read. Here the table is intact and the + ``OrcapodDatabaseReadError`` means the table cannot be read. Here the table is intact and the class differs, so the identity error is raised as is, never wrapped. """ with pytest.raises(TypeIdentityError) as info: read(other_shape) - assert not isinstance(info.value, StoreReadError) + assert not isinstance(info.value, OrcapodDatabaseReadError) diff --git a/tests/test_databases/test_delta_read_your_writes.py b/tests/test_databases/test_delta_read_your_writes.py index d993ae372..cd0156f3a 100644 --- a/tests/test_databases/test_delta_read_your_writes.py +++ b/tests/test_databases/test_delta_read_your_writes.py @@ -17,7 +17,7 @@ import pytest from orcapod.databases import DeltaTableDatabase -from orcapod.errors import StoreReadError +from orcapod.errors import OrcapodDatabaseReadError PATH = ("table",) SCOPE = "scope" @@ -228,7 +228,7 @@ def test_failed_id_refresh_is_retried_by_the_next_read( db.add_records(PATH, _rows(["a", "b"], [1, 20])) # The failed refresh is a failed stored read (ITL-884): never "no stored ids". - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: _ = db.get_all_records(PATH) assert isinstance(info.value.__cause__, OSError) @@ -237,7 +237,7 @@ def test_failed_id_refresh_is_retried_by_the_next_read( @pytest.mark.usefixtures("fail_id_refresh_once") def test_get_records_by_ids_raises_on_failed_id_refresh(tmp_path: Path) -> None: - """A failed stored-ID refresh raises ``StoreReadError``, never "no records" (ITL-884).""" + """A failed stored-ID refresh raises ``OrcapodDatabaseReadError``, never "no records" (ITL-884).""" # Another root stores ``a`` and ``b``: this handle has to read the stored ids (ITL-919). DeltaTableDatabase(tmp_path).at(SCOPE).add_records( PATH, _rows(["a", "b"], [1, 2], [0, 1]), flush=True @@ -245,7 +245,7 @@ def test_get_records_by_ids_raises_on_failed_id_refresh(tmp_path: Path) -> None: db = DeltaTableDatabase(tmp_path).at(SCOPE) db.add_records(PATH, _rows(["c"], [3], [1])) - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: _ = db.get_records_by_ids(PATH, [b"a", b"c"]) assert isinstance(info.value.__cause__, OSError) assert _values(db.get_records_by_ids(PATH, [b"a", b"c"])) == [1, 3] @@ -259,7 +259,7 @@ def test_unreadable_table_with_sibling_pending_rows_raises_and_drops_nothing( """A read never answers from pending rows alone when the stored table is unreadable. ``b`` is stored and ``a`` is pending through a sibling handle. While the stored - table cannot be read, every read method raises ``StoreReadError`` (ITL-884) + table cannot be read, every read method raises ``OrcapodDatabaseReadError`` (ITL-884) rather than returning only the pending rows (ITL-864); once it is readable again the same read returns both rows, so nothing was dropped from the pending batch. """ @@ -287,7 +287,7 @@ def unreadable(*_args: object, **_kwargs: object) -> pa.Table: # (ITL-919). Every read that needs storage still raises. assert _values(READS[method](reader, PATH)) == EXPECTED[method] else: - with pytest.raises(StoreReadError) as info: + with pytest.raises(OrcapodDatabaseReadError) as info: _ = READS[method](reader, PATH) assert isinstance(info.value.__cause__, OSError) diff --git a/tests/test_databases/test_list_table_paths.py b/tests/test_databases/test_list_table_paths.py index e9cd44c20..698b0325a 100644 --- a/tests/test_databases/test_list_table_paths.py +++ b/tests/test_databases/test_list_table_paths.py @@ -1,4 +1,4 @@ -"""``list_table_paths``: the tables a store holds under a path prefix (ITL-954). +"""``list_table_paths``: the tables a database holds under a path prefix (ITL-954). Every database lists, without opening a table, the paths of the tables whose path starts with a prefix, relative to its ``base_path`` like ``table_exists``. Tables nested below @@ -50,7 +50,7 @@ def _fill(store: ArrowDatabaseProtocol, paths: list[tuple[str, ...]] = PATHS) -> @pytest.fixture(params=["in_memory", "delta", "sqlite"]) def make_store(request: pytest.FixtureRequest, tmp_path: Path) -> Callable[[], Store]: - """A factory of database objects over one empty store of each backend.""" + """A factory of database objects over one empty database of each backend.""" connector = SQLiteConnector(str(tmp_path / "store.db")) shared = InMemoryArrowDatabase() @@ -66,7 +66,7 @@ def make() -> Store: class TestEveryBackend: - """Listing behaviour that every store from the ``make_store`` fixture shares.""" + """Listing behaviour that every database from the ``make_store`` fixture shares.""" def test_lists_every_table_sorted(self, make_store: Callable[[], Store]): """Without a prefix, a second store object lists every written path in sorted @@ -368,7 +368,7 @@ class TestWrappers: """Listing through ``ExtensionAwareDatabase`` and ``NoOpArrowDatabase``.""" def test_extension_aware_database_delegates(self): - """The wrapper lists the same paths as the store it wraps.""" + """The wrapper lists the same paths as the database it wraps.""" inner = InMemoryArrowDatabase() _fill(inner) wrapper = ExtensionAwareDatabase(inner, UniversalTypeConverter()) @@ -386,7 +386,7 @@ def test_no_op_database_holds_no_table(self): @pytest.mark.s3 def test_s3_lists_tables_and_nested_tables(s3_base_uri, s3_storage_options): - """A Delta store on an S3 endpoint lists the tables under a prefix, nested ones + """A Delta database on an S3 endpoint lists the tables under a prefix, nested ones included. """ store = DeltaTableDatabase(s3_base_uri, storage_options=s3_storage_options) diff --git a/tests/test_hashing/data/generate_v0_2_0_dict_store.py b/tests/test_hashing/data/generate_v0_2_0_dict_store.py index be6686441..590d73a86 100644 --- a/tests/test_hashing/data/generate_v0_2_0_dict_store.py +++ b/tests/test_hashing/data/generate_v0_2_0_dict_store.py @@ -1,11 +1,11 @@ -"""Build a Delta result store with the released v0.2.0 code (ITL-905 test fixture). +"""Build a Delta result database with the released v0.2.0 code (ITL-905 test fixture). Run from a checkout of the v0.2.0 tag, then pack the store: uv run python generate_v0_2_0_dict_store.py tar -czf v0_2_0_dict_store.tar.gz -C . -The process can hang at interpreter exit inside deltalake after printing; the store is +The process can hang at interpreter exit inside deltalake after printing; the database is complete once the record counts are printed. Two pods write to the store: diff --git a/tests/test_hashing/test_dict_order_hash.py b/tests/test_hashing/test_dict_order_hash.py index 74a69e523..867aaee0a 100644 --- a/tests/test_hashing/test_dict_order_hash.py +++ b/tests/test_hashing/test_dict_order_hash.py @@ -432,7 +432,7 @@ def _extract_data_only(archive: tarfile.TarFile, target: Path) -> None: def _delta_db(tmp_path: Path) -> DeltaTableDatabase: - """Return a Delta result store under ``tmp_path``.""" + """Return a Delta result database under ``tmp_path``.""" return DeltaTableDatabase(tmp_path / "store") @@ -495,7 +495,7 @@ def test_fresh_and_cached_dict_outputs_iterate_identically(self, result_db): class TestV020Store: - """A Delta result store written by the released v0.2.0 code. + """A Delta result database written by the released v0.2.0 code. ``data/v0_2_0_dict_store.tar.gz`` was produced by v0.2.0 running ``total`` over two inputs: ``k=unsorted`` holding ``{"b": 2, "a": 1}`` (stored in that order) and @@ -512,7 +512,7 @@ class TestV020Store: @pytest.fixture def store(self, tmp_path): - """Return the v0.2.0 Delta store, extracted under ``tmp_path``.""" + """Return the v0.2.0 Delta database, extracted under ``tmp_path``.""" # The store is shipped as a tarball: its directory names contain ":". target = tmp_path / "v020_store" with tarfile.open(V020_STORE) as archive: diff --git a/tests/test_hashing/test_pandas_ignore_list.py b/tests/test_hashing/test_pandas_ignore_list.py index 8b2356afe..b50c51027 100644 --- a/tests/test_hashing/test_pandas_ignore_list.py +++ b/tests/test_hashing/test_pandas_ignore_list.py @@ -275,7 +275,7 @@ def test_a_column_stored_under_another_list_is_refused( stored = loose.type_converter.python_dicts_to_arrow_table( [{"x": _frame()}], python_schema={"x": pd.DataFrame} ) - # As a store hands it back: the storage column with the identifier on the field. + # As a database hands it back: the storage column with the identifier on the field. field = stored.schema.field("x") assert isinstance(field.type, pa.ExtensionType) chunk = stored.column("x").chunk(0) diff --git a/tests/test_hashing/test_polars_dataframe_hashing.py b/tests/test_hashing/test_polars_dataframe_hashing.py index e5836d5d7..6dd6016fd 100644 --- a/tests/test_hashing/test_polars_dataframe_hashing.py +++ b/tests/test_hashing/test_polars_dataframe_hashing.py @@ -114,7 +114,7 @@ def test_default_categoricals_hash_equal(self) -> None: def test_value_read_back_from_a_database_hashes_the_same( self, tmp_path: Path ) -> None: - """A frame read back from a Delta store hashes like the original.""" + """A frame read back from a Delta database hashes like the original.""" expected = pl.DataFrame( { "e": pl.Series(["a", None], dtype=pl.Enum(["b", "a"])), diff --git a/tests/test_pipeline/loaded_job/save_job_script.py b/tests/test_pipeline/loaded_job/save_job_script.py index edbf6dfc0..c620f5f8e 100644 --- a/tests/test_pipeline/loaded_job/save_job_script.py +++ b/tests/test_pipeline/loaded_job/save_job_script.py @@ -2,12 +2,12 @@ Run as a script in a separate process, ``python save_job_script.py ``, so that the test loading the job starts from a fresh process: nothing the save created -(function pods, sources, the store object) is reachable from the loading process. +(function pods, sources, the database object) is reachable from the loading process. Writes into ````: * ``c.csv`` and ``dt/`` -- the CSV and Delta inputs; -* ``store/`` -- the Delta store the job ran against; +* ``store/`` -- the Delta database the job ran against; * ``job.json`` -- the saved job; * ``expected/