Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .basedpyright/baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -45553,7 +45553,7 @@
"code": "reportAttributeAccessIssue",
"range": {
"startColumn": 21,
"endColumn": 40,
"endColumn": 43,
"lineCount": 1
}
},
Expand Down
2 changes: 1 addition & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -854,7 +854,7 @@ data. The rules:
the rows of that run's inputs, sync or async (ITL-1093). Before a run in this process (a
fresh or loaded job), after a run that raised, after clear_cache(), and while stale, every
function and operator job node reads empty (zero typed rows; head() is not_computed and
points to run() and as_source()), in every cache mode: no read touches the store. A
points to run() and as_source()), in every cache mode: no read touches the database. A
never-run or cleared node has no modified time, so it is stale; a job node is stale also
when any function or operator job node it depends on is (core/nodes/staleness.py), and a
SourceJobNode is modified when it is bound or its bound source is. A run skips inputs
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -896,7 +896,7 @@ data. The rules:
the rows of that run's inputs, sync or async (ITL-1093). Before a run in this process (a
fresh or loaded job), after a run that raised, after `clear_cache()`, and while stale, every
function and operator job node reads empty (zero typed rows; `head()` is `not_computed` and
points to `run()` and `as_source()`), in every cache mode: no read touches the store. A
points to `run()` and `as_source()`), in every cache mode: no read touches the database. A
never-run or cleared node has no modified time, so it is stale; a job node is stale also
when any function or operator job node it depends on is (`core/nodes/staleness.py`), and a
`SourceJobNode` is modified when it is bound or its bound source is. A run skips inputs
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ While the following is subject to change based on future development, it represe
│ on success:
│ ctx_observer.on_data_end(node_label, tag, data, result, cached=False)
│ emit (tag_out, result) downstream
│ on StoreError (the store could not be read or written):
│ on OrcapodDatabaseError (the database could not be read or written):
│ ctx_observer.on_data_crash(node_label, tag, data, exc)
│ raise, whatever error_policy is — the run stops (ITL-878, ITL-884)
│ on any other exception (the function failed):
Expand Down
17 changes: 17 additions & 0 deletions changelog.d/ITL-1087.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
### Breaking changes and migration

#### The remaining "store" names are "database" names (ITL-1087)

`PipelineJob(database=...)` already names the Arrow database; the names around it now do too. There is no alias for any old name. A saved job file, a stored table, a column, a layout and every hash are unchanged.

- `set_ephemeral_store(store)` on a pipeline, a node and the node protocols is `set_ephemeral_database(database)`. Rename the call and pass the database as `database=` if you used the keyword. The node attribute `ephemeral_result_store` is `ephemeral_result_database`.
- `StoreError` is `OrcapodDatabaseError`. Rename every `except StoreError`. `SideEffectLogDataContextError` keeps its name and derives from `OrcapodDatabaseError`.
- `StoreWriteError` is `OrcapodDatabaseWriteError` and `StoreReadError` is `OrcapodDatabaseReadError`.
- `StoreContentWarning` is `OrcapodDatabaseContentWarning`. Rename it in `warnings.filterwarnings` and `pytest.warns`.
- `describe_store_table` is `describe_database_table`, and the `store_path` parameter of `account_for_stale_tables` and `find_stale_tables` is `database_path` (all in `orcapod.contexts.stored`).
- The module `orcapod.utils.store_errors` is `orcapod.utils.database_errors`; `store_error_of` and `keep_store_error` are `database_error_of` and `keep_database_error`.
- Error and warning messages say "database" where they said "store".

### Fixed

- `PipelineJob.set_ephemeral_database(...)` no longer raises `AttributeError` on a job that contains a side-effect node, live or loaded. The side-effect node takes no ephemeral database and is skipped; function and operator nodes receive it as before (ITL-1147).
2 changes: 1 addition & 1 deletion design_issues/ITL-1040-DG2.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ time with `data.drop`. Their sync paths drop the `_source_` columns from the tab
explicitly. So an async run stored an extra `_source_<dropped column>` in the function node's
pipeline records that a sync run did not, and the stored table's schema depended on the
orchestrator. This happened after the automatic select in front of a pod, and after any
explicit select or drop. On one store, mixing orchestrators then either widened a Delta table
explicit select or drop. On one database, mixing orchestrators then either widened a Delta table
(with nulls) or failed an in-memory commit with a schema mismatch. The cell values were always
the row's own. The row-alignment suite found it (ITL-1034). It had been reported for `Data` alone as ITL-838.

Expand Down
2 changes: 1 addition & 1 deletion design_issues/ITL-1042-NS1.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ run. Its first `iter_data()` / `as_table()` then threw away the in-memory result
node's whole stored table (two `get_all_records` calls) and stamped the node. That made every
operator node downstream of it stale, and their `head()` reported `not_computed` ("inputs
changed since the last computation") with no rows, although `as_table()` and `iter_data()`
still returned every row. `head()` of the function node itself read the store instead of
still returned every row. `head()` of the function node itself read the database instead of
memory. No function was called again and no operator recomputed.

**Rule:** a job node stamps its modified time once per completed run, after it has read all
Expand Down
8 changes: 4 additions & 4 deletions design_issues/ITL-1043-LJ1.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ source's identity config record `"nullable": {"tag": [...], "data": [...]}`, res
`T | None` through the type converter's own rule (`serialization.serialize_nullability`,
`deserialize_output_schema`, `deserialize_schema(nullable=...)`). `PipelineJob.save`
records the run content hash of operator nodes in `run.node_job_content_hashes`; a loaded
operator node returns it, and the pipeline hash saved in its descriptor. The store is the
operator node returns it, and the pipeline hash saved in its descriptor. The database is the
source of truth (developer, November 12, 2026): a loaded REPLAY node that knows its key
reads as the node built from code does, so no rows under its key (no table, or none of its
rows) is an empty result. A node whose file lacks the run content hash cannot name its
Expand All @@ -42,7 +42,7 @@ written before them) loads as before, lossy, and saving it again records nothing
not have. No hash, stored path or golden value of a running job changes. Facts about
stored data stay out of the definition: the job file holds only an optional, advisory
`run.expected_rows` section (rows under each node's run key at save time, written only
when certain); a loaded node returns what the store holds and issues one
`StoreContentWarning` when the number differs. A loaded function node no longer raises on
a store without its run's results (the ITL-904 `node_has_stored_records` flag is no longer
when certain); a loaded node returns what the database holds and issues one
`OrcapodDatabaseContentWarning` when the number differs. A loaded function node no longer raises on
a database without its run's results (the ITL-904 `node_has_stored_records` flag is no longer
written or read).
2 changes: 1 addition & 1 deletion design_issues/ITL-1045-FP1.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@ and no row's `_content_hash` was the hash of its own data.

**Fix:** `as_table` clears a stale cache first, before any of it is used, so the table and the
hashes come from the same rows and the call returns the current rows. For a job node no
public sequence changes the values between the two passes (its results come from the store),
public sequence changes the values between the two passes (its results come from the database),
so the defect was latent there; the same few lines fix its structure. No hash, golden value,
stored table or public API changes.
4 changes: 2 additions & 2 deletions design_issues/ITL-1052-F18.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ current (the result may mix two states of the input), so:
unnoticed by the last stream, hidden by the false staleness above.

The generation and the noted state are in-memory bookkeeping only: they enter no hash, no
saved file and no store. Regressions: `tests/test_core/function_pod/test_pod_stream_staleness.py`
saved file and no database. Regressions: `tests/test_core/function_pod/test_pod_stream_staleness.py`
(chains of two and three, sequential and concurrent, the cached pod stream, a chain through a
join; second reads call no function and read no store; a reset upstream recomputes; a reset
join; second reads call no function and read no database; a reset upstream recomputes; a reset
during a read, or made by the function while the stream computes, leaves the reader stale and
its next read is of one state).

Expand Down
2 changes: 1 addition & 1 deletion design_issues/ITL-1053-PV2.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ ITL-1029) and slice, so `head(n, cfg) == as_table(cfg).slice(0, n)`. A bounded r
(`SpiralDBTableSource`, a partly computed stream, a polled source's batches) sorts the rows it
has: "the rows read, in tag order", not "the first `n` in tag order". Nothing computes or reads
more than before. When ITL-1002 bounds a node's stored read, a node's sorted head becomes "the
rows read, sorted" unless the store returns rows in tag order.
rows read, sorted" unless the database returns rows in tag order.

The notebook preview asks for the tag-sorted head (developer, November 12, 2026: "preview should
sort by tags"; `preview_html`), and its header says "sorted by tags", or for a bounded read
Expand Down
4 changes: 2 additions & 2 deletions design_issues/ITL-1060-FN1.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## `src/orcapod/core/nodes/function_node.py` — Rows read back from a function node's store
## `src/orcapod/core/nodes/function_node.py` — Rows read back from a function node's database

### FN1 — A result row read back from the store got a fresh datagram id instead of its stored `__data_id`
### FN1 — A result row read back from the database got a fresh datagram id instead of its stored `__data_id`
**Status:** resolved
**Severity:** low
**Issue:** ITL-1060
Expand Down
2 changes: 1 addition & 1 deletion design_issues/ITL-1095-CR1.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ one level up only, so a source rebound two nodes up left the nodes below it "cur
**Fix:** (developer, November 13, 2026) a job node shows its current run and `node.as_source()`
shows everything stored. Function and operator job nodes read empty (typed zero rows, `head()`
`not_computed` pointing to `run()` and `as_source()`) until a run in this process, after
`clear_cache()`, after a run that raised and while stale; no read touches the store. A never-run
`clear_cache()`, after a run that raised and while stale; no read touches the database. A never-run
or cleared node has no modified time (stale); `core/nodes/staleness.py` makes a job node stale
when any function or operator job node above it is; a `SourceJobNode` is modified when bound or
when its bound source is. `job.run(load_only=True)` fills the nodes from what is stored and
Expand Down
2 changes: 1 addition & 1 deletion design_issues/ITL-1122-LN1.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
**Issue:** ITL-1122

The rule (developer, November 14, 2026, replacing an earlier form with exceptions for a user's
Arrow table and for a store that keeps the distinction): every column of every table Orcapod
Arrow table and for a database that keeps the distinction): every column of every table Orcapod
presents or stores has its string, binary and list types in their large variants, with no
exception. Measured on `main` 3a7f3850: `ArrowTableSource` and an explicit Arrow schema kept
`string`; `CSVSource` gave `string` (a CSV file keeps no distinction); `CachedSource` gave the
Expand Down
4 changes: 2 additions & 2 deletions design_issues/ITL-1133-ZR3.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ own (`OperatorJobNode.run()`, `_compute_and_store`) kept its operator's result a
the operator's own order (a join lists its data before its system tags; every operator that
keeps its input's order did the same over a source, which lists system tags after data). So one
node had two populated orders, its table before a run (row order) matched only one of them, and
a store holding rows of both refused the second append (`StoreWriteError`, schema mismatch, in
the in-memory store and in a Delta store with a pending batch). Only the operator cache
a database holding rows of both refused the second append (`OrcapodDatabaseWriteError`, schema mismatch, in
the in-memory store and in a Delta database with a pending batch). Only the operator cache
(`odb_v1`) was affected: function pipeline and result tables, side-effect logs and cached
sources store one order on every path. `odb_v1` dates from 2026-11-12 (ITL-967) and was never
released.
Expand Down
2 changes: 1 addition & 1 deletion design_issues/ITL-864-DW1.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ pipeline; the async orchestrator runs only user functions in worker threads. Reg
`tests/test_databases/test_delta_read_your_writes.py` and
`tests/test_pipeline/test_delta_node_records_after_run.py`. Stored ids come from
`_stored_ids`: the snapshot of the loaded table version reads them on first use, through an
id-only scan (a one-column schema); a failed scan raises `StoreReadError` and leaves the
id-only scan (a one-column schema); a failed scan raises `OrcapodDatabaseReadError` and leaves the
snapshot's ids unset (`None`), so the next read scans again instead of treating every stored
id as unstored and returning duplicates. A handle that loaded the table before it had any
stored ids (a fresh root over an existing table) reads them on its first read with pending
Expand Down
30 changes: 15 additions & 15 deletions design_issues/ITL-878-SW1.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
## `src/orcapod/pipeline/job.py`, orchestrators, node write paths — store write failures
## `src/orcapod/pipeline/job.py`, orchestrators, node write paths — database write failures

### SW1 — A failed store write did not fail `PipelineJob.run()`
### SW1 — A failed database write did not fail `PipelineJob.run()`
**Status:** resolved
**Severity:** critical
**Issue:** ITL-878

A node's write to the store ran inside the per-row handler meant for function failures:
A node's write to the database ran inside the per-row handler meant for function failures:
`FunctionNode.execute` (sync) and `_FunctionPodBase.async_execute.process_one` (async) wrap
compute *and* the result-table and pipeline-table writes in one `except Exception`. With the
default `error_policy="continue"` a failed write was skipped like a crashed function: the sync
Expand All @@ -15,30 +15,30 @@ the node read back fewer rows (none when every write failed, e.g. a logical-type
final flush, so the upstream function node's pending pipeline rows were never committed while
its result rows were; async callers got an `ExceptionGroup`.

**Fix:** Contract, for now (developer, November 10, 2026): a failed store write always fails
**Fix:** Contract, for now (developer, November 10, 2026): a failed database write always fails
the run, whatever `error_policy` is. Every node write site (`ResultCache.store`,
`FunctionNode.add_pipeline_record`, `OperatorNode._store_output_stream`, the side-effect
invocation log) raises `orcapod.errors.StoreWriteError`, chained from the backend error and
invocation log) raises `orcapod.errors.OrcapodDatabaseWriteError`, chained from the backend error and
naming the node, the row's tag and the table. The per-row handlers re-raise it, so the run
stops at the first failed write; `error_policy` still decides only about rows whose function
raised. `AsyncPipelineOrchestrator` unwraps it from the `TaskGroup` exception group, so both
orchestrators raise the same error. `PipelineJob.run()` flushes the writes that succeeded
before re-raising (a flush failure there is logged, not raised over the original), and wraps a
failure of its final flush. Nothing stored is rewritten; no hash, path or schema change. An
opt-in mode that tolerates store failures is a possible later option. A fail-loud post-run
hook or an observer callback that raises while the store error propagates is logged and noted
on it (`orcapod.utils.store_errors`), not raised in its place, so a node function that runs
another cached pod cannot turn that pod's store failure into a skipped row. When concurrent
node tasks fail together, the async orchestrator raises the store error of the node first in
topological order, notes the other failures on it and logs other store errors at WARNING.
opt-in mode that tolerates database failures is a possible later option. A fail-loud post-run
hook or an observer callback that raises while the database error propagates is logged and noted
on it (`orcapod.utils.database_errors`), not raised in its place, so a node function that runs
another cached pod cannot turn that pod's database failure into a skipped row. When concurrent
node tasks fail together, the async orchestrator raises the database error of the node first in
topological order, notes the other failures on it and logs other database errors at WARNING.
A failed flush (the result cache's per-row auto-flush or the job's final flush) raises
`StoreWriteError(flush=True)` naming the store and no row: views of one store share their
`OrcapodDatabaseWriteError(flush=True)` naming the database and no row: views of one database share their
pending writes, so the failing batch may be an earlier row's or another table's.
The rule (developer, November 11, 2026): store failures fail the run; a function output that
The rule (developer, November 11, 2026): database failures fail the run; a function output that
cannot be stored (a naive `datetime`, a value not matching the declared type, rejected while
`ResultCache.store` prepares the record, before any write) is a row failure governed by
`error_policy` exactly like a function that raised, not a `StoreWriteError`: the value is
specific to that row, the store is healthy, and every re-run would fail at the same row. The
`error_policy` exactly like a function that raised, not a `OrcapodDatabaseWriteError`: the value is
specific to that row, the database is healthy, and every re-run would fail at the same row. The
async orchestrator ignored `fail_fast` for such rows (ITL-879), unchanged here and fixed since
by ITL-879 (`ITL-879-EP1`).
Regressions: `tests/test_pipeline/test_store_write_failure.py`. Related, not fixed here: the async
Expand Down
6 changes: 3 additions & 3 deletions design_issues/ITL-879-EP1.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
handler (`_FunctionPodBase.async_execute.process_one`) skipped every row whose function
raised, so under `error_policy="fail_fast"` an async run computed every other row and
finished, with a `RunSummary` and a `RowFailuresWarning`, where the sync run raised the row's
exception after storing the rows before it. Any other node failure that was not a store
exception after storing the rows before it. Any other node failure that was not a database
error (a side-effect pod with `on_error="raise"`, an operator that rejected its inputs)
reached the caller as the task group's `ExceptionGroup`, where the sync run raised the
exception itself. Under `"continue"` the two orchestrators already agreed: no raise, the
Expand All @@ -23,12 +23,12 @@ and the orchestrator passes its own. Under `"fail_fast"` the pod re-raises the r
exception after `on_data_crash`; the existing task groups cancel the rows and nodes still
running and every node closes its output in its existing `finally`; the failing node calls
`on_node_end` before it raises, as `execute` does. The orchestrator's cancel and close
machinery is unchanged. Its exception-group handler now always raises ONE exception: a store
machinery is unchanged. Its exception-group handler now always raises ONE exception: a database
error, then a `TypeIdentityError` (both as before), then the exception of the node earliest
in topological order (ties: the order the failures happened), the others noted on it
(`_single_failure`). A function node cancelled by another node's failure commits what it
wrote (`_commit_after_failure`) before the cancellation propagates, as a failing node does:
the job's flush commits neither its ephemeral store nor its result table ahead of its
the job's flush commits neither its ephemeral database nor its result table ahead of its
pipeline records (CodeRabbit, PR #379); it then holds no rows, as a node whose run raised
(ITL-1093). An observer whose `on_node_end` raises as the node fails does not skip that
commit either, as on the sync path, and is not called a second time. Rows run concurrently,
Expand Down
Loading
Loading