Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
b2a3a69
test(databases): failing regression tests for Delta read failures rep…
Nov 11, 2026
79141a5
fix(databases): a Delta table that cannot be read raises StoreReadErr…
Nov 11, 2026
c7ffc7d
docs: state the Delta read-failure and append type-check contract (IT…
Nov 11, 2026
e29890d
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
9a1ae63
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
5a06764
fix(databases): flush raises StoreReadError and the append refusal as…
Nov 11, 2026
3c75c09
fix(pipeline): raise StoreReadError on its own when another node also…
Nov 11, 2026
2e238c6
test(databases): regression tests for Delta reads missing their own w…
Nov 11, 2026
b727e0c
fix(databases): Delta reads return every row the handle family wrote …
Nov 11, 2026
ce32c9e
refactor(pipeline): give the async store-error selection #287's shape…
Nov 11, 2026
3785d71
docs(plans): name the async helper as implemented (ITL-884)
Nov 11, 2026
7d96a01
docs(plans): fix the remaining helper name in the plan's code sample …
Nov 11, 2026
e9c78b5
fix(databases): type Delta stored-id sets as bytes and fix test typin…
Nov 11, 2026
486db1d
docs(tests): docstrings for the store-read-failure tests (ITL-884)
Nov 11, 2026
7123a20
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
d32c84d
style(errors): unquote a local annotation ruff UP037 flags (ITL-884)
Nov 11, 2026
4b415a9
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
8f9adf8
fix(databases): retry a failed Delta stored-id refresh on the next re…
Nov 11, 2026
b8bcc81
fix(databases): Delta reads refresh missing stored ids and combine ev…
Nov 11, 2026
b4387c0
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
65c2bd9
docs(plans): resume notes after merging main (ITL-884)
Nov 11, 2026
ff90d33
fix(databases): Delta column-value query skips the side that lacks th…
Nov 11, 2026
61b2278
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
f6a3c3a
docs(errors): StoreReadError no longer claims nothing was recomputed …
Nov 11, 2026
0369cfc
refactor(databases): callers wrap store errors; no message-only param…
Nov 11, 2026
b55a3e6
fix(databases): the append check accepts new nested struct fields (IT…
Nov 11, 2026
488e2a2
docs(agents): add the no message-only parameters rule (ITL-884)
Nov 11, 2026
8f282b8
Merge origin/main (ITL-878 #287) into ITL-884: one path for store rea…
Nov 11, 2026
4f0b4f1
Merge origin/main (ITL-821 #296) into ITL-884
Nov 11, 2026
65c6330
fix(pipeline): name node and row on async pipeline-record store error…
Nov 11, 2026
cf029bb
Merge origin/main (ITL-875 #297) into ITL-884
Nov 11, 2026
ffe4448
fix(databases): the append check compares the types the Delta writer …
Nov 11, 2026
51dc905
fix(databases): null-typed values append to a typed Delta column (ITL…
Nov 11, 2026
11efe63
fix(databases): a null struct field inside a list no longer hides its…
Nov 11, 2026
f9b2c6c
docs(plans): plan for stacking ITL-864 on PR #290
Nov 11, 2026
7ba60eb
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
a2ac8d9
Merge remote-tracking branch 'origin/eywalker/itl-884-delta-read-fail…
Nov 11, 2026
ba396d4
test(databases): Delta reads raise StoreReadError with pending rows q…
Nov 11, 2026
c35bdf8
docs(plans): progress notes for stacking ITL-864 on #290
Nov 11, 2026
06ecbd5
fix(databases): pending-row filter costs O(pending) and keeps null id…
Nov 11, 2026
5aa4794
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
48678ca
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
97aa5a0
docs(changelog): move the ITL-864 entry to a changelog fragment (ITL-…
Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2380,6 +2380,67 @@ newcomer. Re-checked on `b197544e` (v0.2.0 + #291):
column ... because it's a binary field` lines on every Delta write still appear.
**Open:** awaiting the developer's choice of how to silence them.

---

## `src/orcapod/databases/delta_lake_databases.py` — reading back a handle family's own writes

### DW1 — A read through one `at()` handle could miss rows its family had just written
**Status:** resolved
**Severity:** high
**Issue:** ITL-864

Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()`
returned 1 of 2 or 3 rows (plain `int -> int` pod, sync and async orchestrators), while
`iter_data` and a fresh handle saw every row. Every `at()` handle shares the root's pending
batches, and any handle's `flush()` writes every pending key, so the result-store handle
flushed the pipeline table's rows. But each handle kept its own loaded-table cache, so the
node's pipeline handle kept reading the table version it had loaded during the run. Every read
method missed the rows a sibling flushed. Separately, the read methods merged pending and stored
rows inconsistently: `get_records_by_ids` ignored pending rows, `get_records_with_column_value`
ignored stored rows whenever a pending batch existed (and returned an empty table instead of
`None`), and a pending row whose record id was already stored was returned as well, although a
flush keeps the stored row and drops the pending one.

**Fix:** the loaded-table, existing-ids and cache-dirty dicts are shared across the `at()`
family exactly as pending state is. Every read method returns what a flush would leave: the
stored rows plus the pending rows whose record id is not stored (`_pending_not_stored`), and
`None` when there are none. These dicts are touched only from the thread that drives the
pipeline; the async orchestrator runs only user functions in worker threads. Regression tests:
`tests/test_databases/test_delta_read_your_writes.py` and
`tests/test_pipeline/test_delta_node_records_after_run.py`. A failed stored-id refresh leaves
the cache dirty, so the next read retries it instead of treating every stored id as unstored
and returning duplicates; in `get_records_by_ids` the refresh shares the stored read's error
handling. A handle that loaded the table before it had any stored ids (a fresh root over an
existing table) refreshes them on its first read with pending rows.

When stored and pending rows are read together, the result has the schema a flush would give:
both parts are large-normalized (`null` kept), then combined with
`pa.concat_tables(promote_options="default")`. A column missing on one side is null-filled, and a
`null`-typed column takes the other side's type. Columns whose types differ (`int64` vs `int32`,
a string vs an int, a timestamp unit, a decimal precision) raise `pyarrow.ArrowTypeError`; nothing
is widened. Before this, a read of stored rows next to pending rows with any schema difference,
including a pending `string` column next to the stored `large_string`, raised `ArrowInvalid`;
`get_all_records` already did so on `main`.
A `get_records_with_column_value` query on a column that only the stored or only the pending
rows have skips the side without it (after a flush that side holds null there, and
`column == value` never matches null), so it returns before a flush what it returns after; a
column neither side has still raises `ArrowInvalid`.

Recorded for the nullability audit (ITL-842) and ITL-890: Delta's merge relaxes nullability (a
non-null pending column into a nullable stored one stays nullable) and evolves struct children
(`struct<p>` stored + `struct<p, q>` pending becomes `struct<p, q>`, stored rows `q = null`). The
read path now mirrors both. Whether Orcapod should allow either at all is for those issues. A
pending all-null `null`-typed column reads back as `null`, but Delta cannot store it: the flush
fails and the batch stays pending (unchanged, pinned by
`test_pending_null_typed_column_reads_back_but_does_not_flush`).

Still open: a handle does not see writes made by another process, or by a separate
`DeltaTableDatabase` root on the same path, until it reloads the table; this is unchanged.
Reading with a pending batch also refreshes the stored-id set when it is stale, which loads the
whole table (D3).

---

## `src/orcapod/core/operators/filters.py` — `PolarsFilter` identity and nullability (ITL-828)

### PF1 — An expression predicate could not be hashed, and a filter broke a join inside a job
Expand Down
16 changes: 16 additions & 0 deletions changelog.d/ITL-864.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
### Fixed

#### Delta reads return every row the database's handles wrote (ITL-864)

Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` could
return only some of the rows (1 of 2 or 3) on both orchestrators. Handles made with `at()` share
their pending writes, and any handle's flush writes all of them, but each handle kept its own
loaded copy of the table, so a sibling kept reading the version it had loaded before the flush.
The loaded tables and the stored-id sets are now shared by every handle of one root. Every read
method (`get_all_records`, `get_record_by_id`, `get_records_by_ids`,
`get_records_with_column_value`) returns what a flush would leave: the stored rows plus the
pending rows whose record id is not stored, or `None` when there are none. Before,
`get_records_by_ids` ignored pending rows, `get_records_with_column_value` ignored stored rows
while a pending batch existed, and a pending row whose id was already stored could be returned
twice or instead of the stored row. Writes made by another process or another root on the same
path are still seen only after the table is reloaded. (ITL-864 #293)
Loading