Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
b2a3a69
test(databases): failing regression tests for Delta read failures rep…
Nov 11, 2026
79141a5
fix(databases): a Delta table that cannot be read raises StoreReadErr…
Nov 11, 2026
c7ffc7d
docs: state the Delta read-failure and append type-check contract (IT…
Nov 11, 2026
e29890d
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
9a1ae63
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
5a06764
fix(databases): flush raises StoreReadError and the append refusal as…
Nov 11, 2026
3c75c09
fix(pipeline): raise StoreReadError on its own when another node also…
Nov 11, 2026
2e238c6
test(databases): regression tests for Delta reads missing their own w…
Nov 11, 2026
b727e0c
fix(databases): Delta reads return every row the handle family wrote …
Nov 11, 2026
ce32c9e
refactor(pipeline): give the async store-error selection #287's shape…
Nov 11, 2026
3785d71
docs(plans): name the async helper as implemented (ITL-884)
Nov 11, 2026
7d96a01
docs(plans): fix the remaining helper name in the plan's code sample …
Nov 11, 2026
e9c78b5
fix(databases): type Delta stored-id sets as bytes and fix test typin…
Nov 11, 2026
a87cfd5
test(itl-892): exit-hang reproduction harness, matrix and temporary L…
Nov 11, 2026
ff2c546
fix(itl-892): type harness result dicts for basedpyright
Nov 11, 2026
486db1d
docs(tests): docstrings for the store-read-failure tests (ITL-884)
Nov 11, 2026
12ef298
test(itl-892): mitigation prototypes, native-fs equivalence check, re…
Nov 11, 2026
becdf14
fix(itl-892): basedpyright errors in experiment scripts
Nov 11, 2026
7123a20
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
d32c84d
style(errors): unquote a local annotation ruff UP037 flags (ITL-884)
Nov 11, 2026
4b415a9
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
8f9adf8
fix(databases): retry a failed Delta stored-id refresh on the next re…
Nov 11, 2026
286323c
docs(itl-892): exit-hang findings, Linux round 2 results and DESIGN_I…
Nov 11, 2026
07891f4
ci(itl-892): probe workflow runs only on manual dispatch
Nov 11, 2026
1fe07c5
docs(itl-892): final results, representative native stacks and regres…
Nov 11, 2026
27c39ea
fix(databases): scan local Delta tables through a native filesystem s…
Nov 11, 2026
c6d36ef
Merge origin/main into ITL-892
Nov 11, 2026
b8bcc81
fix(databases): Delta reads refresh missing stored ids and combine ev…
Nov 11, 2026
5658c5a
docs(itl-892): troubleshooting note, changelog, design issue resolved…
Nov 11, 2026
561f599
docs(itl-892): proposed delta-rs upstream issue text
Nov 11, 2026
b4387c0
Merge remote-tracking branch 'origin/main' into eywalker/itl-884-delt…
Nov 11, 2026
65c2bd9
docs(plans): resume notes after merging main (ITL-884)
Nov 11, 2026
ff90d33
fix(databases): Delta column-value query skips the side that lacks th…
Nov 11, 2026
61b2278
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
06071a0
fix(databases): decode percent-encoded local Delta table roots for th…
Nov 11, 2026
7552149
docs(itl-892): docstrings, resolved-root troubleshooting snippet, str…
Nov 11, 2026
01cad57
Merge remote-tracking branch 'origin/main' into eywalker/itl-892-proc…
Nov 11, 2026
e2b3db3
docs(itl-892): decode table_uri in the proposed delta-rs workaround
Nov 11, 2026
934417e
docs(itl-892): plan progress after CodeRabbit round 1
Nov 11, 2026
f6a3c3a
docs(errors): StoreReadError no longer claims nothing was recomputed …
Nov 11, 2026
0369cfc
refactor(databases): callers wrap store errors; no message-only param…
Nov 11, 2026
b55a3e6
fix(databases): the append check accepts new nested struct fields (IT…
Nov 11, 2026
488e2a2
docs(agents): add the no message-only parameters rule (ITL-884)
Nov 11, 2026
8f282b8
Merge origin/main (ITL-878 #287) into ITL-884: one path for store rea…
Nov 11, 2026
4f0b4f1
Merge origin/main (ITL-821 #296) into ITL-884
Nov 11, 2026
65c6330
fix(pipeline): name node and row on async pipeline-record store error…
Nov 11, 2026
cf029bb
Merge origin/main (ITL-875 #297) into ITL-884
Nov 11, 2026
ffe4448
fix(databases): the append check compares the types the Delta writer …
Nov 11, 2026
51dc905
fix(databases): null-typed values append to a typed Delta column (ITL…
Nov 11, 2026
11efe63
fix(databases): a null struct field inside a list no longer hides its…
Nov 11, 2026
f9b2c6c
docs(plans): plan for stacking ITL-864 on PR #290
Nov 11, 2026
7ba60eb
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
a2ac8d9
Merge remote-tracking branch 'origin/eywalker/itl-884-delta-read-fail…
Nov 11, 2026
ba396d4
test(databases): Delta reads raise StoreReadError with pending rows q…
Nov 11, 2026
c35bdf8
docs(plans): progress notes for stacking ITL-864 on #290
Nov 11, 2026
06ecbd5
fix(databases): pending-row filter costs O(pending) and keeps null id…
Nov 11, 2026
0b29ea4
docs(itl-892): plan for rebuilding on the #293 stack
Nov 11, 2026
5aa4794
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
54bda57
Merge remote-tracking branch 'origin/eywalker/itl-864-deltatabledatab…
Nov 11, 2026
ab8723c
Merge remote-tracking branch 'origin/main' into eywalker/itl-892-proc…
Nov 11, 2026
00e724f
test(databases): filtered Delta reads on the native reader keep the r…
Nov 11, 2026
1f73fd1
docs(itl-892): plan progress for the #293 stack rebuild
Nov 11, 2026
48678ca
Merge remote-tracking branch 'origin/main' into eywalker/itl-864-delt…
Nov 11, 2026
5702cd8
docs(itl-892): plan for merging the rebuilt #293 head
Nov 11, 2026
f2c45c1
Merge remote-tracking branch 'origin/eywalker/itl-864-deltatabledatab…
Nov 11, 2026
14c1546
docs(itl-892): plan progress for the #293 head merge
Nov 11, 2026
97aa5a0
docs(changelog): move the ITL-864 entry to a changelog fragment (ITL-…
Nov 12, 2026
80c8368
Merge remote-tracking branch 'origin/eywalker/itl-864-deltatabledatab…
Nov 12, 2026
2902490
docs(changelog): move the ITL-892 entry to a changelog fragment (ITL-…
Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2380,6 +2380,114 @@ newcomer. Re-checked on `b197544e` (v0.2.0 + #291):
column ... because it's a binary field` lines on every Delta write still appear.
**Open:** awaiting the developer's choice of how to silence them.

---

## `src/orcapod/databases/delta_lake_databases.py` — reading back a handle family's own writes

### DW1 — A read through one `at()` handle could miss rows its family had just written
**Status:** resolved
**Severity:** high
**Issue:** ITL-864

Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()`
returned 1 of 2 or 3 rows (plain `int -> int` pod, sync and async orchestrators), while
`iter_data` and a fresh handle saw every row. Every `at()` handle shares the root's pending
batches, and any handle's `flush()` writes every pending key, so the result-store handle
flushed the pipeline table's rows. But each handle kept its own loaded-table cache, so the
node's pipeline handle kept reading the table version it had loaded during the run. Every read
method missed the rows a sibling flushed. Separately, the read methods merged pending and stored
rows inconsistently: `get_records_by_ids` ignored pending rows, `get_records_with_column_value`
ignored stored rows whenever a pending batch existed (and returned an empty table instead of
`None`), and a pending row whose record id was already stored was returned as well, although a
flush keeps the stored row and drops the pending one.

**Fix:** the loaded-table, existing-ids and cache-dirty dicts are shared across the `at()`
family exactly as pending state is. Every read method returns what a flush would leave: the
stored rows plus the pending rows whose record id is not stored (`_pending_not_stored`), and
`None` when there are none. These dicts are touched only from the thread that drives the
pipeline; the async orchestrator runs only user functions in worker threads. Regression tests:
`tests/test_databases/test_delta_read_your_writes.py` and
`tests/test_pipeline/test_delta_node_records_after_run.py`. A failed stored-id refresh leaves
the cache dirty, so the next read retries it instead of treating every stored id as unstored
and returning duplicates; in `get_records_by_ids` the refresh shares the stored read's error
handling. A handle that loaded the table before it had any stored ids (a fresh root over an
existing table) refreshes them on its first read with pending rows.

When stored and pending rows are read together, the result has the schema a flush would give:
both parts are large-normalized (`null` kept), then combined with
`pa.concat_tables(promote_options="default")`. A column missing on one side is null-filled, and a
`null`-typed column takes the other side's type. Columns whose types differ (`int64` vs `int32`,
a string vs an int, a timestamp unit, a decimal precision) raise `pyarrow.ArrowTypeError`; nothing
is widened. Before this, a read of stored rows next to pending rows with any schema difference,
including a pending `string` column next to the stored `large_string`, raised `ArrowInvalid`;
`get_all_records` already did so on `main`.
A `get_records_with_column_value` query on a column that only the stored or only the pending
rows have skips the side without it (after a flush that side holds null there, and
`column == value` never matches null), so it returns before a flush what it returns after; a
column neither side has still raises `ArrowInvalid`.

Recorded for the nullability audit (ITL-842) and ITL-890: Delta's merge relaxes nullability (a
non-null pending column into a nullable stored one stays nullable) and evolves struct children
(`struct<p>` stored + `struct<p, q>` pending becomes `struct<p, q>`, stored rows `q = null`). The
read path now mirrors both. Whether Orcapod should allow either at all is for those issues. A
pending all-null `null`-typed column reads back as `null`, but Delta cannot store it: the flush
fails and the batch stays pending (unchanged, pinned by
`test_pending_null_typed_column_reads_back_but_does_not_flush`).

Still open: a handle does not see writes made by another process, or by a separate
`DeltaTableDatabase` root on the same path, until it reloads the table; this is unchanged.
Reading with a pending batch also refreshes the stored-id set when it is stale, which loads the
whole table (D3).

---

## `src/orcapod/databases/delta_lake_databases.py`, `src/orcapod/core/sources/delta_table_source.py` — exit after a Delta read

### DX1 — A process that read a Delta table can hang or abort at interpreter exit
**Status:** resolved (local stores); cloud stores remain exposed upstream
**Severity:** high
**Issue:** ITL-892

`DeltaTable.to_pyarrow_dataset()` (and `to_pyarrow_table()`) reads through deltalake's
`PyFileSystem(DeltaStorageHandler)`, a filesystem implemented in Python. After a scan returns,
Arrow finishes tearing down the scan on a thread-pool worker. If that worker drops the last
reference to a Python-backed buffer during interpreter finalization, CPython stops the thread,
and `ThreadPool::~ThreadPool` in `__cxa_finalize` waits for it forever. On Linux with Python
3.11/3.12 the process may abort with SIGABRT instead. This is upstream:
apache/arrow#50482 and #54536, with an open fix PR, #54467. The failure follows a scan, not a
write.

Measured over 100 runs each, locked versions: one write plus one scan fails 57% of runs on
macOS arm64 and 5–38% on Linux x86_64. A fresh `DeltaTableDatabase.get_all_records` fails 53%
on macOS and 8–23% on Linux. `DeltaTableSource` fails 4–10%. A `PipelineJob` run fails 0–4%.
Every pyarrow 20.0.0–26.0.1 × deltalake 1.0.2/1.6.6 combination fails, so no version pin helps.
Scanning local tables through a native pyarrow filesystem (`SubTreeFileSystem(table_path,
LocalFileSystem())`) gives 0/100 in every case. Its results are byte-identical. Cloud stores
keep the Python filesystem and stay exposed until pyarrow ships the fix. Findings, scripts and
raw results: `superpowers/specs/itl-892-exit-hang/FINDINGS.md`.

**Fix:** `storage_utils.delta_to_pyarrow_dataset` opens local Delta tables on a native pyarrow
filesystem (`SubTreeFileSystem` over `LocalFileSystem`). Every scan in `DeltaTableDatabase` and
`DeltaTableSource` goes through it. Cloud URIs keep deltalake's reader. The results are
schema- and value-identical: `tests/test_databases/test_delta_native_fs_read.py` compares them
with deltalake's reader for logical types (Path, dataclass, numpy, pandas), partitioned tables
and escaped partition values, and for table roots whose names need percent-encoding
(`table_uri` is percent-encoded, so the helper decodes it with `pyarrow.fs.FileSystem.from_uri`;
a root that is neither a `file://` URI nor an absolute POSIX path, such as a Windows drive path,
is untested and keeps deltalake's reader). `tests/test_databases/test_delta_exit.py` runs fresh
interpreters and fails if one does not exit cleanly; it failed before the fix. The remaining
cloud-store exposure is documented in `docs/troubleshooting.md`.

**Considered alternative, not implemented:** an `atexit` hook that releases the GIL briefly
(`time.sleep(0.05)`) before finalization, the idea of apache/arrow#54467. It measured 0/100 on
macOS and Linux and also covers cloud stores. But it is timing-based, so a loaded machine or a
large scan can outlast it. It would also run in every process that imports Orcapod. If cloud
stores need covering before pyarrow ships the fix, register it lazily on the first cloud-store
scan. When pyarrow ships #54467, bump the floor and reconsider whether the helper is still
needed.

---

## `src/orcapod/core/operators/filters.py` — `PolarsFilter` identity and nullability (ITL-828)

### PF1 — An expression predicate could not be hashed, and a filter broke a join inside a job
Expand Down
16 changes: 16 additions & 0 deletions changelog.d/ITL-864.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
### Fixed

#### Delta reads return every row the database's handles wrote (ITL-864)

Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` could
return only some of the rows (1 of 2 or 3) on both orchestrators. Handles made with `at()` share
their pending writes, and any handle's flush writes all of them, but each handle kept its own
loaded copy of the table, so a sibling kept reading the version it had loaded before the flush.
The loaded tables and the stored-id sets are now shared by every handle of one root. Every read
method (`get_all_records`, `get_record_by_id`, `get_records_by_ids`,
`get_records_with_column_value`) returns what a flush would leave: the stored rows plus the
pending rows whose record id is not stored, or `None` when there are none. Before,
`get_records_by_ids` ignored pending rows, `get_records_with_column_value` ignored stored rows
while a pending batch existed, and a pending row whose id was already stored could be returned
twice or instead of the stored row. Writes made by another process or another root on the same
path are still seen only after the table is reloaded. (ITL-864 #293)
16 changes: 16 additions & 0 deletions changelog.d/ITL-892.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
### Fixed

#### A script that read a local Delta store no longer hangs at exit (ITL-892)

A process that had read a Delta table through `DeltaTableDatabase` or `DeltaTableSource` could
finish its work and then never exit: it hung in Arrow's thread-pool destructor, or, on Linux with
Python 3.11 or 3.12, aborted with `terminate called without an active exception`. It happened in
up to three runs in four on macOS and about one in ten on Linux, on every supported pyarrow and
deltalake version. The cause is upstream (apache/arrow#50482, apache/arrow#54536). deltalake's
default reader goes through a filesystem written in Python, and Arrow can release one of its
buffers on a worker thread while the interpreter is shutting down.

Orcapod now reads local Delta tables through a native pyarrow filesystem, which returns the same
schema and values. Tables on cloud storage (`s3://`, `gs://`, `az://`) still use deltalake's
reader and stay exposed until pyarrow ships the fix; see
[Troubleshooting](docs/troubleshooting.md).
39 changes: 39 additions & 0 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Troubleshooting

## A script finishes its work but the process never exits

**Symptom.** A Python script that used a Delta store prints all of its output and then hangs,
and has to be killed. On Linux with Python 3.11 or 3.12 the process may instead end with
`terminate called without an active exception` and exit code 134 (SIGABRT). Notebooks are not
affected in the same way, because the kernel keeps running.

**Cause.** This is a bug in pyarrow, not in your pipeline
([apache/arrow#50482](https://github.com/apache/arrow/issues/50482),
[apache/arrow#54536](https://github.com/apache/arrow/issues/54536)). deltalake's default reader
goes through a filesystem written in Python. After a read returns, Arrow finishes cleaning it up
on a worker thread. If that cleanup runs while the interpreter is shutting down, the worker
cannot take the GIL and never finishes, and Arrow's thread pool waits for it forever. Your data
is not affected: the hang happens after every write has completed.

**What Orcapod does.** `DeltaTableDatabase` and `DeltaTableSource` read **local** Delta tables
through a native pyarrow filesystem, so they are not affected. Tables on cloud storage
(`s3://`, `gs://`, `az://` and similar) still use deltalake's reader and can still hang at exit.
Reading a Delta table yourself with `deltalake.DeltaTable(...).to_pyarrow_table()` or
`to_pyarrow_dataset()` is affected too, whatever the storage.

**Workarounds.**

- Upgrade pyarrow once a release includes the fix
([apache/arrow#54467](https://github.com/apache/arrow/pull/54467)).
- In a script that reads a cloud Delta store, end it by flushing your outputs and calling
`os._exit(0)`. That skips interpreter shutdown, where the hang happens. It also skips
`atexit` handlers and other cleanup, so close files and flush stores first.
- When reading local Delta tables directly with deltalake, pass a native filesystem. Resolve
`dt.table_uri` first: it is percent-encoded, and newer deltalake versions return a
`file://` URI.

```python
uri = dt.table_uri if "://" in dt.table_uri else "file://" + dt.table_uri
fs, root = pyarrow.fs.FileSystem.from_uri(uri)
dataset = dt.to_pyarrow_dataset(filesystem=pyarrow.fs.SubTreeFileSystem(root, fs))
```
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,4 @@ nav:
- Types: api/types.md
- Reference:
- Hashing: reference/hashing.md
- Troubleshooting: troubleshooting.md
5 changes: 3 additions & 2 deletions src/orcapod/core/sources/delta_table_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

from orcapod.core.sources.base import RootSource
from orcapod.core.sources.stream_builder import SourceStreamBuilder
from orcapod.databases.storage_utils import delta_to_pyarrow_dataset
from orcapod.types import PathLike
from orcapod.utils.lazy_module import LazyModule
from orcapod.utils.schema_utils import _normalize_column_list
Expand Down Expand Up @@ -49,8 +50,8 @@ def __init__(
except deltalake.exceptions.TableNotFoundError:
raise ValueError(f"Delta table not found at {self._delta_table_path}")

table: pa.Table = self._delta_table.to_pyarrow_dataset(
as_large_types=True
table: pa.Table = delta_to_pyarrow_dataset(
self._delta_table, as_large_types=True
).to_table()
# The Delta log stores field-level nullable flags; to_pyarrow_dataset()
# preserves them exactly — no inference needed.
Expand Down
Loading