diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 695708a20..aad884ef1 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2382,6 +2382,117 @@ newcomer. Re-checked on `b197544e` (v0.2.0 + #291): --- +## `src/orcapod/databases/delta_lake_databases.py` — reading back a handle family's own writes + +### DW1 — A read through one `at()` handle could miss rows its family had just written +**Status:** resolved +**Severity:** high +**Issue:** ITL-864 + +Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` +returned 1 of 2 or 3 rows (plain `int -> int` pod, sync and async orchestrators), while +`iter_data` and a fresh handle saw every row. Every `at()` handle shares the root's pending +batches, and any handle's `flush()` writes every pending key, so the result-store handle +flushed the pipeline table's rows. But each handle kept its own loaded-table cache, so the +node's pipeline handle kept reading the table version it had loaded during the run. Every read +method missed the rows a sibling flushed. Separately, the read methods merged pending and stored +rows inconsistently: `get_records_by_ids` ignored pending rows, `get_records_with_column_value` +ignored stored rows whenever a pending batch existed (and returned an empty table instead of +`None`), and a pending row whose record id was already stored was returned as well, although a +flush keeps the stored row and drops the pending one. + +**Fix:** the loaded-table cache and the committed snapshots (`_CommittedSnapshot`, which +replaced the existing-ids and cache-dirty dicts in ITL-919) are shared across the `at()` +family exactly as pending state (pending batches and their indexes) is. Every read method returns what a flush would leave: the +stored rows plus the pending rows whose record id is not stored (`_pending_not_stored`), and +`None` when there are none. These dicts are touched only from the thread that drives the +pipeline; the async orchestrator runs only user functions in worker threads. Regression tests: +`tests/test_databases/test_delta_read_your_writes.py` and +`tests/test_pipeline/test_delta_node_records_after_run.py`. Stored ids come from +`_stored_ids`: the snapshot of the loaded table version reads them on first use, through an +id-only scan (a one-column schema); a failed scan raises `StoreReadError` and leaves the +snapshot's ids unset (`None`), so the next read scans again instead of treating every stored +id as unstored and returning duplicates. A handle that loaded the table before it had any +stored ids (a fresh root over an existing table) reads them on its first read with pending +rows (`_pending_not_stored`). After the family's own commit the ids are extended from the +commit itself, with no re-read (ITL-919). + +When stored and pending rows are read together, the result has the schema a flush would give: +both parts are large-normalized (`null` kept), then combined with +`pa.concat_tables(promote_options="default")`. A column missing on one side is null-filled, and a +`null`-typed column takes the other side's type. Columns whose types differ (`int64` vs `int32`, +a string vs an int, a timestamp unit, a decimal precision) raise `pyarrow.ArrowTypeError`; nothing +is widened. Before this, a read of stored rows next to pending rows with any schema difference, +including a pending `string` column next to the stored `large_string`, raised `ArrowInvalid`; +`get_all_records` already did so on `main`. +A `get_records_with_column_value` query on a column that only the stored or only the pending +rows have skips the side without it (after a flush that side holds null there, and +`column == value` never matches null), so it returns before a flush what it returns after; a +column neither side has still raises `ArrowInvalid`. + +Recorded for the nullability audit (ITL-842) and ITL-890: Delta's merge relaxes nullability (a +non-null pending column into a nullable stored one stays nullable) and evolves struct children +(`struct

` stored + `struct` pending becomes `struct`, stored rows `q = null`). The +read path now mirrors both. Whether Orcapod should allow either at all is for those issues. A +pending all-null `null`-typed column reads back as `null`, but Delta cannot store it: the flush +fails and the batch stays pending (unchanged, pinned by +`test_pending_null_typed_column_reads_back_but_does_not_flush`). + +Still open: a handle does not see writes made by another process, or by a separate +`DeltaTableDatabase` root on the same path, until its own next commit to that table reloads it; +this is unchanged. When it does reload (another writer committed in between), the stored ids +are read again with the id-only scan, and the stored rows on the next lookup, each once per +version (D3, now bounded by ITL-919's snapshot). + +--- + +## `src/orcapod/databases/delta_lake_databases.py`, `src/orcapod/core/sources/delta_table_source.py` — exit after a Delta read + +### DX1 — A process that read a Delta table can hang or abort at interpreter exit +**Status:** resolved (local stores); cloud stores remain exposed upstream +**Severity:** high +**Issue:** ITL-892 + +`DeltaTable.to_pyarrow_dataset()` (and `to_pyarrow_table()`) reads through deltalake's +`PyFileSystem(DeltaStorageHandler)`, a filesystem implemented in Python. After a scan returns, +Arrow finishes tearing down the scan on a thread-pool worker. If that worker drops the last +reference to a Python-backed buffer during interpreter finalization, CPython stops the thread, +and `ThreadPool::~ThreadPool` in `__cxa_finalize` waits for it forever. On Linux with Python +3.11/3.12 the process may abort with SIGABRT instead. This is upstream: +apache/arrow#50482 and #54536, with an open fix PR, #54467. The failure follows a scan, not a +write. + +Measured over 100 runs each, locked versions: one write plus one scan fails 57% of runs on +macOS arm64 and 5–38% on Linux x86_64. A fresh `DeltaTableDatabase.get_all_records` fails 53% +on macOS and 8–23% on Linux. `DeltaTableSource` fails 4–10%. A `PipelineJob` run fails 0–4%. +Every pyarrow 20.0.0–26.0.1 × deltalake 1.0.2/1.6.6 combination fails, so no version pin helps. +Scanning local tables through a native pyarrow filesystem (`SubTreeFileSystem(table_path, +LocalFileSystem())`) gives 0/100 in every case. Its results are byte-identical. Cloud stores +keep the Python filesystem and stay exposed until pyarrow ships the fix. Findings, scripts and +raw results: `superpowers/specs/itl-892-exit-hang/FINDINGS.md`. + +**Fix:** `storage_utils.delta_to_pyarrow_dataset` opens local Delta tables on a native pyarrow +filesystem (`SubTreeFileSystem` over `LocalFileSystem`). Every scan in `DeltaTableDatabase` and +`DeltaTableSource` goes through it. Cloud URIs keep deltalake's reader. The results are +schema- and value-identical: `tests/test_databases/test_delta_native_fs_read.py` compares them +with deltalake's reader for logical types (Path, dataclass, numpy, pandas), partitioned tables +and escaped partition values, and for table roots whose names need percent-encoding +(`table_uri` is percent-encoded, so the helper decodes it with `pyarrow.fs.FileSystem.from_uri`; +a root that is neither a `file://` URI nor an absolute POSIX path, such as a Windows drive path, +is untested and keeps deltalake's reader). `tests/test_databases/test_delta_exit.py` runs fresh +interpreters and fails if one does not exit cleanly; it failed before the fix. The remaining +cloud-store exposure is documented in `docs/troubleshooting.md`. + +**Considered alternative, not implemented:** an `atexit` hook that releases the GIL briefly +(`time.sleep(0.05)`) before finalization, the idea of apache/arrow#54467. It measured 0/100 on +macOS and Linux and also covers cloud stores. But it is timing-based, so a loaded machine or a +large scan can outlast it. It would also run in every process that imports Orcapod. If cloud +stores need covering before pyarrow ships the fix, register it lazily on the first cloud-store +scan. When pyarrow ships #54467, bump the floor and reconsider whether the helper is still +needed. + +--- + ## `src/orcapod/core/operators/filters.py` — `PolarsFilter` identity and nullability (ITL-828) ### PF1 — An expression predicate could not be hashed, and a filter broke a join inside a job @@ -2415,6 +2526,58 @@ function pod downstream of a filter, so a 0.2 store recomputes those results onc --- +## `src/orcapod/databases/` — PipelineJob run time quadratic in rows + +### PQ1 — Every row a function node processes scans the rows already stored +**Status:** resolved +**Severity:** high +**Issue:** ITL-919 + +A function node in a `PipelineJob` does, per row: `ResultCache.lookup` (lookup by input data +hash in the result table), `ResultCache.store` with a flush of the whole store (both +tables), and `add_pipeline_record` (lookup by base entry id for the recomputation index, +then `add_record(skip_duplicates=True)`). Sync and async orchestrators share that path; +operator nodes write with one `add_records` and are not affected. Measured on `c8efc092`, +in-memory store, sync: 7.2 / 20.9 / 64 ms per row at 100 / 400 / 1,000 rows (standalone +pod: 0.09 ms). + +- `InMemoryArrowDatabase`: each lookup filtered the whole table, the duplicate check built + the set of every committed id, and each flush ran `is_in` over all ids, filtered and + concatenated, adding one chunk per row, so every later Arrow operation cost O(rows). + **Fix:** lazily built hash indexes over committed rows (binary and string columns, record + ids), extended on append and rebuilt only after a flush overwrites an id; flush keeps the + committed table at O(log N) record batches by binary-counter compaction. Stored rows, row + order, ids and schemas are unchanged (differential check against `main` on a joined + multi-node pipeline, both orchestrators). Store work per row is now independent of rows + stored (tests count Arrow bytes allocated per row). +- `DeltaTableDatabase`: one Delta MERGE commit per row per table (cost grows with the + table and leaves one file per row), lookups re-opened and read every file, the + existing-id cache was re-read after every flush, and the pending batch was scanned per + lookup (main: 80 / 198 / 509 ms per row at 100 / 400 / 1,000 rows, sync). **Fix:** function + nodes inside `PipelineJob.run` commit every `commit_every_rows` rows (default 100), at node + end, best-effort on failure and at job end, results before pipeline records; a node used + on its own still commits per row. The store keeps, per loaded table version, the stored + ids and rows in memory (indexed), extended after its own commits from the new data files + only; the pending batch is compacted and indexed; a commit appends when every pending id is + new in the version loaded for that commit, and MERGEs otherwise. Stored rows, ids, hashes, + paths and schemas are unchanged (differential against `main`). Store failures that only a + commit reveals now surface at the next commit point and name no row (developer decision, + November 11, 2026). Found on the way: an async run hung on any node failure with more rows + in flight than the channel buffers (ITL-932, fixed: closing a channel never blocks), and an + async rerun of an ephemeral-result node skipped its rows (ITL-923, fixed). Remaining + (ITL-923 item 3): a pending row written with `skip_duplicates=False` whose id is already + stored is still dropped by the commit (Delta keeps the stored row; the in-memory store + overwrites it). Memory: a store object keeps the committed rows of each table it looks rows + up in for as long as it lives, all of them at once (main read one whole table per lookup + and released it), so memory is cumulative over the tables: measured for an `int -> int` + pod, about 820 bytes per stored row for the result table and 746 for the pipeline-record + table, so about 1.6 GB per function node at 1M rows. Keeping only the lookup columns would + need a read per matched row. A follow-up to bound it is for the coordinator to file. The store's own + size-triggered flush (`batch_size`) now commits only the full table, and `PipelineJob.run` + refuses `commit_every_rows >= batch_size`, so the results-first order holds. + +--- + ## `src/orcapod/logical_types/numpy_type.py`, `src/orcapod/logical_types/array_format.py` — array format ### AR1 — `numpy.ndarray` stored as a `.npy` file and hashed by file bytes diff --git a/changelog.d/ITL-864.md b/changelog.d/ITL-864.md new file mode 100644 index 000000000..cfc4bc3dc --- /dev/null +++ b/changelog.d/ITL-864.md @@ -0,0 +1,16 @@ +### Fixed + +#### Delta reads return every row the database's handles wrote (ITL-864) + +Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` could +return only some of the rows (1 of 2 or 3) on both orchestrators. Handles made with `at()` share +their pending writes, and any handle's flush writes all of them, but each handle kept its own +loaded copy of the table, so a sibling kept reading the version it had loaded before the flush. +The loaded tables and the stored-id sets are now shared by every handle of one root. Every read +method (`get_all_records`, `get_record_by_id`, `get_records_by_ids`, +`get_records_with_column_value`) returns what a flush would leave: the stored rows plus the +pending rows whose record id is not stored, or `None` when there are none. Before, +`get_records_by_ids` ignored pending rows, `get_records_with_column_value` ignored stored rows +while a pending batch existed, and a pending row whose id was already stored could be returned +twice or instead of the stored row. Writes made by another process or another root on the same +path are still seen only after the table is reloaded. (ITL-864 #293) diff --git a/changelog.d/ITL-892.md b/changelog.d/ITL-892.md new file mode 100644 index 000000000..f6510cb02 --- /dev/null +++ b/changelog.d/ITL-892.md @@ -0,0 +1,16 @@ +### Fixed + +#### A script that read a local Delta store no longer hangs at exit (ITL-892) + +A process that had read a Delta table through `DeltaTableDatabase` or `DeltaTableSource` could +finish its work and then never exit: it hung in Arrow's thread-pool destructor, or, on Linux with +Python 3.11 or 3.12, aborted with `terminate called without an active exception`. It happened in +up to three runs in four on macOS and about one in ten on Linux, on every supported pyarrow and +deltalake version. The cause is upstream (apache/arrow#50482, apache/arrow#54536). deltalake's +default reader goes through a filesystem written in Python, and Arrow can release one of its +buffers on a worker thread while the interpreter is shutting down. + +Orcapod now reads local Delta tables through a native pyarrow filesystem, which returns the same +schema and values. Tables on cloud storage (`s3://`, `gs://`, `az://`) still use deltalake's +reader and stay exposed until pyarrow ships the fix; see +[Troubleshooting](docs/troubleshooting.md). diff --git a/changelog.d/ITL-919.md b/changelog.d/ITL-919.md new file mode 100644 index 000000000..f7312d516 --- /dev/null +++ b/changelog.d/ITL-919.md @@ -0,0 +1,52 @@ +### Added + +- **`PipelineJob.run(commit_every_rows=100)`** — how many rows a function node processes + between commits of its results and pipeline records (an `int >= 1`). Nodes also commit when + they finish, before a failed run raises, and at the end of the job; `1` commits after every + row. A hard crash on a durable store loses at most the rows the running node processed + since its last commit, and the rerun computes only those. (ITL-919) + +### Fixed + +#### A `PipelineJob` on an in-memory store runs in time linear in its rows (ITL-919) + +A `PipelineJob` on an `InMemoryArrowDatabase` runs in time linear in its rows. Every row a +function node processed used to filter, copy or rebuild the whole stored table (the +result lookup, the recomputation-index lookup, the duplicate check and the flush, each +over a table that gained one chunk per row), so 1,000 rows took about a minute. Lookups +now use hash indexes over the stored rows and a flush appends without copying them; +stored rows, ids and hashes are unchanged. + +#### A `PipelineJob` on a Delta store runs in time linear in its rows (ITL-919) + +A `PipelineJob` on a `DeltaTableDatabase` committed every row of every function node as +its own Delta version (a MERGE that reads the whole table) and re-read the stored table for +each per-row lookup, so 1,000 rows took about eight minutes. Function nodes inside +`job.run()` now commit every `commit_every_rows` rows (default 100, see *Added*); a batch +whose record ids are all new is appended instead of merged; and lookups are answered from +an indexed in-memory copy of the committed rows, read once and extended with each commit. +Stored rows, record ids, hashes, paths and schemas are unchanged. Behaviour changes: a store +failure that only a commit reveals (an unreadable table, a failed commit) surfaces at the +node's next commit point and names the node but no row, and the rows computed since the +last commit are computed again on the next run; `commit_every_rows=1` restores per-row +surfacing. A `DeltaTableDatabase` object keeps answering from the table version it loaded +until its own next commit (as before; it no longer re-reads the table per lookup). + +#### An asynchronous run whose node fails raises instead of hanging (ITL-932) + +On the asynchronous orchestrator, a node failure while more rows were in flight than the +channel buffers hold (about 16 to 64 rows) hung the run forever: the failure cancelled the +readers, and closing a full channel waited for room that never came. Closing a channel no +longer blocks; readers still receive every item sent before the close. + +#### An asynchronous rerun of an ephemeral-result node recomputes its rows (ITL-923) + +Rerunning a node with `NodeConfig(is_result_ephemeral=True)` on the asynchronous +orchestrator, after its ephemeral results were gone, neither recomputed nor recorded the +rows, and reading the node afterwards raised `EmptyDataAccessError`. It now recomputes them, +as the synchronous orchestrator does. + +#### A failed side-effect completion lookup names the row + +A `StoreReadError` from a side-effect pod's check of earlier deliveries now names the pod +and the row, as a failed write of the delivery record does. diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index 4a6a5aa77..a79cb207f 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -248,6 +248,52 @@ print("function calls:", len(calls)) flush tries them again. A flush that fails for any other reason raises `RuntimeError`, with the first failure as its cause. +### When results are committed + +Inside `job.run()`, each function node commits its results and pipeline records to the +store in batches: every `commit_every_rows` rows (default 100), when the node finishes, +before a failed run raises, and when the job ends. Committing in batches keeps a run's time +linear in its rows; on a `DeltaTableDatabase` each commit is one Delta version per table, so +a 1,000-row node writes 10 versions per table instead of 1,000. + +```python +job.run(commit_every_rows=500) # fewer, larger commits +``` + +`commit_every_rows=1` commits after every row. A node or a cached pod used on its own, +outside `job.run()`, also commits after every row. On a `DeltaTableDatabase`, +`commit_every_rows` must be smaller than the store's `batch_size` (default 1000): +the store commits every pending table by itself once one table holds that many pending +rows, which would not keep the order described below, so `job.run()` refuses a larger value +with `ValueError`. Create the store with a larger `batch_size` to commit less often. For the same reason, `job.run()` first commits +whatever such a store already holds pending (writes made without a flush, or the batches of a +run that failed); if that fails, the run stops before computing anything. + +**Memory.** A `DeltaTableDatabase` keeps the committed rows of every table it looks rows up +in (each function node's result and pipeline-record tables) in memory for as long as the +store object lives, so that a lookup never re-reads the table. These copies are held +together, so plan memory for all of them at once: for a function returning one `int`, about +1.6 KB per stored row per function node (result and pipeline-record tables together), more +for larger outputs. Before, each lookup read one whole table and released it. To release +the copies, use a new store object, for example one per job. + +**What a hard crash loses.** If the process dies without raising (power loss, `kill -9`, +the machine restarts), a durable store such as `DeltaTableDatabase` keeps every batch +committed before the crash; at most the rows the running node processed since its last +commit are lost, and nothing stored is damaged. Run the job again: it computes only the rows +whose results were not committed. A node commits its results before their pipeline records, +so a crash between those two commits leaves results stored without pipeline records. Reading +the node (`as_table()`, `get_all_records()`) does not show those rows until the next run, +which finds their results in the store, computes nothing for them and writes their pipeline +records. + +**Reads see the version this process loaded.** A `DeltaTableDatabase` keeps the committed +rows of each table it has read in memory and answers lookups from them, extending them with +its own commits. Rows another process commits to the same table are not seen by a store +object that has already loaded the table until its own next commit to that table reloads it; +a new `DeltaTableDatabase` object reads them at once. This was already the case before +batching; only the re-reading per lookup is gone. + A node from a job that has run also provides: - **`get_all_records()`** -- every record this step of the job has stored, as a PyArrow @@ -263,13 +309,20 @@ A node from a job that has run also provides: A failed write to the store always fails the run. If a node's results or records cannot be stored -- the disk is full, the connection drops, or the database cannot hold one of the output types -- `job.run()` raises `StoreWriteError` (from `orcapod.errors`) at the first -failed write, on both the synchronous and the asynchronous orchestrator. The error names the -node, the row being stored and the table, and the database's own exception is chained as its -cause. When committing pending writes fails, the error has `flush=True` and names the store -instead of a row and table: every view of a store shares its pending writes, so the write -that failed may belong to an earlier row or another table. A run that returns normally has -stored every result that could be turned into a record; a result rejected before that is a -row failure (see below). +failed write, on both the synchronous and the asynchronous orchestrator. A row's write that +fails names the node, the row being stored and the table, and the database's own exception +is chained as its cause. When committing pending writes fails, the error has `flush=True` +and names the node and the store instead of a row and table: a commit holds a batch of rows +(see *When results are committed*), and every view of a store shares its pending writes, so +the write that failed may belong to any row of the batch or another table. A run that +returns normally has stored every result that could be turned into a record; a result +rejected before that is a row failure (see below). + +With batched commits, a store failure that only committing reveals (an unreadable table, a +full disk) surfaces at the node's next commit point, not at the row that caused it: the rows +the node computed since its last commit are not stored and are computed again on the next +run. `job.run(commit_every_rows=1)` commits after every row, so such a failure surfaces at +its row, at the cost of one commit per row. This holds whatever `error_policy` the orchestrator was given. `error_policy="continue"` (the default) skips rows whose *function* raised and keeps going; it never skips a row because it @@ -285,8 +338,10 @@ first in the pipeline, the one the synchronous orchestrator would have stopped a failures are logged and noted on it. A node whose failure the asynchronous orchestrator cancelled before it happened is not reported. -After the failure, rows stored before it are kept: `job.run()` commits the writes that -succeeded before it re-raises, and nothing already stored is rewritten. Once the cause is +After the failure, rows stored before it are kept: every batch committed before the failure +stays, and `job.run()` tries once more to commit the writes still pending before it +re-raises (a failed commit's batch stays pending, so this is its one retry; if it fails +again, that is noted on the error), and nothing already stored is rewritten. Once the cause is fixed, run the job again: stored function results are not recomputed, only the missing ones are. An operator with `cache_mode=LOG` recomputes its output on every run, as it always does. A side effect is different: if its delivery completed but the write to its invocation log diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 000000000..b0e5f3d94 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,39 @@ +# Troubleshooting + +## A script finishes its work but the process never exits + +**Symptom.** A Python script that used a Delta store prints all of its output and then hangs, +and has to be killed. On Linux with Python 3.11 or 3.12 the process may instead end with +`terminate called without an active exception` and exit code 134 (SIGABRT). Notebooks are not +affected in the same way, because the kernel keeps running. + +**Cause.** This is a bug in pyarrow, not in your pipeline +([apache/arrow#50482](https://github.com/apache/arrow/issues/50482), +[apache/arrow#54536](https://github.com/apache/arrow/issues/54536)). deltalake's default reader +goes through a filesystem written in Python. After a read returns, Arrow finishes cleaning it up +on a worker thread. If that cleanup runs while the interpreter is shutting down, the worker +cannot take the GIL and never finishes, and Arrow's thread pool waits for it forever. Your data +is not affected: the hang happens after every write has completed. + +**What Orcapod does.** `DeltaTableDatabase` and `DeltaTableSource` read **local** Delta tables +through a native pyarrow filesystem, so they are not affected. Tables on cloud storage +(`s3://`, `gs://`, `az://` and similar) still use deltalake's reader and can still hang at exit. +Reading a Delta table yourself with `deltalake.DeltaTable(...).to_pyarrow_table()` or +`to_pyarrow_dataset()` is affected too, whatever the storage. + +**Workarounds.** + +- Upgrade pyarrow once a release includes the fix + ([apache/arrow#54467](https://github.com/apache/arrow/pull/54467)). +- In a script that reads a cloud Delta store, end it by flushing your outputs and calling + `os._exit(0)`. That skips interpreter shutdown, where the hang happens. It also skips + `atexit` handlers and other cleanup, so close files and flush stores first. +- When reading local Delta tables directly with deltalake, pass a native filesystem. Resolve + `dt.table_uri` first: it is percent-encoded, and newer deltalake versions return a + `file://` URI. + + ```python + uri = dt.table_uri if "://" in dt.table_uri else "file://" + dt.table_uri + fs, root = pyarrow.fs.FileSystem.from_uri(uri) + dataset = dt.to_pyarrow_dataset(filesystem=pyarrow.fs.SubTreeFileSystem(root, fs)) + ``` diff --git a/mkdocs.yml b/mkdocs.yml index 5b4709c30..a33a00059 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -53,3 +53,4 @@ nav: - Types: api/types.md - Reference: - Hashing: reference/hashing.md + - Troubleshooting: troubleshooting.md diff --git a/src/orcapod/channels.py b/src/orcapod/channels.py index 68cfd028d..8111912f4 100644 --- a/src/orcapod/channels.py +++ b/src/orcapod/channels.py @@ -99,6 +99,10 @@ def __init__(self, channel: Channel[T]) -> None: async def receive(self) -> T: if self._channel._drained: raise ChannelClosed() + if self._channel._close_pending and self._channel._queue.empty(): + # Closed while the queue was full: every item is received; the end now. + self._channel._drained = True + raise ChannelClosed() item = await self._channel._queue.get() if isinstance(item, _Sentinel): self._channel._drained = True @@ -143,9 +147,20 @@ async def send(self, item: T) -> None: await self._channel._queue.put(item) async def close(self) -> None: + """Signal that no more items will be sent, without waiting. + + Never blocks (ITL-932): on a failed run every reader is cancelled while + writers close their outputs in ``finally`` blocks, and a close that waited + for room in a full queue would hang the run. When the queue is full, the + close is recorded instead of queued; readers receive every queued item and + then ``ChannelClosed``, exactly as with a queued close. + """ if not self._channel._closed.is_set(): self._channel._closed.set() - await self._channel._queue.put(_CLOSED) + try: + self._channel._queue.put_nowait(_CLOSED) + except asyncio.QueueFull: + self._channel._close_pending = True # --------------------------------------------------------------------------- @@ -166,6 +181,8 @@ class Channel(Generic[T]): _queue: asyncio.Queue[T | _Sentinel] = field(init=False) _closed: asyncio.Event = field(init=False, default_factory=asyncio.Event) _drained: bool = field(init=False, default=False) + # Closed while the queue was full: the close is not queued (ITL-932). + _close_pending: bool = field(init=False, default=False) def __post_init__(self) -> None: self._queue = asyncio.Queue(maxsize=self.buffer_size) @@ -193,16 +210,21 @@ class _BroadcastReader(Generic[T]): multiple downstream consumers can read at their own pace. """ - __slots__ = ("_queue",) + __slots__ = ("_closed", "_queue") def __init__(self, buffer_size: int) -> None: self._queue: asyncio.Queue[T | _Sentinel] = asyncio.Queue(maxsize=buffer_size) + # Set once this reader's stream has ended, or by a close that found its + # queue full (ITL-932): then the end comes once the queue is drained. + self._closed = False async def receive(self) -> T: + if self._closed and self._queue.empty(): + raise ChannelClosed() item = await self._queue.get() if isinstance(item, _Sentinel): - # Re-enqueue so repeated receive() calls also raise - await self._queue.put(item) + # Remember the end so repeated receive() calls also raise. + self._closed = True raise ChannelClosed() return item # type: ignore[return-value] @@ -264,7 +286,15 @@ async def send(self, item: T) -> None: await reader._queue.put(item) async def close(self) -> None: + """Signal every reader that no more items will be sent, without waiting. + + Never blocks on a slow or cancelled reader (ITL-932); see + ``_ChannelWriter.close``. + """ if not self._broadcast._closed: self._broadcast._closed = True for reader in self._broadcast._readers: - await reader._queue.put(_CLOSED) + try: + reader._queue.put_nowait(_CLOSED) + except asyncio.QueueFull: + reader._closed = True diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index 757b43692..b8cbaacfe 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -76,6 +76,15 @@ def set_ignore_schema(self, ignore_schema: tuple[str, ...] | None) -> None: """ self._cache.set_ignore_schema(ignore_schema) + def set_auto_flush(self, on: bool = True) -> None: + """Set whether each stored result is committed (flushed) right away. + + Args: + on: ``True`` (the default) commits after every stored result; + ``False`` leaves the commit to the caller. + """ + self._cache.set_auto_flush(on) + def lookup_cached_data(self, data: DataProtocol) -> DataProtocol | None: """Look up a cached result for ``data`` without triggering computation. diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 3454c8856..398ec851c 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -17,6 +17,7 @@ from __future__ import annotations import asyncio +import contextlib import logging import uuid from abc import abstractmethod @@ -39,7 +40,7 @@ StoreReadError, StoreWriteError, ) -from orcapod.utils.store_errors import keep_store_error +from orcapod.utils.store_errors import commit_best_effort, keep_store_error from orcapod.protocols.core_protocols import ( FunctionPodProtocol, DataFunctionExecutorProtocol, @@ -49,6 +50,7 @@ TagProtocol, TrackerManagerProtocol, ) +from orcapod.databases.utils import as_table_committing from orcapod.protocols.database_protocols import ArrowDatabaseProtocol from orcapod.protocols.observability_protocols import ( DataExecutionLoggerProtocol, @@ -822,6 +824,11 @@ def __init__( # .node_config setter or PipelineJob.apply_node_config(). self._node_config: NodeConfig = NodeConfig() + # Batched commits, set only while PipelineJob.run runs the node (ITL-919). + self._commit_every_rows: int | None = None + self._rows_since_commit = 0 + self._commit_failed = False + self.attach_databases( pipeline_database=pipeline_database, result_database=result_database, @@ -1105,6 +1112,9 @@ def from_descriptor( # FunctionJobNode — orchestrator config node._node_config = NodeConfig() + node._commit_every_rows = None + node._rows_since_commit = 0 + node._commit_failed = False # FunctionJobNode — DB persistence state (wired by attach_databases) node._pipeline_database = None @@ -1193,6 +1203,155 @@ def _require_pipeline_database(self) -> ArrowDatabaseProtocol: ) return self._pipeline_database + # ------------------------------------------------------------------ + # Batched commits (ITL-919) + # ------------------------------------------------------------------ + + @contextlib.contextmanager + def _batched_commits(self, rows_per_commit: int) -> Iterator[None]: + """Commit this node's writes every ``rows_per_commit`` rows inside the block. + + ``PipelineJob.run`` runs each function node inside this block. Its result + caches stop committing after every stored result, and the node commits its + pending writes (ephemeral results, then its result table, then the pipeline + records) every ``rows_per_commit`` rows and at its end. Outside the block + (a node run on its own) every result is committed as it is stored. + + Args: + rows_per_commit: How many processed rows' writes one commit holds. + + Yields: + Nothing; the node commits in batches while the block runs. + """ + pods = [ + pod + for pod in (self._cached_function_pod, self._ephemeral_cached_pod) + if isinstance(pod, CachedFunctionPod) + ] + self._commit_every_rows = rows_per_commit + self._rows_since_commit = 0 + self._commit_failed = False + for pod in pods: + pod.set_auto_flush(False) + try: + yield + finally: + self._commit_every_rows = None + for pod in pods: + pod.set_auto_flush(True) + + def _ephemeral_store_in_use(self) -> ArrowDatabaseProtocol | None: + """Return the ephemeral store this node writes results to, if it does.""" + if self._node_config.is_result_ephemeral and self._ephemeral_cached_pod: + return self.ephemeral_result_store + return None + + def _commit_result_table(self) -> None: + """Commit this node's pending result rows, before its pipeline records. + + Every view of a store shares one set of pending writes, and a flush commits + them table by table in no guaranteed order. Committing the result table on + its own first means a crash between the two commits leaves results without + pipeline records (found in the result cache on the rerun, so nothing is + recomputed), never pipeline records pointing at results that were lost. + Only a store that commits table by table + (``TableCommittingDatabaseProtocol``, the Delta store) needs and offers this; + for another store the pipeline flush that follows commits everything. + """ + pod = self._cached_function_pod + if not isinstance(pod, CachedFunctionPod): + return + result_database = as_table_committing(pod.result_database) + if result_database is not None: + result_database.flush_batch(pod.record_path) + + def _commit_pending_writes(self) -> None: + """Commit this node's pending writes, results first (see ``_batched_commits``). + + Raises: + StoreReadError: If a stored table could not be read; it names this node. + StoreWriteError: If a commit failed, naming this node and the store + (``flush=True``, no row: the batch holds many rows). The failed + writes stay pending, and earlier commits are kept. + """ + pipeline_database = self._require_pipeline_database() + ephemeral = self._ephemeral_store_in_use() + try: + if ephemeral is not None: + try: + ephemeral.flush() + except StoreReadError: + raise + except Exception as exc: + raise StoreWriteError( + ephemeral.base_path, node_label=self.label, flush=True + ) from exc + try: + self._commit_result_table() + pipeline_database.flush() + except StoreReadError: + raise + except Exception as exc: + raise StoreWriteError( + pipeline_database.base_path, node_label=self.label, flush=True + ) from exc + except StoreError as exc: + self._commit_failed = True + exc.add_context(node_label=self.label) + raise + self._rows_since_commit = 0 + + def _row_processed(self) -> None: + """Count a processed row; commit once a batch is full (only inside a job run).""" + if self._commit_every_rows is None: + return + self._rows_since_commit += 1 + if self._rows_since_commit >= self._commit_every_rows: + self._commit_pending_writes() + + def _commit_at_node_end(self) -> None: + """Commit what is still pending when the node finishes (only inside a job run).""" + if self._commit_every_rows is not None: + self._commit_pending_writes() + + def _commit_after_failure(self, run_error: BaseException) -> None: + """Commit what succeeded before a failure, never replacing the run's error. + + The job flushes its store after a failed run, but not a node's ephemeral + store, a separate database, and not table by table: so the node commits its + ephemeral results and then its result table here, best-effort (a failure + is noted on ``run_error``). Skipped when the failure was this node's own + commit: the job's flush is that commit's one retry. + + Args: + run_error: The error the node is failing with. + """ + if self._commit_every_rows is None or self._commit_failed: + return + ephemeral = self._ephemeral_store_in_use() + if ephemeral is not None: + commit_best_effort(run_error, ephemeral.flush) + commit_best_effort(run_error, self._commit_result_table) + + @staticmethod + def _commit_or_end_node( + commit: Any, ctx_obs: Any, node_label: str, node_hash: str + ) -> None: + """Run a commit step; if it fails, end the node for the observer and re-raise. + + Args: + commit: ``_row_processed`` or ``_commit_at_node_end``. + ctx_obs: The node's contextualized observer. + node_label: The node's label, for the observer. + node_hash: The node's content hash, for the observer. + """ + try: + commit() + except StoreError as exc: + with keep_store_error(exc, "Observer on_node_end"): + ctx_obs.on_node_end(node_label, node_hash) + raise + # ------------------------------------------------------------------ # as_node — return the lightweight FunctionNode equivalent # ------------------------------------------------------------------ @@ -1291,21 +1450,27 @@ def execute( self.get_cached_results(base_entry_ids=base_entry_ids) output: list[tuple[TagProtocol, DataProtocol]] = [] - for tag, data, base_entry_id in upstream_entries: - ctx_obs.on_data_start(node_label, tag, data) - - if base_entry_id in self._cached_output_datas and not isinstance( - self._cached_output_datas[base_entry_id][1], EmptyData - ): - tag_out, result = self._cached_output_datas[base_entry_id] - ctx_obs.on_data_end(node_label, tag, data, result, cached=True) - if result is not None: - output.append((tag_out, result)) - else: + try: + for tag, data, base_entry_id in upstream_entries: + ctx_obs.on_data_start(node_label, tag, data) + + if base_entry_id in self._cached_output_datas and not isinstance( + self._cached_output_datas[base_entry_id][1], EmptyData + ): + tag_out, result = self._cached_output_datas[base_entry_id] + ctx_obs.on_data_end(node_label, tag, data, result, cached=True) + if result is not None: + output.append((tag_out, result)) + continue + pkt_logger = ctx_obs.create_data_logger(tag, data) try: tag_out, result = self._process_data_internal( - tag, data, logger=pkt_logger, run_id=run_id + tag, + data, + logger=pkt_logger, + run_id=run_id, + base_entry_id=base_entry_id, ) except StoreError as exc: # A failed store read or write fails the run whatever the error @@ -1328,10 +1493,21 @@ def execute( if error_policy == "fail_fast": ctx_obs.on_node_end(node_label, node_hash) raise - else: - ctx_obs.on_data_end(node_label, tag, data, result, cached=False) - if result is not None: - output.append((tag_out, result)) + continue + + ctx_obs.on_data_end(node_label, tag, data, result, cached=False) + if result is not None: + output.append((tag_out, result)) + self._commit_or_end_node( + self._row_processed, ctx_obs, node_label, node_hash + ) + + self._commit_or_end_node( + self._commit_at_node_end, ctx_obs, node_label, node_hash + ) + except Exception as run_error: + self._commit_after_failure(run_error) + raise ctx_obs.on_node_end(node_label, node_hash) # Mark this node as freshly computed so subsequent iter_data() calls @@ -1346,6 +1522,7 @@ def _process_data_internal( *, logger: DataExecutionLoggerProtocol | None = None, run_id: str | None = None, + base_entry_id: bytes | None = None, ) -> tuple[TagProtocol, DataProtocol | None]: """Core compute + persist + cache. @@ -1359,6 +1536,8 @@ def _process_data_internal( logger: Optional data execution logger. run_id: Optional pipeline run identifier threaded through to ``InvocationContext.pipeline_run_id``. + base_entry_id: The row's ``compute_base_entry_id``, when the caller + already has it; computed here otherwise. When ``node_config.is_result_ephemeral=True``: - Uses ``_ephemeral_cached_pod`` for both compute and storage. @@ -1383,11 +1562,12 @@ def _process_data_internal( # add_pipeline_record is intentionally skipped for EmptyData inputs because # EmptyData.as_table() raises EmptyDataAccessError (no payload to read source # columns from). Pipeline DB write for this path is deferred. + if base_entry_id is None: + base_entry_id = self.compute_base_entry_id(tag, data) if isinstance(data, EmptyData): if self._cached_function_pod is not None: cached = self._cached_function_pod.lookup_cached_data(data) if cached is not None: - base_entry_id = self.compute_base_entry_id(tag, data) self._cached_output_datas[base_entry_id] = (tag, cached) self._cached_output_table = None self._cached_content_hash_column = None @@ -1422,9 +1602,10 @@ def _process_data_internal( ) ) if self._pipeline_database is not None: - self.add_pipeline_record( + self._add_pipeline_record( tag, data, + base_entry_id, data_record_id=output_data.datagram_uuid, computed=result_computed, is_ephemeral=True, @@ -1440,9 +1621,10 @@ def _process_data_internal( self._cached_function_pod.RESULT_COMPUTED_FLAG, False ) ) - self.add_pipeline_record( + self._add_pipeline_record( tag, data, + base_entry_id, data_record_id=output_data.datagram_uuid, computed=result_computed, output_data_hash=output_data.content_hash(), @@ -1453,7 +1635,6 @@ def _process_data_internal( ) # Store by base_entry_id (stable across recomputation cycles) and invalidate caches - base_entry_id = self.compute_base_entry_id(tag, data) self._cached_output_datas[base_entry_id] = (tag_out, output_data) self._cached_output_table = None self._cached_content_hash_column = None @@ -1509,22 +1690,24 @@ async def _async_process_data_internal( data: DataProtocol, *, logger: DataExecutionLoggerProtocol | None = None, + base_entry_id: bytes | None = None, ) -> tuple[TagProtocol, DataProtocol | None]: """Async counterpart of ``_process_data_internal``. Mirrors the sync ephemeral/persistent branch logic using async variants of ``process_data``. See ``_process_data_internal`` for full behaviour - documentation. + documentation, including ``base_entry_id``. Returns: A ``(tag, output_data)`` 2-tuple. """ # Guard: same logic as sync _process_data_internal — see inline comment there. + if base_entry_id is None: + base_entry_id = self.compute_base_entry_id(tag, data) if isinstance(data, EmptyData): if self._cached_function_pod is not None: cached = self._cached_function_pod.lookup_cached_data(data) if cached is not None: - base_entry_id = self.compute_base_entry_id(tag, data) self._cached_output_datas[base_entry_id] = (tag, cached) self._cached_output_table = None self._cached_content_hash_column = None @@ -1559,9 +1742,10 @@ async def _async_process_data_internal( ) ) if self._pipeline_database is not None: - self.add_pipeline_record( + self._add_pipeline_record( tag, data, + base_entry_id, data_record_id=output_data.datagram_uuid, computed=result_computed, is_ephemeral=True, @@ -1577,9 +1761,10 @@ async def _async_process_data_internal( self._cached_function_pod.RESULT_COMPUTED_FLAG, False ) ) - self.add_pipeline_record( + self._add_pipeline_record( tag, data, + base_entry_id, data_record_id=output_data.datagram_uuid, computed=result_computed, output_data_hash=output_data.content_hash(), @@ -1590,7 +1775,6 @@ async def _async_process_data_internal( ) # Store by base_entry_id (stable across recomputation cycles) and invalidate caches - base_entry_id = self.compute_base_entry_id(tag, data) self._cached_output_datas[base_entry_id] = (tag_out, output_data) self._cached_output_table = None self._cached_content_hash_column = None @@ -1724,9 +1908,39 @@ def add_pipeline_record( so that downstream ``EmptyData`` flow-through can locate its cached result keyed by the same hash (= the downstream's ``INPUT_DATA_HASH_COL``). """ + self._add_pipeline_record( + tag, + input_data, + self.compute_base_entry_id(tag, input_data), + data_record_id=data_record_id, + computed=computed, + is_ephemeral=is_ephemeral, + output_data_hash=output_data_hash, + ) + + def _add_pipeline_record( + self, + tag: TagProtocol, + input_data: DataProtocol, + base_entry_id: bytes, + data_record_id: uuid.UUID, + computed: bool, + is_ephemeral: bool = False, + output_data_hash: ContentHash | None = None, + ) -> None: + """``add_pipeline_record`` with the row's base entry id already computed. + + Args: + tag: The tag associated with the input data. + input_data: The input data that was processed. + base_entry_id: ``compute_base_entry_id(tag, input_data)``. + data_record_id: UUID of the result record in the result database. + computed: Whether the result was freshly computed. + is_ephemeral: Whether the result is stored in the ephemeral store. + output_data_hash: Content hash of the output, if any. + """ pipeline_database = self._require_pipeline_database() self._ensure_pdb_schema() - base_entry_id = self.compute_base_entry_id(tag, input_data) # Determine the next recomputation index by querying all existing rows # for this base_entry_id. No await is used here, so within a single-threaded @@ -2417,14 +2631,22 @@ async def async_execute( ) # Local dict: correlation_key → (original_tag, original_input_data) - input_store: dict[bytes, tuple[TagProtocol, DataProtocol]] = {} + input_store: dict[bytes, tuple[TagProtocol, DataProtocol, bytes]] = {} async def route_inputs() -> None: """Stage 1: send cache hits to output; stamp misses for computation.""" try: async for tag, data in input_channel: base_entry_id = self.compute_base_entry_id(tag, data) - if base_entry_id in cached_by_base_entry_id: + # A cached EmptyData is a result that is gone (an + # ephemeral result from an earlier run): compute the row + # again, as the sync path does (ITL-923). + if base_entry_id in cached_by_base_entry_id and not ( + isinstance( + cached_by_base_entry_id[base_entry_id][1], + EmptyData, + ) + ): cached_tag, cached_data = cached_by_base_entry_id[ base_entry_id ] @@ -2435,7 +2657,11 @@ async def route_inputs() -> None: await output.send((cached_tag, cached_data)) else: correlation_key = uuid.uuid4().bytes - input_store[correlation_key] = (tag, data) + input_store[correlation_key] = ( + tag, + data, + base_entry_id, + ) stamped_tag = tag.with_meta_columns( **{_TAG_NODE_INPUT_REF: correlation_key} ) @@ -2447,16 +2673,19 @@ async def record_and_forward() -> None: """Stage 3: record pipeline entry, strip key, emit to output.""" async for output_tag, output_data in result_channel.reader: correlation_key = output_tag.get_meta_value(_TAG_NODE_INPUT_REF) - original_tag, input_data = input_store.pop(correlation_key) + original_tag, input_data, base_entry_id = input_store.pop( + cast(bytes, correlation_key) + ) result_computed = bool( output_data.get_meta_value( execution_pod.RESULT_COMPUTED_FLAG, False ) ) try: - self.add_pipeline_record( + self._add_pipeline_record( original_tag, input_data, + base_entry_id, data_record_id=output_data.datagram_uuid, computed=result_computed, is_ephemeral=is_ephemeral, @@ -2470,9 +2699,6 @@ async def record_and_forward() -> None: ) raise # Update in-memory cache so iter_data() sees the result. - base_entry_id = self.compute_base_entry_id( - original_tag, input_data - ) clean_tag = output_tag.drop_meta_columns( _TAG_NODE_INPUT_REF, ignore_missing=True ) @@ -2482,6 +2708,9 @@ async def record_and_forward() -> None: ) self._cached_output_table = None self._cached_content_hash_column = None + # Commits every commit_every_rows rows inside a job (ITL-919); + # no await in between, so no other row interleaves. + self._row_processed() await output.send((clean_tag, output_data)) # Capture outer self so _NodeLabelObserver methods can @@ -2541,10 +2770,7 @@ def on_data_end( if correlation_key is not None: original = input_store.pop(correlation_key, None) if original is not None: - orig_tag, orig_data = original - base_entry_id = fn_node.compute_base_entry_id( - orig_tag, orig_data - ) + _, _, base_entry_id = original fn_node._cached_output_datas[base_entry_id] = ( clean, None, @@ -2588,6 +2814,8 @@ def create_data_logger(self, tag: Any, data: Any) -> Any: ) ) tg.create_task(record_and_forward()) + # Commit the last rows inside a job (ITL-919); a no-op otherwise. + self._commit_at_node_end() else: # No-DB path: delegate dispatch entirely to the pod. # The pod's async_execute() closes `output` in its own finally. @@ -2596,5 +2824,8 @@ def create_data_logger(self, tag: Any, data: Any) -> Any: ) ctx_obs.on_node_end(node_label, node_hash) + except Exception as run_error: + self._commit_after_failure(run_error) + raise finally: await output.close() diff --git a/src/orcapod/core/sources/delta_table_source.py b/src/orcapod/core/sources/delta_table_source.py index bfc2b0520..86a187932 100644 --- a/src/orcapod/core/sources/delta_table_source.py +++ b/src/orcapod/core/sources/delta_table_source.py @@ -6,6 +6,7 @@ from orcapod.core.sources.base import RootSource from orcapod.core.sources.stream_builder import SourceStreamBuilder +from orcapod.databases.storage_utils import delta_to_pyarrow_dataset from orcapod.types import PathLike from orcapod.utils.lazy_module import LazyModule from orcapod.utils.schema_utils import _normalize_column_list @@ -49,8 +50,8 @@ def __init__( except deltalake.exceptions.TableNotFoundError: raise ValueError(f"Delta table not found at {self._delta_table_path}") - table: pa.Table = self._delta_table.to_pyarrow_dataset( - as_large_types=True + table: pa.Table = delta_to_pyarrow_dataset( + self._delta_table, as_large_types=True ).to_table() # The Delta log stores field-level nullable flags; to_pyarrow_dataset() # preserves them exactly — no inference needed. diff --git a/src/orcapod/databases/_row_index.py b/src/orcapod/databases/_row_index.py new file mode 100644 index 000000000..b9da1f6e8 --- /dev/null +++ b/src/orcapod/databases/_row_index.py @@ -0,0 +1,170 @@ +"""Hash indexes over a table's rows, shared by the Arrow databases (ITL-919). + +A database that answers per-row lookups by filtering its whole stored table makes a run +quadratic in its rows. These helpers keep, per stored table, lazily built hash indexes from +a column value to the positions of the rows holding it, extended in place when rows are +appended, and keep a growing table at a logarithmic number of record batches. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any + +from orcapod.utils.lazy_module import LazyModule + +if TYPE_CHECKING: + import pyarrow as pa + import pyarrow.compute as pc +else: + pa = LazyModule("pyarrow") + pc = LazyModule("pyarrow.compute") + + +@dataclass +class KeyIndex: + """Hash indexes over the rows of one stored table. + + Each column index maps a value to the positions, in ascending order, of the rows of + ``table`` holding it. The indexes describe ``table`` only: they are valid while + ``table`` is the table they were built for, and are rebuilt for any other table. + + Attributes: + table: The indexed table. + columns: Column name to its index, for the columns indexed so far. + """ + + table: "pa.Table" + columns: dict[str, dict[Any, list[int]]] = field(default_factory=dict) + + def column(self, name: str) -> dict[Any, list[int]]: + """Return the index of column ``name``, building it on first use. + + Args: + name: A column of ``table``. + + Returns: + The value-to-positions index of the column. + """ + positions = self.columns.get(name) + if positions is None: + positions = {} + for position, value in enumerate(self.table.column(name).to_pylist()): + positions.setdefault(value, []).append(position) + self.columns[name] = positions + return positions + + def append(self, rows: "pa.Table", appended: "pa.Table") -> None: + """Extend the indexes for ``rows`` appended after the indexed table's rows. + + Args: + rows: The rows that were appended, with every indexed column. + appended: The new table: the indexed table followed by ``rows``. It + becomes the indexed table. + """ + offset = self.table.num_rows + for name, positions in self.columns.items(): + for position, value in enumerate(rows.column(name).to_pylist(), offset): + positions.setdefault(value, []).append(position) + self.table = appended + + +def is_indexed_lookup(arrow_type: "pa.DataType", value: Any) -> bool: + """Whether an equality lookup of ``value`` in a column can use a hash index. + + Only binary and string columns queried with a value of the matching Python type + qualify: for those, Python equality of the values is exactly Arrow's equality. Every + other lookup filters the table, as Arrow defines that comparison. + + Args: + arrow_type: The column's Arrow type. + value: The value looked up. + + Returns: + Whether the lookup can be answered from a hash index. + """ + if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): + return type(value) is bytes + if pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type): + return type(value) is str + return False + + +def compact_tail(table: "pa.Table") -> "pa.Table": + """Merge the table's trailing record batches while an earlier one is not larger. + + Called after each append, this keeps batch sizes strictly decreasing, like a binary + counter: a table of N rows has at most log2(N) + 1 batches, and each row is copied + O(log N) times over the table's life. Every Arrow operation on a table costs time per + batch, so without it a table grown one append at a time makes every later operation + cost time proportional to the appends so far. + + Args: + table: The table just appended to. + + Returns: + The same rows, in the same order, with the trailing batches merged. + """ + batches = [batch for batch in table.to_batches() if batch.num_rows > 0] + while len(batches) >= 2 and batches[-2].num_rows <= batches[-1].num_rows: + merged = pa.Table.from_batches(batches[-2:]).combine_chunks() + batches[-2:] = merged.to_batches() + return pa.Table.from_batches(batches, schema=table.schema) + + +def rows_at(table: "pa.Table", positions: Sequence[int]) -> "pa.Table": + """Return the rows of ``table`` at ``positions``, in the given order. + + Uses zero-copy one-row slices: ``take`` on a multi-chunk table concatenates every + chunk first, copying all rows of the table. + + Args: + table: The table to select from. + positions: Row positions. + + Returns: + The selected rows, with ``table``'s schema. + """ + rows = [table.slice(position, 1) for position in positions] + return pa.concat_tables(rows) if rows else table.slice(0, 0) + + +def matching_rows( + key_index: KeyIndex, + pairs: Sequence[tuple[str, Any]], + expression: "pc.Expression", +) -> "pa.Table": + """Return the rows of ``key_index.table`` where every column equals its value. + + The pairs that can use a hash index select the rows at the intersection of their + indexes' positions. When every pair can, those rows are the result; when only some + can, just those rows are filtered by ``expression``; when none can, the whole table + is. Rows come back in table order. + + Args: + key_index: The indexes of the table to search. + pairs: ``(column, value)`` equality conditions, all to hold. + expression: The same conditions as an Arrow expression. + + Returns: + The matching rows (possibly none). + """ + table = key_index.table + indexed = [ + (column, value) + for column, value in pairs + if column in table.column_names + and is_indexed_lookup(table.schema.field(column).type, value) + ] + if not indexed: + return table.filter(expression) + + matches: set[int] | None = None + for column, value in indexed: + found = key_index.column(column).get(value, []) + matches = set(found) if matches is None else matches.intersection(found) + candidates = rows_at(table, sorted(matches or ())) + if len(indexed) == len(pairs): + return candidates + return candidates.filter(expression) diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index 8c34c1f04..dba0fc3e8 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -3,12 +3,24 @@ import logging from collections import defaultdict from collections.abc import Collection, Mapping +from dataclasses import dataclass from pathlib import Path from typing import TYPE_CHECKING, Any, Literal, cast +from orcapod.databases._row_index import ( + KeyIndex, + compact_tail, + is_indexed_lookup, + matching_rows, + rows_at, +) from orcapod.databases.utils import coerce_record_id from orcapod.errors import InputValidationError, StoreReadError -from orcapod.databases.storage_utils import is_cloud_uri, parse_base_path +from orcapod.databases.storage_utils import ( + delta_to_pyarrow_dataset, + is_cloud_uri, + parse_base_path, +) from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -17,6 +29,7 @@ import polars as pl import pyarrow as pa import pyarrow.compute as pc + import pyarrow.dataset as pa_ds from upath import UPath else: deltalake = LazyModule("deltalake") @@ -272,14 +285,29 @@ def _store_error_to_raise(errors: list[Exception]) -> Exception | None: return None -class _DeltaTableReadError(Exception): - """A Delta table's data could not be read; the backend error is the ``__cause__``. +@dataclass +class _CommittedSnapshot: + """What one version of a stored Delta table holds, kept in memory (ITL-919). - Raised by ``DeltaTableDatabase._read_delta_table`` so that its callers, which - know the table's path, can tell a failed read from a bad filter and raise - ``StoreReadError``. + Per-row lookups are answered from this snapshot instead of re-reading the table. + Each part is read on first use and extended in place after this store's own + commits, so its cost follows the rows written, not the rows stored. + + Attributes: + version: The table version described. + ids: The stored record ids (never null), or ``None`` until first needed. + rows: Indexes over every stored row, read exactly as ``get_all_records`` + reads them, or ``None`` until first needed. + files: The data files ``rows`` was read from (set together with ``rows``). + dataset_schema: The schema of the dataset ``rows`` was read from. """ + version: int + ids: set[bytes] | None = None + rows: KeyIndex | None = None + files: frozenset[str] = frozenset() + dataset_schema: "pa.Schema | None" = None + class DeltaTableDatabase: """ @@ -311,6 +339,9 @@ def __init__( _scoped_path: tuple[str, ...] = (), _shared_pending_batches: "dict[str, pa.Table] | None" = None, _shared_pending_record_ids: "defaultdict[str, set[bytes]] | None" = None, + _shared_delta_table_cache: "dict[str, deltalake.DeltaTable] | None" = None, + _shared_snapshots: "dict[str, _CommittedSnapshot] | None" = None, + _shared_pending_indexes: "dict[str, KeyIndex] | None" = None, ): self._root_uri, self._storage_options = parse_base_path( base_path, storage_options @@ -335,9 +366,16 @@ def __init__( ) # For cloud paths: create_base_path is silently ignored (no directory needed). - self._delta_table_cache: dict[str, deltalake.DeltaTable] = {} - # Shared pending state — when constructed via at(), these dicts are shared - # with the root instance so that root.flush() can flush all scoped writes. + # Shared state — when constructed via at(), these dicts are shared with the + # root instance. Any handle's flush() writes every pending key of the family + # and refreshes the loaded table and existing-ids entries for it, so these + # caches must be family-wide too: a per-handle cache would keep serving the + # snapshot a sibling loaded before the flush (ITL-864). All of them are + # touched only from the thread that drives the pipeline (the async + # orchestrator runs only user functions in worker threads). + self._delta_table_cache: dict[str, deltalake.DeltaTable] = ( + _shared_delta_table_cache if _shared_delta_table_cache is not None else {} + ) if _shared_pending_batches is not None: self._pending_batches = _shared_pending_batches else: @@ -346,8 +384,15 @@ def __init__( self._pending_record_ids = _shared_pending_record_ids else: self._pending_record_ids: dict[str, set[bytes]] = defaultdict(set) - self._existing_ids_cache: dict[str, set[str]] = defaultdict(set) - self._cache_dirty: dict[str, bool] = defaultdict(lambda: True) + # The committed rows of each loaded table version (ITL-919), shared like the + # loaded tables: a snapshot describes the cached table's version only. + self._snapshots: dict[str, _CommittedSnapshot] = ( + _shared_snapshots if _shared_snapshots is not None else {} + ) + # Indexes over each pending batch, valid while it is the key's batch. + self._pending_indexes: dict[str, KeyIndex] = ( + _shared_pending_indexes if _shared_pending_indexes is not None else {} + ) def _clear_pending(self): """Clear all pending state.""" @@ -544,79 +589,113 @@ def _handle_record_id_column( f"Record ID column '{self.RECORD_ID_COLUMN}' not found in the table and cannot be renamed." ) - def _create_record_id_filter(self, record_id: str | bytes) -> list: - """ - Create a proper filter expression for Delta Lake. + def _snapshot( + self, record_key: str, delta_table: deltalake.DeltaTable + ) -> _CommittedSnapshot: + """Return the snapshot of ``delta_table``'s version, starting an empty one if needed. Args: - record_id: The entry ID to filter by + record_key: The table's absolute record key. + delta_table: The loaded table the snapshot must describe. Returns: - List containing the filter expression for Delta Lake + The snapshot for the table's version; its parts are read on first use. """ - return [(self.RECORD_ID_COLUMN, "=", coerce_record_id(record_id))] + version = delta_table.version() + snapshot = self._snapshots.get(record_key) + if snapshot is None or snapshot.version != version: + snapshot = _CommittedSnapshot(version) + self._snapshots[record_key] = snapshot + return snapshot - def _create_record_ids_filter(self, record_ids: list[bytes]) -> list: - """ - Create a proper filter expression for multiple entry IDs. + @classmethod + def _scan_ids(cls, delta_table: deltalake.DeltaTable) -> set[bytes]: + """Read the record ids stored in ``delta_table`` (nulls left out). Args: - record_ids: List of entry IDs to filter by + delta_table: The loaded table. Returns: - List containing the filter expression for Delta Lake + The stored record ids. + + Raises: + Exception: The backend's error when the table's data cannot be read. """ - return [(self.RECORD_ID_COLUMN, "in", record_ids)] + stored = _stored_arrow_schema(delta_table) + if cls.RECORD_ID_COLUMN not in stored.names: + logger.warning(f"Delta table missing {cls.RECORD_ID_COLUMN} column") + return set() + # Read through a one-column schema: the other columns' types are never + # converted, so a logical type stored in a list cannot break the id read. + id_field = stored.field(cls.RECORD_ID_COLUMN) + id_schema = pa.schema( + [id_field.with_type(arrow_utils.normalize_to_large_types(id_field.type))] + ) + dataset = delta_to_pyarrow_dataset(delta_table, schema=id_schema) + ids = dataset.to_table()[cls.RECORD_ID_COLUMN] + return {record_id for record_id in ids.to_pylist() if record_id is not None} - def _refresh_existing_ids_cache(self, record_path: tuple[str, ...]) -> None: - """Refresh the cache of existing IDs in the Delta table.""" - record_key = self._get_record_key(record_path) + @staticmethod + def _scan_rows( + delta_table: deltalake.DeltaTable, + ) -> tuple[pa.Table, frozenset[str], pa.Schema]: + """Read every row of ``delta_table`` as ``get_all_records`` reads it. - delta_table = self._get_delta_table(record_path) + Args: + delta_table: The loaded table. - if delta_table is None: - self._existing_ids_cache[record_key] = set() - self._cache_dirty[record_key] = False - return + Returns: + The rows (view types normalized: PyArrow has no comparison kernels for + ``string_view``, ENG-601), the data + files they were read from and the dataset's schema. - try: - # Get all existing IDs from Delta table using standard RECORD_ID_COLUMN - # TODO: replace this with more targetted loading of only the target column and in batches - arrow_table = delta_table.to_pyarrow_table() - except Exception as e: - # Leave the cache dirty so the next call reads the table again. - raise self._read_failed(record_path) from e + Raises: + Exception: The backend's error when the table's data cannot be read. + """ + # deltalake builds a FileSystemDataset; its data files identify a version. + dataset = cast( + "pa_ds.FileSystemDataset", + delta_to_pyarrow_dataset(delta_table, as_large_types=True), + ) + table = arrow_utils.normalize_table_view_types(dataset.to_table()) + return table.combine_chunks(), frozenset(dataset.files), dataset.schema - if arrow_table.num_rows == 0: - self._existing_ids_cache[record_key] = set() - elif self.RECORD_ID_COLUMN not in arrow_table.column_names: - # TODO: replace this with proper checking of the table schema first! - logger.warning(f"Delta table missing {self.RECORD_ID_COLUMN} column") - self._existing_ids_cache[record_key] = set() - else: - existing_ids = cast( - set[str], set(arrow_table[self.RECORD_ID_COLUMN].to_pylist()) - ) - self._existing_ids_cache[record_key] = existing_ids + def _stored_ids(self, record_path: tuple[str, ...]) -> set[bytes]: + """Return the record ids stored at ``record_path`` (in the loaded version). - self._cache_dirty[record_key] = False - logger.debug( - f"Refreshed existing IDs cache: {len(self._existing_ids_cache)} IDs" - ) + Raises: + StoreReadError: If the table exists but its ids could not be read. + """ + delta_table = self._get_delta_table(record_path) + if delta_table is None: + return set() + snapshot = self._snapshot(self._get_record_key(record_path), delta_table) + if snapshot.ids is None: + try: + snapshot.ids = self._scan_ids(delta_table) + except Exception as e: + raise self._read_failed(record_path) from e + return snapshot.ids - def _get_existing_ids(self, record_path: tuple[str, ...]) -> set[str]: - """Get the set of existing IDs in the Delta table, using cache when possible.""" - record_key = self._get_record_key(record_path) - if ( - self._cache_dirty.get(record_key) - or record_key not in self._delta_table_cache - ): - self._refresh_existing_ids_cache(record_path) - return self._existing_ids_cache.get(record_key) or set() + def _stored_rows(self, record_path: tuple[str, ...]) -> KeyIndex | None: + """Return the indexed rows stored at ``record_path``, or ``None`` if no table. - def _invalidate_cache(self, record_path: tuple[str, ...]) -> None: - """Mark the existing IDs cache as dirty.""" - self._cache_dirty[self._get_record_key(record_path)] = True + Raises: + StoreReadError: If the table exists but could not be read. + """ + delta_table = self._get_delta_table(record_path) + if delta_table is None: + return None + snapshot = self._snapshot(self._get_record_key(record_path), delta_table) + if snapshot.rows is None: + try: + table, files, schema = self._scan_rows(delta_table) + except Exception as e: + raise self._read_failed(record_path) from e + snapshot.rows = KeyIndex(table) + snapshot.files = files + snapshot.dataset_schema = schema + return snapshot.rows def add_record( self, @@ -706,9 +785,15 @@ def add_records( # Step 5: Add to pending batch (no overwrite logic needed) self._add_to_pending_batch(record_path, schema_compatible_table) - # Step 6: Auto-flush if needed - if flush or self._should_auto_flush(record_path): + # Step 6: Flush if asked to; otherwise commit this table alone once its + # pending batch reaches batch_size. Only the full table: committing every + # pending table here would commit another table's rows, such as pipeline + # records whose results are still pending, out of their writer's order + # (ITL-919). + if flush: self.flush() + elif self._should_auto_flush(record_path): + self.flush_batch(record_path) def _deduplicate_within_table(self, table: pa.Table) -> pa.Table: # TODO: consider erroring out if duplicates are found @@ -747,7 +832,7 @@ def _filter_existing_records( existing_in_pending = input_ids.intersection( self._pending_record_ids[record_key] ) - existing_in_delta = input_ids.intersection(self._get_existing_ids(record_path)) + existing_in_delta = input_ids.intersection(self._stored_ids(record_path)) all_existing = existing_in_pending.union(existing_in_delta) if not all_existing: @@ -777,7 +862,7 @@ def _check_all_conflicts( ) # Check conflicts with Delta table - existing_ids = self._get_existing_ids(record_path) + existing_ids = self._stored_ids(record_path) delta_conflicts = input_ids.intersection(existing_ids) if delta_conflicts: raise ValueError( @@ -871,9 +956,15 @@ def _add_to_pending_batch(self, record_path: tuple[str, ...], table: pa.Table): if pending_batch is None: self._pending_batches[record_key] = table else: - self._pending_batches[record_key] = pa.concat_tables([pending_batch, table]) - - pending_ids = cast(list[str], table[self.RECORD_ID_COLUMN].to_pylist()) + # Kept at O(log n) record batches, and its indexes extended in place, so a + # lookup's cost does not grow with the rows pending (ITL-919). + grown = compact_tail(pa.concat_tables([pending_batch, table])) + self._pending_batches[record_key] = grown + index = self._pending_indexes.get(record_key) + if index is not None and index.table is pending_batch: + index.append(table, grown) + + pending_ids = cast(list[bytes], table[self.RECORD_ID_COLUMN].to_pylist()) self._pending_record_ids[record_key].update(pending_ids) def _should_auto_flush(self, record_path: tuple[str, ...]) -> bool: @@ -884,6 +975,130 @@ def _should_auto_flush(self, record_path: tuple[str, ...]) -> bool: and self._pending_batches[record_key].num_rows >= self.batch_size ) + def _pending_rows(self, record_path: tuple[str, ...]) -> KeyIndex | None: + """Return the indexed pending rows at ``record_path``, or ``None`` if none. + + The indexes are built on first use and describe the current pending batch + only (``_add_to_pending_batch`` extends them; a flush or schema change + replaces the batch, and with it the indexes). + """ + record_key = self._get_record_key(record_path) + pending = self._pending_batches.get(record_key) + if pending is None or pending.num_rows == 0: + return None + index = self._pending_indexes.get(record_key) + if index is None or index.table is not pending: + index = KeyIndex(pending) + self._pending_indexes[record_key] = index + return index + + def _without_stored_ids( + self, record_path: tuple[str, ...], pending: pa.Table + ) -> pa.Table | None: + """Return the pending rows a flush would insert: those whose id is not stored. + + A flush inserts only rows whose record id is not stored yet (the merge is + ``when_not_matched_insert_all``), so a pending row with a stored id is + never returned by a read: the stored row is. The stored ids are read even + when ``pending`` is empty, so a read with rows pending raises + ``StoreReadError`` whenever the stored ids cannot be read. + + Args: + record_path: Path components relative to this database view. + pending: Pending rows of the table at ``record_path``. + + Returns: + The rows whose record id is not stored, or ``None`` if there are none. + """ + stored_ids = self._stored_ids(record_path) + if stored_ids and pending.num_rows > 0: + # The mask is built from the pending ids against the cached set, so its + # cost follows the pending rows, not the stored table. A null id is + # never matched by the merge (``null = null`` is not true), so its row + # is kept. + keep = [ + record_id is None or record_id not in stored_ids + for record_id in pending[self.RECORD_ID_COLUMN].to_pylist() + ] + if not all(keep): + pending = pending.filter(pa.array(keep, type=pa.bool_())) + return pending if pending.num_rows > 0 else None + + def _pending_not_stored(self, record_path: tuple[str, ...]) -> pa.Table | None: + """Return every pending row a flush would insert at ``record_path``, or ``None``. + + Args: + record_path: Path components relative to this database view. + + Returns: + The pending rows whose record id is not stored, or ``None``. + """ + pending = self._pending_rows(record_path) + if pending is None: + return None + return self._without_stored_ids(record_path, pending.table) + + def _combine_stored_and_pending( + self, + stored: pa.Table | None, + pending: pa.Table | None, + record_id_column: str | None, + ) -> pa.Table | None: + """Combine stored rows with pending rows into one read result. + + Combined rows have the schema a flush would give them: both parts take + their large layouts (a stored table reads back large), a column missing + on one side is null-filled, a ``null``-typed column takes the other + side's type, nullability is merged (nullable if either side is), and + struct children are merged, as Delta's merge evolves the stored schema. + A column whose types still differ raises ``pyarrow.ArrowTypeError``; + nothing is widened. A single part is returned as it is. + + Args: + stored: Rows read from the Delta table, or ``None``. + pending: Pending rows that are not stored (see ``_pending_not_stored``), + already filtered by the read's condition, or ``None``. + record_id_column: Name to give the record id column, or ``None`` to + drop it. + + Returns: + The combined rows, or ``None`` if there are none. + """ + parts = [t for t in (stored, pending) if t is not None and t.num_rows > 0] + if not parts: + return None + if len(parts) == 1: + table = parts[0] + else: + table = pa.concat_tables( + [self._large_layout(t) for t in parts], promote_options="default" + ) + return self._handle_record_id_column(table, record_id_column) + + @staticmethod + def _large_layout(table: pa.Table) -> pa.Table: + """Return ``table`` with every column in its large layout. + + Only layouts change (``string`` to ``large_string``, ``list`` to + ``large_list``, recursively); ``null`` stays ``null``, and names, + nullability and metadata are kept. + + Args: + table: The table to normalize. + + Returns: + The table with large-normalized column types. + """ + schema = table.schema + for index, field in enumerate(table.schema): + schema = schema.set( + index, + field.with_type( + arrow_utils.normalize_to_large_types(field.type, keep_null=True) + ), + ) + return table if schema.equals(table.schema) else table.cast(schema) + def get_all_records( self, record_path: tuple[str, ...], @@ -893,45 +1108,33 @@ def get_all_records( """ Get all records from both pending batch and Delta table. + Every read method returns the rows a flush would leave: the stored rows + plus the pending rows whose record id is not stored yet, as seen through + any handle of this ``at()`` family. + Returns: Combined Arrow table with all records, or None if no records exist Raises: StoreReadError: If the Delta table exists but could not be read. """ - record_key = self._get_record_key(record_path) - tables_to_combine = [] # Add Delta table data if (delta_table := self._get_delta_table(record_path)) is not None: try: - delta_table_data = delta_table.to_pyarrow_dataset( - as_large_types=True + delta_table_data = delta_to_pyarrow_dataset( + delta_table, as_large_types=True ).to_table() except Exception as e: raise self._read_failed(record_path) from e if delta_table_data.num_rows > 0: tables_to_combine.append(delta_table_data) - # Add pending batch data - if ( - retrieve_pending - and (pending_batch := self._pending_batches.get(record_key)) is not None - ): - if pending_batch.num_rows > 0: - tables_to_combine.append(pending_batch) - - if not tables_to_combine: - return None - - if len(tables_to_combine) == 1: - table_to_return = tables_to_combine[0] - else: - table_to_return = pa.concat_tables(tables_to_combine) - - # Handle record_id_column if specified - return self._handle_record_id_column(table_to_return, record_id_column) + # Add the pending rows a flush would insert + pending = self._pending_not_stored(record_path) if retrieve_pending else None + stored = pa.concat_tables(tables_to_combine) if tables_to_combine else None + return self._combine_stored_and_pending(stored, pending, record_id_column) def get_records_with_column_value( self, @@ -942,14 +1145,12 @@ def get_records_with_column_value( ) -> "pa.Table | None": if flush: self.flush_batch(record_path) - # check if record_id is found in pending batches - record_key = self._get_record_key(record_path) - pending_batch = self._pending_batches.get(record_key) + pair_list: list[tuple[str, Any]] if isinstance(column_values, Mapping): # Convert Mapping to list of tuples - pair_list = list(column_values.items()) - elif isinstance(column_values, Collection): + pair_list = list(cast(Mapping[str, Any], column_values).items()) + else: # Ensure it's a list of tuples pair_list = cast(list[tuple[str, Any]], list(column_values)) @@ -958,26 +1159,40 @@ def get_records_with_column_value( for next_expression in expressions[1:]: combined_expression = combined_expression & next_expression - if pending_batch is not None: - filtered_table = pending_batch.filter(combined_expression) - return self._handle_record_id_column(filtered_table, record_id_column) - - # Now check the Delta table - delta_table = self._get_delta_table(record_path) - if delta_table is None: - return None - - # Use schema-preserving read - try: - result = self._read_delta_table(delta_table, expression=combined_expression) - except _DeltaTableReadError as e: - raise self._read_failed(record_path) from e.__cause__ - - if len(result) == 0: - return None - - # Handle (remove/rename) the record id column before returning - return self._handle_record_id_column(result, record_id_column) + pending = self._pending_rows(record_path) + if pending is not None: + # Rows are pending: the stored ids decide which of them a read returns. + self._stored_ids(record_path) + stored = self._stored_rows(record_path) + + # A side without a queried column holds null there once flushed, and + # ``column == value`` never matches null, so it contributes no rows. A + # column neither side has is unknown: the filter raises as before. + queried = {c for c, _ in pair_list} + stored_columns = set() if stored is None else set(stored.table.column_names) + pending_columns = set() if pending is None else set(pending.table.column_names) + if queried <= stored_columns | pending_columns: + if not queried <= pending_columns: + pending = None + if not queried <= stored_columns: + stored = None + + # Both sides are answered from indexes (ITL-919): no re-read, no scan. + pending_matches = ( + None + if pending is None + else self._without_stored_ids( + record_path, matching_rows(pending, pair_list, combined_expression) + ) + ) + result = ( + None + if stored is None + else matching_rows(stored, pair_list, combined_expression) + ) + return self._combine_stored_and_pending( + result, pending_matches, record_id_column + ) def get_record_by_id( self, @@ -1001,14 +1216,19 @@ def get_record_by_id( self.flush_batch(record_path) record_id = coerce_record_id(record_id) - # check if record_id is found in pending batches + # Serve a pending record only if its id is not stored: a flush keeps the + # stored row and drops the pending one. record_key = self._get_record_key(record_path) - if record_id in self._pending_record_ids[record_key]: + if record_id in self._pending_record_ids[ + record_key + ] and record_id not in self._stored_ids(record_path): # Return the pending record after removing the entry id column - pending_batch = self._pending_batches[record_key] - assert pending_batch is not None, "Pending batch should not be None" - filtered_table = pending_batch.filter( - pc.field(self.RECORD_ID_COLUMN) == record_id + pending = self._pending_rows(record_path) + assert pending is not None, "Pending batch should not be None" + filtered_table = self._rows_with_ids( + pending, + [record_id], + pending.table.schema.field(self.RECORD_ID_COLUMN).type, ) if filtered_table.num_rows != 1: raise ValueError( @@ -1016,19 +1236,12 @@ def get_record_by_id( ) return self._handle_record_id_column(filtered_table, record_id_column) - # Now check the Delta table - delta_table = self._get_delta_table(record_path) - if delta_table is None: + # Now check the stored rows (the indexed snapshot, ITL-919) + stored = self._stored_rows(record_path) + if stored is None: return None - - # Use schema-preserving read - filter_expr = self._create_record_id_filter(record_id) - try: - result = self._read_delta_table(delta_table, filters=filter_expr) - except _DeltaTableReadError as e: - raise self._read_failed(record_path) from e.__cause__ - - if len(result) == 0: + result = self._rows_with_ids(stored, [record_id]) + if result.num_rows == 0: return None # Handle (remove/rename) the record id column before returning @@ -1070,84 +1283,23 @@ def get_records_by_ids( if len(record_ids) == 0: return None - # check inside the batch - delta_table = self._get_delta_table(record_path) - if delta_table is None: - return None - # Use schema-preserving read with filters - filter_expr = self._create_record_ids_filter(record_ids_list) - try: - result = self._read_delta_table(delta_table, filters=filter_expr) - except _DeltaTableReadError as e: - raise self._read_failed(record_path) from e.__cause__ - - if len(result) == 0: - return None - - # Handle record_id column based on parameter - return self._handle_record_id_column(result, record_id_column) - - def _read_delta_table( - self, - delta_table: deltalake.DeltaTable, - filters: list | None = None, - expression: pc.Expression | None = None, - ) -> pa.Table: - """ - Read table using to_pyarrow_dataset with original schema preservation. - - Args: - delta_table: The Delta table to read from - filters: Optional filters to apply - expression: Optional filter expression, used instead of ``filters`` - - Returns: - Arrow table with preserved schema - - Raises: - _DeltaTableReadError: If the table's data could not be read, with the - backend's error as its ``__cause__``. An error in the filter itself - (an unknown column, say) is raised as is. - """ - filter_expr = None - # Use to_pyarrow_dataset with as_large_types for Polars compatible arrow table loading - try: - dataset = delta_table.to_pyarrow_dataset(as_large_types=True) - except Exception as e: - raise _DeltaTableReadError from e - if filters and expression is None: - for filt in filters: - if len(filt) == 3: - col, op, val = filt - if op == "=": - expr = pc.equal(pc.field(col), pa.scalar(val)) # type: ignore - elif op == "in": - expr = pc.is_in(pc.field(col), pa.array(val)) # type: ignore - else: - logger.warning( - f"Unsupported filter operation: {op}. Falling back to table-level filter application which may be less efficient." - ) - # Fallback to table-level filtering - return dataset.to_table()(filters=filters) - - if filter_expr is None: - filter_expr = expr - else: - filter_expr = pc.and_(filter_expr, expr) # type: ignore - elif expression is not None: - filter_expr = expression - - # Convert view types before filtering: PyArrow has no comparison/sort - # kernels for string_view, so filtering the scan directly crashes - # (ENG-601). Trades predicate pushdown for correctness. - try: - scanned = dataset.to_table() - except Exception as e: - raise _DeltaTableReadError from e - table = arrow_utils.normalize_table_view_types(scanned) - if filter_expr is not None: - return table.filter(filter_expr) - return table + stored = self._stored_rows(record_path) + pending = self._pending_rows(record_path) + result = None + if stored is not None: + result = self._rows_with_ids(stored, record_ids_list) + + pending_matches = None + if pending is not None: + id_type = pending.table.schema.field(self.RECORD_ID_COLUMN).type + # Reading the stored ids drops the pending rows a flush would not + # insert; it raises StoreReadError when the stored table cannot be read. + pending_matches = self._without_stored_ids( + record_path, self._rows_with_ids(pending, record_ids_list, id_type) + ) + return self._combine_stored_and_pending( + result, pending_matches, record_id_column + ) def to_config(self, db_registry: Any = None) -> dict[str, Any]: """Serialize database configuration to a JSON-compatible dict.""" @@ -1272,7 +1424,7 @@ def _flush_by_key(self, record_key: str) -> None: try: # Combine all tables in the batch. Convert view types so they never - # reach storage (see _read_delta_table / ENG-601). + # reach storage (ENG-601). combined_table = arrow_utils.normalize_table_view_types( pending_batch.combine_chunks() ) @@ -1306,6 +1458,8 @@ def _flush_by_key(self, record_key: str) -> None: schema_mode="merge" if self.allow_schema_evolution else "overwrite", storage_options=self._storage_options or None, ) + base = _CommittedSnapshot(version=-1, ids=set()) + inserted_ids = self._non_null_ids(combined_table) logger.debug( f"Created new Delta table for {record_key} with {len(combined_table)} records" ) @@ -1320,28 +1474,38 @@ def _flush_by_key(self, record_key: str) -> None: f"{combined_table.schema.field(column).type}). " "The stored rows are unchanged and nothing was written." ) - delta_table.merge( - source=combined_table, - predicate=f"target.{self.RECORD_ID_COLUMN} = source.{self.RECORD_ID_COLUMN}", - source_alias="source", - target_alias="target", - merge_schema=self.allow_schema_evolution, - ).when_not_matched_insert_all().execute() + base = self._snapshot_for_commit( + record_key, delta_table, table_uri, abs_path_components + ) + assert base.ids is not None + incoming_ids = self._non_null_ids(combined_table) + inserted_ids = incoming_ids - base.ids + if len(inserted_ids) == len(incoming_ids) and self._stores_as_is( + delta_table, combined_table + ): + # Every pending id is new in the version just loaded and the rows + # have the stored schema: append (ITL-919). A MERGE reads the whole + # stored table on each commit; it remains for every other batch. + deltalake.write_deltalake( + table_uri, + combined_table, + mode="append", + storage_options=self._storage_options or None, + ) + else: + # Some pending ids are stored already: insert only the others. + delta_table.merge( + source=combined_table, + predicate=f"target.{self.RECORD_ID_COLUMN} = source.{self.RECORD_ID_COLUMN}", + source_alias="source", + target_alias="target", + merge_schema=self.allow_schema_evolution, + ).when_not_matched_insert_all().execute() logger.debug( f"Appended batch of {len(combined_table)} records to {record_key}" ) - # Update cache - self._delta_table_cache[record_key] = deltalake.DeltaTable( - table_uri, storage_options=self._storage_options or None - ) - - # Invalidate the existing-IDs cache directly by record_key to avoid - # double-prefixing that would occur if _invalidate_cache() were used - # (it prepends _path_prefix again via _get_record_key). - self._cache_dirty[record_key] = True - except Exception as e: logger.error(f"Error flushing batch for {record_key}: {e}") # Put the tables back in the pending queue @@ -1349,6 +1513,185 @@ def _flush_by_key(self, record_key: str) -> None: self._pending_record_ids[record_key] = pending_ids raise + # The rows are committed: nothing below may put them back in the queue. + try: + committed = deltalake.DeltaTable( + table_uri, storage_options=self._storage_options or None + ) + self._delta_table_cache[record_key] = committed + self._snapshots[record_key] = self._extend_snapshot( + base, committed, inserted_ids + ) + except Exception as e: + # The next read loads the table again and reports a failure itself. + logger.warning(f"Could not reload {record_key} after committing: {e}") + self._delta_table_cache.pop(record_key, None) + self._snapshots.pop(record_key, None) + + def _rows_with_ids( + self, + rows: KeyIndex, + record_ids: list[bytes], + value_type: pa.DataType | None = None, + ) -> pa.Table: + """Return the rows whose record id is one of ``record_ids``, in table order. + + Served from the record-id index (ITL-919); a table whose id column is not + binary is filtered instead, with Arrow's comparison. + + Args: + rows: The indexed rows (stored or pending). + record_ids: The ids looked up. + value_type: The Arrow type to build the looked-up ids with for the + filter, or ``None`` to infer it from the ids. + + Returns: + The matching rows (possibly none). + """ + id_type = rows.table.schema.field(self.RECORD_ID_COLUMN).type + if not all(is_indexed_lookup(id_type, r) for r in record_ids): + return rows.table.filter( + pc.is_in( + pc.field(self.RECORD_ID_COLUMN), + pa.array(record_ids, type=value_type), + ) + ) + id_index = rows.column(self.RECORD_ID_COLUMN) + positions = {p for r in set(record_ids) for p in id_index.get(r, ())} + return rows_at(rows.table, sorted(positions)) + + @staticmethod + def _stores_as_is(delta_table: deltalake.DeltaTable, batch: pa.Table) -> bool: + """Whether ``batch`` has exactly the stored schema once the Delta writer converts it. + + Only such a batch is appended: an append stores it unchanged, as the MERGE + would. A batch that needs a cast (a ``null``-typed column, a nullability + difference) or evolves the schema goes through the MERGE, which handles it. + + Args: + delta_table: The loaded stored table. + batch: The rows about to be committed. + + Returns: + Whether an append stores ``batch`` exactly as the MERGE would. + """ + try: + incoming = _as_delta_writer_stores(batch.schema) + except Exception: + # A type Delta cannot store directly (null): leave it to the MERGE. + return False + return incoming.equals(_stored_arrow_schema(delta_table), check_metadata=False) + + @classmethod + def _non_null_ids(cls, table: pa.Table) -> set[bytes]: + """Return the record ids of ``table``'s rows, nulls left out.""" + return { + record_id + for record_id in table[cls.RECORD_ID_COLUMN].to_pylist() + if record_id is not None + } + + def _snapshot_for_commit( + self, + record_key: str, + delta_table: deltalake.DeltaTable, + table_uri: str, + abs_path_components: tuple[str, ...], + ) -> _CommittedSnapshot: + """Return the stored ids of the version a commit is about to build on. + + ``delta_table`` was loaded for this commit, so whether the pending ids are + new is decided against the table as it is now, never against an older + cached version. The cached snapshot is reused when it describes this + version; otherwise the ids are read from it. + + Args: + record_key: The table's absolute record key. + delta_table: The table freshly loaded for the commit. + table_uri: The table's location, for the error. + abs_path_components: The table's absolute path, for the error. + + Returns: + A snapshot of the loaded version with its ``ids`` read. + + Raises: + StoreReadError: If the stored ids could not be read. + """ + cached = self._snapshots.get(record_key) + if ( + cached is not None + and cached.ids is not None + and cached.version == delta_table.version() + ): + return cached + try: + ids = self._scan_ids(delta_table) + except Exception as e: + raise StoreReadError(table_uri, abs_path_components) from e + return _CommittedSnapshot(delta_table.version(), ids=ids) + + def _extend_snapshot( + self, + base: _CommittedSnapshot, + committed: deltalake.DeltaTable, + inserted_ids: set[bytes], + ) -> _CommittedSnapshot: + """Return the snapshot of ``committed``, built from ``base`` plus this commit. + + When ``committed`` is the version right after ``base`` (no other writer + committed in between), its ids are ``base``'s plus the inserted ones, and + its rows, if ``base`` has them, are ``base``'s plus the rows of the data + files the commit added, read as ``get_all_records`` reads them. Anything + else (a skipped version, a removed file, a changed schema) leaves that + part to be read again on first use. + + Args: + base: The snapshot of the version the commit built on. + committed: The table reloaded after the commit. + inserted_ids: The non-null record ids the commit inserted. + + Returns: + The new snapshot. + """ + version = committed.version() + if version != base.version + 1: + return _CommittedSnapshot(version) + assert base.ids is not None + snapshot = _CommittedSnapshot(version, ids=base.ids | inserted_ids) + if base.rows is None: + return snapshot + dataset = cast( + "pa_ds.FileSystemDataset", + delta_to_pyarrow_dataset(committed, as_large_types=True), + ) + files = frozenset(dataset.files) + if ( + base.dataset_schema is None + or not base.files <= files + or not dataset.schema.equals(base.dataset_schema) + ): + return snapshot + added = files - base.files + new_rows = arrow_utils.normalize_table_view_types( + pa.concat_tables( + [ + fragment.to_table(schema=dataset.schema) + for fragment in dataset.get_fragments() + if fragment.path in added + ] + ) + if added + else dataset.schema.empty_table() + ) + if not new_rows.schema.equals(base.rows.table.schema): + return snapshot + rows = base.rows + rows.append(new_rows, compact_tail(pa.concat_tables([rows.table, new_rows]))) + snapshot.rows = rows + snapshot.files = files + snapshot.dataset_schema = dataset.schema + return snapshot + @staticmethod def _incompatible_append_column( delta_table: deltalake.DeltaTable, batch: pa.Table @@ -1453,6 +1796,9 @@ def at(self, *path_components: str) -> "DeltaTableDatabase": _scoped_path=new_scoped_path, _shared_pending_batches=root_instance._pending_batches, _shared_pending_record_ids=root_instance._pending_record_ids, + _shared_delta_table_cache=root_instance._delta_table_cache, + _shared_snapshots=root_instance._snapshots, + _shared_pending_indexes=root_instance._pending_indexes, ) def list_sources(self) -> list[tuple[str, ...]]: diff --git a/src/orcapod/databases/extension_aware_database.py b/src/orcapod/databases/extension_aware_database.py index fd96817d5..608d00e08 100644 --- a/src/orcapod/databases/extension_aware_database.py +++ b/src/orcapod/databases/extension_aware_database.py @@ -22,7 +22,7 @@ from __future__ import annotations -from collections.abc import Collection, Mapping +from collections.abc import Callable, Collection, Mapping from typing import TYPE_CHECKING, Any from orcapod.logical_types.database_hooks import register_discovered_logical_types @@ -167,6 +167,26 @@ def add_records( def flush(self) -> None: self._db.flush() + @property + def batch_size(self) -> int: + """The wrapped database's ``batch_size``. + + Raises: + AttributeError: If the wrapped database has none, so this wrapper + satisfies ``TableCommittingDatabaseProtocol`` only when the + wrapped database does. + """ + return getattr(self._db, "batch_size") # noqa: B009 # AttributeError when absent, by design + + @property + def flush_batch(self) -> Callable[[tuple[str, ...]], None]: + """The wrapped database's ``flush_batch`` (commits one table's writes). + + Raises: + AttributeError: If the wrapped database has none (see ``batch_size``). + """ + return getattr(self._db, "flush_batch") # noqa: B009 # AttributeError when absent, by design + # ── Structural delegation ───────────────────────────────────────────────── @property diff --git a/src/orcapod/databases/in_memory_databases.py b/src/orcapod/databases/in_memory_databases.py index d25fd1b5a..ed2a7a68a 100644 --- a/src/orcapod/databases/in_memory_databases.py +++ b/src/orcapod/databases/in_memory_databases.py @@ -5,6 +5,7 @@ from collections.abc import Collection, Mapping from typing import TYPE_CHECKING, Any, cast +from orcapod.databases._row_index import KeyIndex, compact_tail, matching_rows from orcapod.databases.utils import coerce_record_id from orcapod.utils.lazy_module import LazyModule @@ -29,6 +30,10 @@ class InMemoryArrowDatabase: Supports the same pending-batch semantics as DeltaTableDatabase: records are buffered in a pending batch and become part of the committed store only after flush() is called (or flush=True is passed to a write method). + + Lookups by record ID or by a binary or string column value use hash indexes over + the committed rows, and each flush appends without copying the stored rows, so the + cost of a per-row write, flush or lookup does not grow with the rows already stored. """ RECORD_ID_COLUMN = "__record_id" @@ -40,6 +45,7 @@ def __init__( _shared_tables: "dict[str, pa.Table] | None" = None, _shared_pending_batches: "dict[str, pa.Table] | None" = None, _shared_pending_record_ids: "dict[str, set[bytes]] | None" = None, + _shared_indexes: "dict[str, KeyIndex] | None" = None, _root: InMemoryArrowDatabase | None = None, _scoped_path: tuple[str, ...] = (), ): @@ -58,6 +64,9 @@ def __init__( if _shared_pending_record_ids is not None else defaultdict(set) ) + self._indexes: dict[str, KeyIndex] = ( + _shared_indexes if _shared_indexes is not None else {} + ) # ------------------------------------------------------------------ # Path helpers @@ -146,22 +155,37 @@ def _deduplicate_within_table(self, table: "pa.Table") -> "pa.Table": # Internal helpers for duplicate detection # ------------------------------------------------------------------ - def _committed_ids(self, record_key: str) -> "set[bytes]": + def _key_index(self, record_key: str, committed: "pa.Table") -> KeyIndex: + """Return the indexes of ``committed``, the key's committed table. + + Starts them on first use; ``flush`` extends them as rows are appended. + """ + key_index = self._indexes.get(record_key) + if key_index is None or key_index.table is not committed: + key_index = KeyIndex(committed) + self._indexes[record_key] = key_index + return key_index + + def _committed_id_index(self, record_key: str) -> dict[Any, list[int]]: + """Return the record-ID index of the key's committed rows (empty if none).""" committed = self._tables.get(record_key) if committed is None or committed.num_rows == 0: - return set() - existing_ids = committed[self.RECORD_ID_COLUMN].to_pylist() - # TODO: evaluate the efficiency of this implementation - return {id for id in existing_ids if id is not None} + return {} + return self._key_index(record_key, committed).column(self.RECORD_ID_COLUMN) def _filter_existing_records( self, record_key: str, table: "pa.Table" ) -> "pa.Table": """Filter out records whose IDs are already in pending or committed store.""" input_ids = set(table[self.RECORD_ID_COLUMN].to_pylist()) - all_existing = input_ids & ( - self._pending_record_ids[record_key] | self._committed_ids(record_key) - ) + pending_ids = self._pending_record_ids[record_key] + committed_ids = self._committed_id_index(record_key) + all_existing = { + record_id + for record_id in input_ids + if record_id in pending_ids + or (record_id is not None and record_id in committed_ids) + } if not all_existing: return table mask = pc.invert( @@ -272,16 +296,30 @@ def flush(self) -> None: committed = self._tables.get(record_key) if committed is None: - self._tables[record_key] = pending - else: - # Insert-if-not-exists: keep committed rows not overwritten by new batch, - # then append the new batch on top. - new_ids = set(pending[self.RECORD_ID_COLUMN].to_pylist()) + self._tables[record_key] = pending.combine_chunks() + continue + + # Insert-if-not-exists: keep committed rows not overwritten by new batch, + # then append the new batch on top. + new_ids = pending[self.RECORD_ID_COLUMN].to_pylist() + committed_ids = self._committed_id_index(record_key) + if any(record_id in committed_ids for record_id in new_ids): mask = pc.invert( pc.is_in(committed[self.RECORD_ID_COLUMN], pa.array(list(new_ids))) ) kept = committed.filter(mask) - self._tables[record_key] = pa.concat_tables([kept, pending]) + self._tables[record_key] = pa.concat_tables( + [kept, pending] + ).combine_chunks() + # Rows moved: the indexes are rebuilt on the next lookup. + self._indexes.pop(record_key, None) + continue + + appended = compact_tail(pa.concat_tables([committed, pending])) + self._tables[record_key] = appended + key_index = self._indexes.get(record_key) + if key_index is not None and key_index.table is committed: + key_index.append(pending, appended) # ------------------------------------------------------------------ # Path scoping @@ -322,6 +360,7 @@ def at(self, *path_components: str) -> "InMemoryArrowDatabase": _shared_tables=self._tables, _shared_pending_batches=self._pending_batches, _shared_pending_record_ids=self._pending_record_ids, + _shared_indexes=self._indexes, _root=new_root, _scoped_path=new_scoped_path, ) @@ -492,12 +531,18 @@ def get_records_with_column_value( self.flush() record_key = self._get_record_key(record_path) - table = self._combined_table(record_key) - if table is None: + committed = self._tables.get(record_key) + if committed is not None and committed.num_rows == 0: + committed = None + pending = self._pending_batches.get(record_key) + if pending is not None and pending.num_rows == 0: + pending = None + if committed is None and pending is None: return None + pair_list: list[tuple[str, Any]] if isinstance(column_values, Mapping): - pair_list = list(column_values.items()) + pair_list = list(cast(Mapping[str, Any], column_values).items()) else: pair_list = cast(list[tuple[str, Any]], list(column_values)) @@ -506,7 +551,26 @@ def get_records_with_column_value( for expr in expressions[1:]: combined_expr = combined_expr & expr - filtered = table.filter(combined_expr) + parts = [] + if committed is not None: + parts.append( + self._filter_committed(record_key, committed, pair_list, combined_expr) + ) + if pending is not None: + parts.append(pending.filter(combined_expr)) + filtered = parts[0] if len(parts) == 1 else pa.concat_tables(parts) if filtered.num_rows == 0: return None return self._handle_record_id_column(filtered, record_id_column) + + def _filter_committed( + self, + record_key: str, + committed: "pa.Table", + pair_list: list[tuple[str, Any]], + expression: "pc.Expression", + ) -> "pa.Table": + """Return the committed rows matching ``expression``, in table order.""" + return matching_rows( + self._key_index(record_key, committed), pair_list, expression + ) diff --git a/src/orcapod/databases/storage_utils.py b/src/orcapod/databases/storage_utils.py index 544d20e69..69410c432 100644 --- a/src/orcapod/databases/storage_utils.py +++ b/src/orcapod/databases/storage_utils.py @@ -1,20 +1,30 @@ """ Storage utility helpers for DeltaTableDatabase. -Provides two public functions: +Provides three public functions: - parse_base_path: Converts any path type (str, Path, UPath) into a URI string and a merged dict of deltalake-compatible storage_options. - is_cloud_uri: Returns True for cloud storage schemes (s3://, gs://, etc.). +- delta_to_pyarrow_dataset: Opens a Delta table as a pyarrow dataset, reading local + tables through a native filesystem. """ from __future__ import annotations import logging -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any + +from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: from pathlib import Path + + import deltalake + import pyarrow.dataset as pa_ds + import pyarrow.fs as pa_fs from upath import UPath +else: + pa_fs = LazyModule("pyarrow.fs") logger = logging.getLogger(__name__) @@ -38,6 +48,42 @@ def is_cloud_uri(uri: str) -> bool: return scheme in _CLOUD_SCHEMES +def delta_to_pyarrow_dataset( + delta_table: deltalake.DeltaTable, **kwargs: Any +) -> pa_ds.Dataset: + """Open a Delta table as a pyarrow dataset, natively for local tables. + + By default ``DeltaTable.to_pyarrow_dataset`` reads through + ``PyFileSystem(DeltaStorageHandler)``, a filesystem implemented in Python. Arrow + finishes tearing down a scan on a thread-pool worker after the scan returns. If that + worker releases a Python-backed buffer while the interpreter is finalizing, the process + hangs at exit in Arrow's thread-pool destructor, or aborts on Linux with Python + 3.11/3.12 (ITL-892; apache/arrow#50482, apache/arrow#54536). + + A local table is therefore read through a native pyarrow filesystem rooted at the + table, which returns the same schema and values without Python file objects. Any other + URI, a cloud store or a ``filesystem`` passed by the caller, keeps deltalake's reader. + + ``table_uri`` is percent-encoded: deltalake 1.0.2 and 1.2.1 return a bare path such as + ``/data/a%20b/``, and 1.6.6 returns ``file:///data/a%20b/``. pyarrow's own URI parser + decodes it to the native path. A root that is neither a ``file://`` URI nor an absolute + POSIX path (a Windows drive path, untested) keeps deltalake's reader. + + Args: + delta_table: The Delta table to open. + **kwargs: Passed to ``DeltaTable.to_pyarrow_dataset`` unchanged. + + Returns: + The pyarrow dataset over the table's current version. + """ + uri = delta_table.table_uri + file_uri = f"file://{uri}" if uri.startswith("/") else uri + if kwargs.get("filesystem") is None and file_uri.startswith("file://"): + filesystem, root = pa_fs.FileSystem.from_uri(file_uri) + kwargs["filesystem"] = pa_fs.SubTreeFileSystem(root, filesystem) + return delta_table.to_pyarrow_dataset(**kwargs) + + def _extract_upath_options(upath: UPath) -> dict[str, str]: """ Extract storage options from a UPath and translate to deltalake's AWS_* format. diff --git a/src/orcapod/databases/utils.py b/src/orcapod/databases/utils.py index 22da24227..62c9b34a5 100644 --- a/src/orcapod/databases/utils.py +++ b/src/orcapod/databases/utils.py @@ -2,6 +2,11 @@ from __future__ import annotations +from typing import TYPE_CHECKING, cast + +if TYPE_CHECKING: + from orcapod.protocols.database_protocols import TableCommittingDatabaseProtocol + def coerce_record_id(record_id: str | bytes) -> bytes: """Encode ``record_id`` to bytes if it is a ``str``. @@ -20,3 +25,31 @@ def coerce_record_id(record_id: str | bytes) -> bytes: if isinstance(record_id, str): return record_id.encode() return record_id + + +def as_table_committing(database: object) -> TableCommittingDatabaseProtocol | None: + """Return ``database`` if it commits table by table, else ``None`` (ITL-919). + + The one check for ``TableCommittingDatabaseProtocol``, used by ``PipelineJob.run`` + (the ``batch_size`` rule) and by function nodes (results committed before their + pipeline records). It looks the attributes up on the object, so a wrapper that + forwards them only when the database it wraps has them (``ExtensionAwareDatabase``) + qualifies exactly when that database does; ``isinstance`` on the protocol would + not, because it inspects the class. + + Args: + database: Any database object. + + Returns: + ``database``, typed as the protocol, or ``None``. + """ + try: + batch_size = getattr(database, "batch_size") # noqa: B009 # AttributeError means "no" + flush_batch = getattr(database, "flush_batch") # noqa: B009 + except AttributeError: + return None + if isinstance(batch_size, bool) or not isinstance(batch_size, int): + return None + if not callable(flush_batch): + return None + return cast("TableCommittingDatabaseProtocol", database) diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index b2bb855ba..448990fb6 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -2,6 +2,7 @@ from __future__ import annotations +import contextlib import logging from collections.abc import Iterator from typing import TYPE_CHECKING, Any @@ -12,11 +13,13 @@ from orcapod.core.nodes.source_node import SourceJobNode from orcapod.side_effects import SideEffectJobNode from orcapod.core.tracker import AutoRegisteringContextBasedTracker -from orcapod.errors import StoreReadError, StoreWriteError +from orcapod.errors import StoreError, StoreReadError, StoreWriteError from orcapod.pipeline.base import AbstractPipelineBase from orcapod.pipeline.dag import OrcaDAG from orcapod.protocols import core_protocols as cp +from orcapod.databases.utils import as_table_committing from orcapod.types import CacheMode, NodeConfig +from orcapod.utils.store_errors import flush_best_effort if TYPE_CHECKING: from pathlib import Path @@ -30,34 +33,11 @@ logger = logging.getLogger(__name__) - -def _flush_best_effort( - run_error: BaseException, *databases: ArrowDatabaseProtocol -) -> None: - """Flush each database, noting instead of raising if a flush fails. - - Used when a run has already failed: the run's own error is the one to report, - and a second failure while committing the writes that did succeed must not - replace it. A flush failure is logged and added as a note to ``run_error``, - so the caller learns that pending results may not have been stored. - - Args: - run_error: The error the run failed with, which the caller re-raises. - *databases: The databases to flush. - """ - for database in databases: - try: - database.flush() - except Exception as flush_error: - logger.exception( - "Could not flush pending writes after the run failed; " - "the rows they hold were not stored" - ) - run_error.add_note( - "Flushing pending writes after this failure also failed " - + f"({type(flush_error).__name__}: {flush_error}); results that were " - + "still pending may not have been stored." - ) +_PENDING_BEFORE_RUN_NOTE = ( + "These writes were left pending in the store before this run started (written " + "without a flush, or by a run that failed). The run stopped before computing " + "anything." +) class PipelineJob(AbstractPipelineBase[JobNode]): @@ -734,6 +714,7 @@ def run( orchestrator: "SyncPipelineOrchestrator | AsyncPipelineOrchestrator | None" = None, observer: "ExecutionObserverProtocol | None" = None, materialize_results: bool = True, + commit_every_rows: int = 100, ) -> "PipelineJob": """Execute the resolvable subgraph of this job in place. @@ -758,19 +739,37 @@ def run( result access — DB records are written regardless, and ``job.nodes["label"].get_all_records()`` works the same in either case. + commit_every_rows: How many rows a function node processes between + commits of its results and pipeline records to the store + (default 100). Each node also commits when it finishes, before a + failed run raises, and the job commits at its end. On a durable + store (``DeltaTableDatabase``) a hard crash (power loss, killed + process) loses at most the rows of the running node since its + last commit; re-running the job recomputes only those. A larger + value commits less often (faster, fewer files); ``1`` commits + after every row. A node run on its own, outside ``run()``, + commits after every row. It must be smaller than the store's + ``batch_size`` when the store has one (``DeltaTableDatabase``, + default 1000): the store commits every pending table by itself + once one table's pending rows reach that size, which would + break the node's results-before-records commit order. Returns: ``self`` — the same ``PipelineJob`` instance, with ``_has_run`` and ``_run_id`` set. Raises: - ValueError: If no store is set. + ValueError: If no store is set, or ``commit_every_rows`` is not a + positive integer smaller than the store's ``batch_size``. RuntimeError: If no pipeline has been compiled. StoreWriteError: If a node's results or records could not be written to the store, whatever the orchestrator's ``error_policy``. The run stops at the first failed write. Writes that succeeded before it are flushed first, so rows stored before the failure are kept, and re-running the job does not recompute stored function results. + A failed commit of a node's batch names the node but no row + (``flush=True``): the batch's writes stay pending and are retried + once by the flush that follows the failure. StoreReadError: If a stored table exists but could not be read, whatever the orchestrator's ``error_policy``, also when the read was needed to write a node's results. Its rows are not recomputed @@ -782,11 +781,32 @@ def run( from orcapod.pipeline.observer import NoOpObserver from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator + if ( + isinstance(commit_every_rows, bool) + or not isinstance(commit_every_rows, int) + or commit_every_rows < 1 + ): + raise ValueError( + "commit_every_rows must be a positive integer (rows per commit), " + f"got {commit_every_rows!r}" + ) if not self._compiled: raise RuntimeError( "PipelineJob has no compiled pipeline. " "Record invocations inside a 'with job:' block first." ) + table_committing = as_table_committing(self._store) + if ( + table_committing is not None + and commit_every_rows >= table_committing.batch_size + ): + raise ValueError( + f"commit_every_rows={commit_every_rows} must be smaller than the " + f"store's batch_size={table_committing.batch_size}: the store commits a " + "table by itself once it holds batch_size pending rows, which could " + "commit pipeline records before their results. Use a smaller " + "commit_every_rows, or create the store with a larger batch_size." + ) if self._store is None: raise ValueError( "PipelineJob.run() requires a store. " @@ -869,19 +889,42 @@ def run( pipeline_db = self._store.at(*self._name) result_db = pipeline_db.at("_result") + # A store that commits a table by itself at batch_size: commit what it already + # holds pending (writes made without a flush, or a failed run's batches) before + # this run writes. Those rows would otherwise count towards a table's + # batch_size, which could then be reached mid-run and commit this run's pipeline + # records ahead of their results (ITL-919). + if table_committing is not None: + try: + pipeline_db.flush() + except StoreError as exc: + exc.add_note(_PENDING_BEFORE_RUN_NOTE) + raise + except Exception as exc: + flush_error = StoreWriteError(pipeline_db.base_path, flush=True) + flush_error.add_note(_PENDING_BEFORE_RUN_NOTE) + raise flush_error from exc + effective_orchestrator = orchestrator or SyncPipelineOrchestrator() try: - effective_orchestrator.run( - exec_dag, - observer=effective_observer, - run_id=run_id, - pipeline_uri=pipeline_uri, - materialize_results=materialize_results, - ) + with contextlib.ExitStack() as batched: + # Function nodes commit every commit_every_rows rows (ITL-919). + for node in exec_dag.nodes(): + if isinstance(node, FunctionJobNode): + batched.enter_context( + node._batched_commits(commit_every_rows) # pyright: ignore[reportPrivateUsage] # the job owns its nodes' commit cadence + ) + effective_orchestrator.run( + exec_dag, + observer=effective_observer, + run_id=run_id, + pipeline_uri=pipeline_uri, + materialize_results=materialize_results, + ) except Exception as run_error: # Commit the writes that did succeed before the failure, so their rows # do not look dropped, then report the failure itself. - _flush_best_effort(run_error, pipeline_db, result_db) + flush_best_effort(run_error, pipeline_db, result_db) raise # Flush databases. Every view of the store shares one set of pending diff --git a/src/orcapod/protocols/database_protocols.py b/src/orcapod/protocols/database_protocols.py index 3215a342c..98ad15e41 100644 --- a/src/orcapod/protocols/database_protocols.py +++ b/src/orcapod/protocols/database_protocols.py @@ -159,6 +159,27 @@ class ArrowDatabaseWithMetadataProtocol( pass +class TableCommittingDatabaseProtocol(Protocol): + """A database that commits table by table (ITL-919). + + Such a database (``DeltaTableDatabase``) can commit one table's pending writes + on its own, and commits a table by itself once that table holds ``batch_size`` + pending rows. A function node uses ``flush_batch`` to commit its results before + its pipeline records; ``PipelineJob.run`` keeps ``commit_every_rows`` below + ``batch_size``. A wrapper satisfies this protocol only when the database it + wraps does. Check with ``orcapod.databases.utils.as_table_committing``. + """ + + @property + def batch_size(self) -> int: + """Pending rows of one table at which the database commits that table.""" + ... + + def flush_batch(self, record_path: tuple[str, ...]) -> None: + """Commit the pending writes of the table at ``record_path`` only.""" + ... + + @runtime_checkable class DatabaseRegistryProtocol(Protocol): """Protocol for a database config registry used during pipeline save/load. diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 854eab5ba..0cc8218b4 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -238,7 +238,12 @@ def _execute_side_effect_row( and pipeline_database is not None and table_path is not None ): - prior = pipeline_database.get_record_by_id(table_path, record_id) + try: + prior = pipeline_database.get_record_by_id(table_path, record_id) + except StoreError as exc: + # A failed lookup fails the run like a failed write: name pod and row. + exc.add_context(node_label=pod_name, tag=tag.as_dict()) + raise if prior is not None: return (tag, data) # already completed — re-emit without re-delivery diff --git a/src/orcapod/utils/store_errors.py b/src/orcapod/utils/store_errors.py index a28c1a640..20d447d18 100644 --- a/src/orcapod/utils/store_errors.py +++ b/src/orcapod/utils/store_errors.py @@ -8,11 +8,15 @@ from __future__ import annotations import logging -from collections.abc import Iterator +from collections.abc import Callable, Iterator from contextlib import contextmanager +from typing import TYPE_CHECKING from orcapod.errors import StoreError +if TYPE_CHECKING: + from orcapod.protocols.database_protocols import ArrowDatabaseProtocol + logger = logging.getLogger(__name__) @@ -63,3 +67,44 @@ def keep_store_error(store_error: StoreError | None, during: str) -> Iterator[No yield except Exception as failure: note_callback_failure(store_error, during, failure) + + +def commit_best_effort(run_error: BaseException, commit: Callable[[], None]) -> None: + """Commit pending writes after a run failed, noting instead of raising a failure. + + The run's own error is the one to report, and a second failure while committing + the writes that did succeed must not replace it. A failed commit is logged and + added as a note to ``run_error``, so the caller learns that pending results may + not have been stored. + + Args: + run_error: The error the run failed with, which the caller re-raises. + commit: Commits the pending writes, such as a database's ``flush``. + """ + try: + commit() + except Exception as flush_error: + logger.exception( + "Could not flush pending writes after the run failed; " + "the rows they hold were not stored" + ) + run_error.add_note( + "Flushing pending writes after this failure also failed " + + f"({type(flush_error).__name__}: {flush_error}); results that were " + + "still pending may not have been stored." + ) + + +def flush_best_effort( + run_error: BaseException, *databases: ArrowDatabaseProtocol +) -> None: + """Flush each database after a run failed, noting instead of raising a failure. + + See ``commit_best_effort``. + + Args: + run_error: The error the run failed with, which the caller re-raises. + *databases: The databases to flush. + """ + for database in databases: + commit_best_effort(run_error, database.flush) diff --git a/superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md b/superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md new file mode 100644 index 000000000..546c5e252 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md @@ -0,0 +1,174 @@ +# ITL-864 DeltaTableDatabase read-your-writes Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Every read method of `DeltaTableDatabase` returns exactly the rows a flush would leave, through every handle of an `at()` family, so `node.get_all_records()` returns every row right after the first run on a Delta store. + +**Architecture:** Two defects, one contract. (1) `at()` handles share the root's pending batches, and any handle's `flush()` writes every pending key, but each handle keeps its own `_delta_table_cache`, so a handle that loaded a table earlier keeps serving a stale `DeltaTable` snapshot after a sibling flushed. The fix shares `_delta_table_cache`, `_existing_ids_cache` and `_cache_dirty` across the family, exactly as pending state is already shared. (2) The read methods merge pending and stored rows inconsistently (`get_records_by_ids` ignores pending; `get_records_with_column_value` ignores stored whenever a pending batch exists; overlapping ids show twice). The fix: every read returns stored rows plus pending rows whose record id is not stored (Delta's flush is `when_not_matched_insert_all`, so the stored row wins), `None` when empty. + +**Tech Stack:** Python, pyarrow, deltalake, pytest, `uv run`. + +Approved by the coordinator (option A, Nov 10, 2026). No public API, hash, stored schema or dependency change. Error handling is untouched (PR #290 / ITL-884 rewrites it in the same file; merge main after it lands). + +--- + +## Reproduction (Step 1 of the brief) + +`PipelineJob`, `DictSource` of 2 or 3 ints, `int -> int` pod, `DeltaTableDatabase`: + +| store | orchestrator | n | `get_all_records` | `iter_data` | +|---|---|---|---|---| +| delta | sync | 2 / 3 | 1 / 1 | 2 / 3 | +| delta | async | 2 / 3 | 1 / 2 | 2 / 3 | +| memory | both | 2 / 3 | 2 / 3 | 2 / 3 | + +Trace: the result-store handle (`at("pipeline", "_result")`) flushes the pipeline table's rows; the node's pipeline handle (`at("pipeline")`) cached the table at version 0 and still reads version 0 while the table is at version 2. Its pending batch is empty; a fresh handle reads all rows. + +## Contract and per-method findings (database level) + +Contract: a read through a handle reflects all writes of its `at()` family, pending and flushed together, and equals what a flush would leave: no duplicates, no missing rows. + +| case | `get_all_records` | `get_record_by_id` | `get_records_by_ids` | `get_records_with_column_value` | +|---|---|---|---|---| +| A sibling handle flushed | stale | stale | stale | stale | +| B stored + pending, one handle | ok | ok | misses pending | misses stored when pending exists; empty table instead of `None` | +| C pending id already stored, `skip_duplicates=False` | duplicate | returns pending row, flush keeps stored | (dup) | (dup) | +| D same, `skip_duplicates=True` | ok | ok | ok | ok | +| `record_id_column=` | returns (no hang) | | | | + +Other backends: `InMemoryArrowDatabase` shares `_tables` across `at()` and `ConnectorArrowDatabase` re-reads committed rows on every call, so neither goes stale; both merge pending + committed in every method. Both flush with the NEW row winning on id overlap (Delta keeps the stored row) and show both rows before flush under `skip_duplicates=False`. Report only. + +Thread safety: the async orchestrator runs only the user function in `run_in_executor` (`core/executors/local.py`); every database read and write happens on the event-loop thread, as for the already-shared pending dicts, so sharing the caches adds no cross-thread access. + +## File structure + +- Modify: `src/orcapod/databases/delta_lake_databases.py` — `__init__` / `at()` share caches; new helpers `_pending_not_stored` and `_combine_stored_and_pending`; the four read methods use them. +- Create: `tests/test_databases/test_delta_read_your_writes.py` — database-level regression tests. +- Create: `tests/test_pipeline/test_delta_node_records_after_run.py` — `PipelineJob` regression, both orchestrators. +- Modify: `DESIGN_ISSUES.md` — one entry appended at the end. + +--- + +### Task 1: Failing database-level tests + +**Files:** Create `tests/test_databases/test_delta_read_your_writes.py` + +- [ ] **Step 1: Write the tests** + +Helpers: `_rows(db, ids, vals, groups)` builds a table with `__record_id` (`large_binary`), `v` (`int64`), `g` (`int64`); `_values(table)` returns sorted `v` or `None`. A `READS` parametrization maps each read method to a call and its expected values: + +```python +READS = { + "get_all_records": (lambda db: db.get_all_records(PATH), [1, 2]), + "get_record_by_id": (lambda db: db.get_record_by_id(PATH, b"b"), [2]), + "get_records_by_ids": (lambda db: db.get_records_by_ids(PATH, [b"a", b"b"]), [1, 2]), + "get_records_with_column_value": ( + lambda db: db.get_records_with_column_value(PATH, {"g": 0}), [1, 2]), +} +``` + +Tests: +1. `test_sibling_handle_sees_rows_flushed_by_another_handle[method][flusher]`: `root = DeltaTableDatabase(tmp_path)`, `h1, h2 = root.at("s"), root.at("s")`; `h1` writes `a=1` with `flush=True` and reads once (loads its snapshot); `h2` writes `b=2`; read through `h1` and through `root` (path `("s",) + PATH`) before flush; flush through `flusher` in (`h2`, `root`, `h1`); read through `h1`, `root` and a new `root.at("s")` after flush. All equal the expected values. +2. `test_pending_and_stored_rows_are_both_returned[method]`: one handle, stored `a=1 (g=0), b=2 (g=1)`, pending `c=3 (g=1)`: `get_records_by_ids([a, c]) == [1, 3]`, `get_records_with_column_value({"g": 1}) == [2, 3]`, `{"g": 0} == [1]`, `get_all_records == [1, 2, 3]`, `get_record_by_id(c) == [3]`. +3. `test_no_match_returns_none`: stored + pending, `get_records_with_column_value({"g": 9})` and `get_records_by_ids([b"zz"])` are `None`. +4. `test_pending_row_with_stored_id_reads_as_flush_keeps_it[skip_duplicates]`: stored `a=1`; pending add of `a=10, d=4`; before flush each method returns `a=1` and `d=4` once (`get_all_records == [1, 4]`, `get_record_by_id(a) == [1]`, `get_records_by_ids([a, d]) == [1, 4]`, `get_records_with_column_value({"g": 0}) == [1, 4]`); after `flush()` the same values. +5. `test_record_id_column_is_returned_for_merged_rows`: stored + pending, `get_all_records(PATH, record_id_column="rid")` has column `rid` with all three ids and no `__record_id`; same for `get_records_by_ids` and `get_records_with_column_value`. + +- [ ] **Step 2: Run, verify the expected failures** + +Run: `uv run pytest tests/test_databases/test_delta_read_your_writes.py -v` +Expected: test 1 fails for every method after flush; test 2 fails for `get_records_by_ids` / `get_records_with_column_value`; test 3 fails (empty table, not `None`); test 4 fails for `skip_duplicates=False`. + +### Task 2: Failing PipelineJob test + +**Files:** Create `tests/test_pipeline/test_delta_node_records_after_run.py` + +- [ ] **Step 1: Write the test** + +```python +@pytest.mark.parametrize("orchestrator", [SyncPipelineOrchestrator, AsyncPipelineOrchestrator]) +def test_node_returns_every_row_right_after_first_run(tmp_path, orchestrator): + source = op.sources.DictSource([{"id": i, "x": 10 * i} for i in range(3)], tag_columns=["id"]) + + @op.function_pod(output_keys=["result"]) + def double(x: int) -> int: + return x * 2 + + job = PipelineJob(store=DeltaTableDatabase(tmp_path)) + with job: + double.pod(source, label="double") + job.run(orchestrator=orchestrator()) + + records = job.nodes["double"].get_all_records() + assert records is not None + assert sorted(records.column("result").to_pylist()) == [0, 20, 40] +``` + +- [ ] **Step 2: Run, verify it fails** (`1 != 3` rows) for both orchestrators. + +- [ ] **Step 3: Commit the failing tests** + +```bash +git add tests/test_databases/test_delta_read_your_writes.py tests/test_pipeline/test_delta_node_records_after_run.py +git commit -m "test(databases): regression tests for Delta reads missing own writes (ITL-864)" +``` + +### Task 3: Share the snapshot caches across the `at()` family + +**Files:** Modify `src/orcapod/databases/delta_lake_databases.py` (`__init__`, `at()`) + +- [ ] **Step 1:** Add keyword-only private constructor parameters `_shared_delta_table_cache`, `_shared_existing_ids_cache`, `_shared_cache_dirty`, used when given (same pattern as `_shared_pending_batches`); `at()` passes the root instance's dicts. Comment why: any family member's `flush()` writes every pending key and refreshes the cache entry for it, so the snapshot cache must be family-wide or a sibling keeps reading the old version. +- [ ] **Step 2:** Run Task 1 test 1 and the Task 2 test: expected PASS for test 1 and Task 2. + +### Task 4: Merge stored and pending rows the way flush does + +**Files:** Modify `src/orcapod/databases/delta_lake_databases.py` (read methods) + +- [ ] **Step 1:** Add helpers: + +```python +def _pending_not_stored(self, record_path): + """Pending rows whose record id is not stored; flush inserts only these.""" + pending = self._pending_batches.get(self._get_record_key(record_path)) + if pending is None or pending.num_rows == 0: + return None + stored_ids = self._get_existing_ids(record_path) + if stored_ids: + id_type = pending.schema.field(self.RECORD_ID_COLUMN).type + pending = pending.filter(pc.invert(pc.is_in( + pending[self.RECORD_ID_COLUMN], value_set=pa.array(list(stored_ids), type=id_type)))) + return pending if pending.num_rows > 0 else None + +def _combine_stored_and_pending(self, stored, pending, record_id_column): + parts = [t for t in (stored, pending) if t is not None and t.num_rows > 0] + if not parts: + return None + table = parts[0] if len(parts) == 1 else pa.concat_tables(parts) + return self._handle_record_id_column(table, record_id_column) +``` + +- [ ] **Step 2:** `get_all_records`: pending part uses `_pending_not_stored`. `get_record_by_id`: serve from pending only when the id is pending and not stored. `get_records_by_ids`: stored read (existing try/except unchanged) plus pending filtered by the ids. `get_records_with_column_value`: stored read (existing try/except unchanged) plus pending filtered by the expression; `None` when empty. +- [ ] **Step 3:** Run `uv run pytest tests/test_databases tests/test_pipeline -q`: all pass. +- [ ] **Step 4:** Commit `fix(databases): Delta reads return every row the handle family wrote (ITL-864)`. + +### Task 5: Design issue entry, full verification, PR + +- [ ] Append one `DESIGN_ISSUES.md` entry (resolved, with **Fix:**; note that writes from other processes are still not seen until a handle reloads the table — follow-up for the coordinator to file). +- [ ] `uv run --with ruff ruff check .`, `ruff format --check .`, `uv run --with basedpyright basedpyright` (exit code), `uv run pytest tests/` (golden hashes unchanged). +- [ ] Merge `origin/main`, rerun if it moved, push, PR "Fixes ITL-864", `orca linear attach`, `orca worktree set --pr`. + +--- + +## Review rounds and continuation notes (successor-readable) + +Done and pushed (verify, do not redo): +- Round 1 (e9c78b5c): a failed ID refresh keeps the cache dirty; `get_records_by_ids` refresh shares the stored-read handler (8f9adf8b). +- Round 2 (8f9adf8b): refresh when the record key has no ID-cache entry; stored + pending combined with `normalize_table_to_large_types` + `pa.concat_tables(promote_options="default")` (b8bcc817). Tests: `test_fresh_handle_drops_pending_rows_with_stored_ids`, evolved-schema tests. + +Round 3 (CodeRabbit review of b8bcc817, outside-diff, `get_records_with_column_value`), approved fix: +- A side (stored schema, read without loading data; pending column names) that lacks a queried column contributes no rows; after a flush its rows hold null there and `col == value` never matches null. A column on neither side keeps today's `ArrowInvalid` ("No match for FieldRef"). +- Tests: `test_query_on_column_one_side_lacks_matches_read_after_flush[column_only_pending|column_only_stored]`, `test_query_on_column_neither_side_has_raises[flushed]`. +- Also the round-1 outside-diff trivial: `_add_to_pending_batch` cast `list[str]` -> `list[bytes]`. +- Reply: signed top-level PR comment (no thread for outside-diff findings), and one for the round-1 trivial. + +Then: merge origin/main (b197544e); wait for PR #290 (ITL-884, StoreReadError) to merge; merge main again, take #290's error handling around `_pending_not_stored` / `_combine_stored_and_pending`; a failed ID refresh must raise `StoreReadError` on every read method; full suite; worker_done only after the post-#290 merge is green. diff --git a/superpowers/plans/2026-11-10-itl-892-exit-hang.md b/superpowers/plans/2026-11-10-itl-892-exit-hang.md new file mode 100644 index 000000000..2cdd8d916 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-892-exit-hang.md @@ -0,0 +1,295 @@ +# ITL-892 Exit Hang Investigation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Reproduce and characterise the interpreter-exit hang after a Delta read (ITL-892), evaluate +mitigations with hang rates over at least 40 runs each, and report to the coordinator before any +`src/` or `uv.lock` change. + +**Architecture:** One timeout harness (`harness.py`) runs a target script N times as child processes, +counts exits that miss the deadline, and samples the native stack of the first hung child (`sample` +on macOS, `py-spy`/`gdb` not required). Target scripts are small and parameterised by argv so one +harness drives every matrix cell. Version cells run in throwaway environments +(`uv run --no-project --with pyarrow==X --with deltalake==Y`) so `uv.lock` never changes. + +**Tech Stack:** Python 3.11/3.12/3.13 (uv-managed), pyarrow, deltalake, polars, Orcapod +`PipelineJob` on `DeltaTableDatabase`, GitHub Actions (temporary, non-required job) for Linux. + +Constraints (from the dispatch brief): no change to `uv.lock` or `src/` before the coordinator +approves a mitigation. All experiments live in `superpowers/specs/itl-892-exit-hang/`. Draft PR body +says `Refs ITL-892`. + +--- + +### Task 1: Harness and minimal pure reproduction + +**Files:** +- Create: `superpowers/specs/itl-892-exit-hang/harness.py` +- Create: `superpowers/specs/itl-892-exit-hang/repro_pure.py` + +- [ ] Write `repro_pure.py`: builds a small Delta table in a temp dir and performs one operation + chosen by `--op` (`write`, `merge`, `to_table`, `to_dataset`, `dataset_scan`, `write_and_read` + — the last is the ITL-864 worker's script: 1 write, 4 merges, a `to_pyarrow_dataset().to_table()` + after each). Flags: `--import-polars`, `--cpu-threads N` (`pa.set_cpu_count`), `--io-threads N` + (`pa.set_io_thread_count`), `--mitigation {none,del,gc,os_exit,pool_shutdown}`, `--mp {none,fork,spawn}`. + Prints `done` with flush then returns. +- [ ] Write `harness.py`: `harness.py N TIMEOUT -- `; runs N children with up to `-j` in + parallel, each in its own session; a child that misses TIMEOUT is sampled once (first hang only, + `sample 2`) then killed; prints `hung/N`, exit-code histogram, whether `done` was printed + before the hang, and writes a JSON line to `results.jsonl` with the cell label. +- [ ] Run the ITL-864 script 40 times on Python 3.12 with the locked versions. Expected: a hang rate + near 50% and a stack in `ThreadPool::~ThreadPool`. If 0/40, vary parallelism (-j1 vs -j8) and + table size before concluding. + +### Task 2: Characterise the trigger + +- [ ] Run each `--op` alone, 40 runs each: which operation is necessary. +- [ ] With the triggering op: `--cpu-threads 1`, `--io-threads 1`, env `OMP_NUM_THREADS=1`, + `ARROW_IO_THREADS=1`, `RAYON_NUM_THREADS=1`; `--import-polars`; deltalake without pyarrow + imported where the API allows (`DeltaTable.to_pyarrow_*` needs pyarrow; use `write_deltalake` + with an arro3 table / `QueryBuilder` to separate the runtimes); `--mp fork/spawn`. +- [ ] Python 3.11 and 3.13 at the triggering cell. + +### Task 3: Orcapod paths + +**Files:** +- Create: `superpowers/specs/itl-892-exit-hang/repro_orcapod.py` + +- [ ] Modes: `pipeline-sync`, `pipeline-async` (a two-source `PipelineJob` with one function pod on + `DeltaTableDatabase`, then `get_all_records`), `plain-read` (`DeltaTableDatabase` write, flush, + read). 40 runs each on 3.12 via the project environment; 3.11/3.13 via `uv run --isolated --python`. + +### Task 4: Version bisect and upstream search + +- [ ] Grid over pyarrow {20.0.0, 21, 22, 23.0.1, 24, 25.0.1, 26.0.1} × deltalake {1.0.2, 1.1.x, + 1.2.x, 1.3.x, 1.4.2, 1.5.1, 1.6.6} at the triggering cell, 40 runs each (narrow by halving once a + boundary is visible); record newest non-hanging and oldest hanging combinations. +- [ ] Search apache/arrow and delta-io/delta-rs issues (WebSearch, `gh search issues`) for + `ThreadPool` destructor / exit hang / `__cxa_finalize`; link matches. + +### Task 5: Mitigations + +- [ ] (a) Best version combination from Task 4: copy the worktree to a scratch dir, `uv lock + --upgrade-package`, run golden hash tests and the full suite there (never in this worktree). +- [ ] (b) Orcapod-side shutdown, prototyped in the repro scripts only: drop `DeltaTable`/dataset + references + `gc.collect()`; an `atexit` hook; pyarrow pool shutdown if exposed; `os._exit` after + flush (for comparison only). +- [ ] (c) Alternative read path: `DeltaTable.to_pyarrow_table`, `QueryBuilder`/arro3, polars + `scan_delta`/`read_delta`, reading the parquet files listed by `DeltaTable.file_uris()` with + pyarrow directly. +- [ ] (d) Documentation workaround text. +- [ ] 40 runs per option; record hang rate. + +### Task 6: Linux + +- [ ] Docker is not installed on this machine. Push a temporary, non-required workflow + `.github/workflows/itl-892-exit-hang-probe.yml` (`workflow_dispatch` + push on this branch only) + running the harness on ubuntu-latest for 3.11/3.12/3.13. Remove it before the PR leaves draft. + +### Task 7: Report + +**Files:** +- Create: `superpowers/specs/itl-892-exit-hang/FINDINGS.md` +- Modify: `DESIGN_ISSUES.md` (one appended entry for ITL-892) + +- [ ] Findings: hang-rate tables, stack, trigger, versions, upstream links, options (a)-(d) with + evidence and cost, recommendation, proposed regression check and its reliability. +- [ ] Commit (Conventional Commits, Co-Authored-By trailer), push, open draft PR "Refs ITL-892". +- [ ] `ask` the coordinator with results and options; do not implement a mitigation before approval. + +--- + +## Phase 2: approved mitigation, option (c) plus (d) + +Approved by the coordinator after reading FINDINGS.md (November 10, 2026). The scope is the +read path only. Do not implement (b3); record it in DESIGN_ISSUES. Keep the `src/` change to +one helper plus one-line call-site changes, because PR #290 (ITL-884) and PR #293 (ITL-864) +rewrite the same read methods and merge first. PR body: `Fixes ITL-892`. + +### Task 8: Failing regression test + +**Files:** +- Create: `tests/test_databases/test_delta_exit.py` + +- [ ] Each case starts `RUNS = 20` fresh interpreters. The child script is inline and uses + only the public API: + - `DeltaTableDatabase` `add_records` + `flush`, then `get_all_records` on a new handle. + - `DeltaTableSource` over a table written with `write_deltalake`. +- [ ] The deadline is 120 s; run 4 children at a time. +- [ ] Fail if any run misses the deadline or exits non-zero. The message names the defect: + "process did not exit after a Delta read (ITL-892)". +- [ ] Run it: expect FAIL on the current code (macOS: about 53% / 8% per run). + +### Task 9: Equivalence test (type strictness) + +**Files:** +- Create: `tests/test_databases/test_delta_native_fs_read.py` + +- [ ] Write stores through Orcapod's public API: a `PipelineJob` on `DeltaTableDatabase` with + function pods returning `int`, `Path`, a dataclass, `numpy.ndarray`, `pandas.DataFrame`, + `list[int]` and a nullable value. For every table in `db.list_sources()`, assert that + `db.get_all_records(path)` equals deltalake's default reader + (`DeltaTable(uri).to_pyarrow_dataset(as_large_types=True).to_table()`): schema equality with + `check_metadata=True`, then values. +- [ ] `DeltaTableSource` over a hand-written table: narrow ints, decimal, tz timestamp, nested + list and struct, non-nullable field, partitioned by a column with values `"a b"`, `"x/y"`, + `"é%"` and `None`. Its `as_table()` must equal the same `DeltaTableSource` built under a + monkeypatched default reader. +- [ ] Passes before and after the change; the guard is that it keeps passing after. + +### Task 10: Helper and call sites + +**Files:** +- Modify: `src/orcapod/databases/storage_utils.py` (new `delta_to_pyarrow_dataset`) +- Modify: `src/orcapod/databases/delta_lake_databases.py` (3 call sites: + `_refresh_existing_ids_cache`, `get_all_records`, `_read_delta_table`) +- Modify: `src/orcapod/core/sources/delta_table_source.py` (1 call site) + +- [ ] `delta_to_pyarrow_dataset(delta_table, **kwargs)`: if `delta_table.table_uri` is a + local path (no scheme, or `file://`), pass + `filesystem=pa.fs.SubTreeFileSystem(, pa.fs.LocalFileSystem())`. Otherwise use + deltalake's default reader. Forward every other keyword unchanged. The docstring names + ITL-892 and the upstream issues. +- [ ] `to_pyarrow_table()` becomes `delta_to_pyarrow_dataset(dt).to_table()`; deltalake's own + `to_pyarrow_table` does exactly that, so behaviour is identical. +- [ ] Tasks 8 and 9 pass. The full suite and the golden hash tests are unchanged. + +### Task 11: Docs, changelog, design issues, cleanup + +- [ ] `docs/troubleshooting.md` (new; in the mkdocs nav): the process-exit note, the remaining + exposure for cloud stores, apache/arrow#50482, #54536 and PR #54467, and the workaround. +- [ ] `CHANGELOG.md` `[0.3.0]` entry. +- [ ] DESIGN_ISSUES DX1: status resolved, with a **Fix:** note, and (b3) recorded as the + considered alternative and a possible lazy hook for cloud stores. +- [ ] Delete `.github/workflows/itl-892-exit-hang-probe.yml`. +- [ ] Run ruff, basedpyright and the full suite. Merge `main` after #290 and #293 land. Mark + the PR ready, take it through CodeRabbit, and send worker_done with the delta-rs issue text. + +--- + +## Phase 3: CodeRabbit round 1 (resumed after a machine restart, November 11, 2026) + +State on arrival: Tasks 8-11 were done, committed and pushed (head 561f5999), CI was green and +the PR was ready for review. CodeRabbit round 1 (review 5855612064: 8 inline findings and 5 +nitpicks in the review body) was unanswered. The coordinator approved this plan. + +### Task 12: Percent-encoded table roots (behaviour fix) + +`DeltaTable.table_uri` is percent-encoded on every supported deltalake: 1.0.2 and 1.2.1 return +`/x/a%20b/t/`, and 1.6.6 returns `file:///x/a%20b/`. The helper passed it undecoded to +`SubTreeFileSystem`, so a store under `a b`, `pct%` or `é ü` was unreadable. `get_all_records` +swallowed the error, so the store looked empty. + +- [x] Failing test first, in `test_delta_native_fs_read.py`: for store roots `a b`, `pct%`, + `p%20q`, `é ü`, `x#y`, `q?r`, `semi;c` and `plus+` (skip a name the installed deltalake + itself cannot write), `DeltaTableDatabase.get_all_records` and `DeltaTableSource.as_table()` + equal the default reader. Assert a correct read, never an empty result. After #290 an + unreadable table raises `StoreReadError`. +- [x] Fix: `pa_fs.FileSystem.from_uri(uri if "://" in uri else "file://" + uri)`, and use the + result only when it is a `LocalFileSystem`. Windows drive paths are untested; record a note. +- [x] Fix the troubleshooting snippet the same way. + +### Task 13: Docstrings and research-script checks + +- [x] Docstrings in both new test files and in the `superpowers/specs/itl-892-exit-hang/*.py` + scripts. +- [x] `repro_orcapod.py`: assert that the fresh read returns 3 rows. `repro_pure.py`: exit + non-zero when the child process fails. Spot-check the patched mode. +- [x] Decline the `harness.py` worker-slot finding. The committed results came from that + harness, and the effect exists only after a hang, so it cannot touch the 0/100 fixed rates. + +### Task 14: Merge, replies, next round + +- [x] Merge origin/main (b197544e), run the full suite (6908 passed, 100 skipped, 15 xfailed), push, and reply on every thread, plus a signed + top-level comment for the nitpicks (done: replies on all 8 threads and issuecomment-6791908147). +- [ ] Next CodeRabbit round. Merge main again after #290 (watch for `StoreReadError`: the + tests must assert a raise or a correct read) and after #293. + +--- + +## Phase 4: rebuild on the stack #290 -> #293 -> #295 (November 11, 2026) + +Brief from the coordinator (task_5f115155397b): merge the rebuilt #293 +(`eywalker/itl-864-deltatabledatabase-reading-through-the`, head 06ecbd5c) into this branch, +keep all three behaviours, prove it by test, push, refresh the PR body. Brainstorming skipped: +the design was approved in Phase 2; this phase is a merge with the coordinator's DO list as +the spec. State on arrival: #290 has since been squash-merged to main as 1425b840, whose tree +equals #290's head 11efe638, so the stack is now main -> #293 -> #295. + +Rules to keep after the merge: +- #290: an unreadable table raises `StoreReadError`, never "no rows"; `_read_delta_table` + keeps main's signature (no message-only parameter) and raises `_DeltaTableReadError`; the + caller wraps it with the path. +- #293: reads return stored plus pending-not-stored rows; `_refresh_existing_ids_cache` + stays dirty on failure. +- #295: every scan opens through `delta_to_pyarrow_dataset`. +- Hazard: the filter must stay OUTSIDE the `_DeltaTableReadError` try blocks; a bad filter + (unknown column, type mismatch) is a caller bug and must not become `StoreReadError`. + +### Task 15: Merge #293's branch, then origin/main + +- [x] `git merge origin/eywalker/itl-864-deltatabledatabase-reading-through-the` (merge + commit). Resolve `delta_lake_databases.py`: take #293's code and swap each + `to_pyarrow_dataset`/`to_pyarrow_table` scan for `delta_to_pyarrow_dataset(...)` inside + the existing try blocks. CHANGELOG: one `[0.3.0] — unreleased` section with every entry. + DESIGN_ISSUES: all entries. +- [x] Fix the stale comment on the `dataset.to_table()(filters=...)` ignore: #290 did not + remove that line (still on main). Report it out of scope. +- [x] `git merge origin/main` (expected no content change; both conflicts took this side). +- [x] #293's `fail_id_refresh_once` fixture patched `to_pyarrow_table`; the refresh now opens + through `to_pyarrow_dataset` (no `as_large_types`), so the fixture fails that call instead. +- [ ] Merge the NEW #293 head (after #293 merges main 1425b840); the coordinator sends it. + +### Task 16: Interaction tests (`tests/test_databases/test_delta_native_fs_read.py`) + +- [x] Filtered reads (`get_record_by_id`, `get_records_by_ids`, + `get_records_with_column_value`) of an unreadable stored table raise `StoreReadError`. +- [x] A bad filter (`get_records_with_column_value` on an unknown column, or a value whose + type does not compare) raises the same non-`StoreReadError` error before and after a flush. +- [x] Filtered reads return stored plus pending rows equal to `get_all_records` filtered in + memory, with the native reader in use. +- [x] `test_delta_exit.py` still passes. +- [x] Done in `tests/test_databases/test_delta_native_fs_filtered_reads.py` (36 cases); moving the + filter inside the read's try block fails the 2 bad-filter cases (mutation check). + +### Task 17: Verify, push, PR body, CodeRabbit, worker_done + +- [ ] Full suite on 3.11/3.12/3.13, ruff check, ruff format --check, basedpyright (exit + code), golden tests unchanged (`git diff --stat origin/main -- tests/test_hashing/golden`). +- [ ] Push; PR body: stacked note at the top, merge notes, fresh counts. +- [ ] CI green, CodeRabbit round on the new head incl. review bodies. + +--- + +## Phase 5: merge the rebuilt #293 head 48678ca5 (November 11, 2026, task_8f9c4a65455b) + +The previous worker was killed by OOM after pushing 1f73fd10 (Tasks 15-16 done, clean tree). +The new #293 head 48678ca5 differs from the merged 06ecbd5c only by main's #298 (ITL-828, +merge commits 5aa47946 and 48678ca5): no #293 code change. Brainstorming skipped: the design +was approved in Phase 2 and the coordinator's DO list is the spec. + +### Task 18: Merge + +- [x] `git merge origin/eywalker/itl-864-deltatabledatabase-reading-through-the` (merge commit): f2c45c1d, conflicts only CHANGELOG + DESIGN_ISSUES (kept both sides). + Expected conflicts: `CHANGELOG.md` (keep one `[0.3.0] — unreleased` section with every entry), + `DESIGN_ISSUES.md` (keep all entries). No conflict expected in `delta_lake_databases.py`. +- [x] Check `git merge-base --is-ancestor origin/main HEAD`. +- [x] Golden: `git diff origin/main -- tests/test_hashing/golden` must be empty. +- [x] Commit and push straight away (WIP safety). + +### Task 19: Verify (ONE suite at a time, no xdist; memory is short) + +- [x] `uv run pytest tests/test_databases -q` first. +- [x] (7548 passed, 100 skipped, 19 xfailed each; use UV_PROJECT_ENVIRONMENT per interpreter so other uv calls cannot swap .venv mid-run) Full suite on 3.13, 3.12, 3.11 in turn: `uv run --python 3.1x pytest tests/ -q -p no:cacheprovider`. +- [x] (all clean, basedpyright exit 0) `uv run --with ruff ruff check .`, `ruff format --check .`, `uv run --with basedpyright basedpyright; echo $?`. + +### Task 20: PR, CI, CodeRabbit, report + +- [x] PR body: first line `Stacked on #293: until it merges, this PR's diff also shows its changes; review after it.`; update merge notes and counts. +- [ ] CI: wait at most 20 minutes (Actions outage), report state. +- [ ] CodeRabbit on the new head: inline threads and review bodies. #293-line findings: reply + they belong there and report; mechanical on this PR's lines: fix; others: `ask`. +- [ ] worker_done with: head, conflicts/resolutions, counts, golden diff, CI/CodeRabbit state, + ITL-919 watch-outs. + +Progress note: CodeRabbit review 5865082626 (on 1f73fd10): 3 inline findings, all on #293's lines; replied on each thread (r4543139876, r4543140113, r4543140365) that they belong to #293. diff --git a/superpowers/plans/2026-11-10-pr-293-coderabbit-id-cache-refresh.md b/superpowers/plans/2026-11-10-pr-293-coderabbit-id-cache-refresh.md new file mode 100644 index 000000000..6aea99fca --- /dev/null +++ b/superpowers/plans/2026-11-10-pr-293-coderabbit-id-cache-refresh.md @@ -0,0 +1,59 @@ +# PR #293 CodeRabbit follow-up: stored-ID cache refresh failures (ITL-864) + +Revision plan approved by the coordinator (task_355ce670f6bb), provided each finding +reproduces. Branch `eywalker/itl-864-deltatabledatabase-reading-through-the`. + +## Findings + +1. **Comment 4534785488 (Major), `delta_lake_databases.py:658`.** When + `_refresh_existing_ids_cache` fails after the Delta table loaded, it stores an empty ID + set and marks the cache **clean**. The loaded table is cached, so `_get_existing_ids` + never refreshes again: a pending row whose id is already stored is served next to the + stored row (a duplicate) by every read method. + *Fix:* the failure path marks the cache dirty, so the next read retries the refresh. + +2. **Comment 4534785497 (Minor), `delta_lake_databases.py:891`.** In + `get_records_by_ids`, `_pending_not_stored` refreshes the ID cache before the + method's stored-read `try`, so a refresh failure escapes while a stored-read failure is + logged and returns `None`. + *Fix:* run the refresh inside the same `try` as the stored read (one handler, no second + `return None`). After PR #290 (ITL-884) merges, that handler is gone and both failures + raise `StoreReadError` naming the table. + +## Steps (TDD) + +1. Regression tests in `tests/test_databases/test_delta_read_your_writes.py`, public + read methods only, with the backend's `DeltaTable.to_pyarrow_table` (used only by the + ID refresh) patched to fail once: + - stored `a=1`, flushed; pending `a=10`, `d=4`; one read fails in the refresh; every + read method then returns `a=1`, `d=4` and never `a=10`. + - stored `a=1`, flushed; pending `c=3`; `get_records_by_ids` with the refresh failing + follows the method's stored-read error behaviour (now `None`), and the retry returns + `[1, 3]`. +2. Run them, confirm both fail for the reasons above. +3. Apply the two fixes; rerun the file, then the full suite, ruff, basedpyright. +4. Commit, push, signed replies on both threads, wait for CI and CodeRabbit. + +## After PR #290 merges + +Merge `main`; in the read methods take #290's error handling (`StoreReadError`, no +`try`/`return None`) around this branch's merge helpers. #290's +`_refresh_existing_ids_cache` leaves `_cache_dirty` untouched on failure, which is still +clean when the refresh ran only because the table was not cached yet, so keep an explicit +`self._cache_dirty[record_key] = True` there. Change the test of finding 2 to expect +`StoreReadError` and extend it to every read method. Rerun the full suite (3.11, 3.12, +3.13), golden tests unchanged. + +## Round 2 (CodeRabbit review of 8f9adf8b; coordinator approved both behaviour fixes) + +- **Outside-diff (Major), `_get_existing_ids`:** a handle that loaded the table before any + stored ids were read (a fresh root over an existing table) never refreshed them, so pending + duplicates were served. Refresh also when the record key has no id entry. Test: + `test_fresh_handle_drops_pending_rows_with_stored_ids` (four read methods). +- **4535100450 (Major), `_combine_stored_and_pending`:** stored and pending rows with any + schema difference raised `ArrowInvalid`. Large-normalize both parts (`keep_null=True`), + then `pa.concat_tables(promote_options="default")`. Contract tests assert the read before + flush equals the read after flush (missing column either way, nullability merge, struct + child added, `string`/`list` layouts); type differences keep raising `ArrowTypeError`; a + `null`-typed new column is pinned (reads back, flush fails). +- **4535100459 (nit):** Google docstrings on the five tests and the wrapper. diff --git a/superpowers/plans/2026-11-11-itl-864-stack-on-290.md b/superpowers/plans/2026-11-11-itl-864-stack-on-290.md new file mode 100644 index 000000000..0e6a703b8 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-864-stack-on-290.md @@ -0,0 +1,60 @@ +# ITL-864 / PR #293: rebuild on top of PR #290 (ITL-884) and main + +Date: 2026-11-11. Coordinator task task_0f94cf0766e2. Stack approved by the developer: +#290 -> #293 -> #295 -> Delta half of ITL-919. Previous plan: +`superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md`. + +**Goal:** the branch contains main (c8efc092) and #290 (head 11efe638) by merge commits, +with both behaviours intact: +- #290: an existing table that cannot be loaded or read raises `StoreReadError`, never + "no rows"; `_read_delta_table(delta_table, filters=None, expression=None)` raises the + private `_DeltaTableReadError` and the caller wraps it with `self._read_failed(record_path)`. +- #293: every read method returns stored rows plus pending rows whose id is not stored, + through any handle of the `at()` family (shared table / ids / dirty caches). + +No public API, hash, stored-schema or dependency change. + +## Tasks + +1. `git merge origin/main` (conflicts expected only where #293 touches the same code as + main; resolve, commit, push WIP). +2. `git merge origin/eywalker/itl-884-delta-read-failures-are-returned-as-no-rows`. + Resolution rules in `delta_lake_databases.py`: + - `_refresh_existing_ids_cache`: #290's shape (try only around the load/scan, raise + `self._read_failed(record_path) from e`), with #293's `set[bytes]` typing and the + cache left dirty on failure. + - `get_all_records`: #290's raise on a failed scan + #293's `_pending_not_stored` / + `_combine_stored_and_pending`. + - `get_records_with_column_value`: #293's one-side-lacks-the-column logic; the stored + read uses #290's `except _DeltaTableReadError -> StoreReadError`; filter errors + propagate as is. + - `get_record_by_id`: #293's "pending and not stored" check + #290's wrapping. + - `get_records_by_ids`: drop #293's `except Exception: return None` (that is exactly + what #290 forbids); `_pending_not_stored` propagates `StoreReadError`; the read uses + #290's wrapping. + - CHANGELOG: one `[0.3.0] — unreleased` section; DESIGN_ISSUES: all entries. +3. Tests: run #290's and #293's tests together. Add interaction tests (public interface, + in `tests/test_databases/test_delta_read_your_writes.py`): + - unreadable stored table with pending rows queued through a sibling handle: every read + method raises `StoreReadError` (pending rows are not returned alone as if the table + were empty), and the pending rows stay queued (nothing dropped); + - a failed stored-id refresh raises `StoreReadError` on every read method, and a later + read after the table is readable again returns stored + pending. + Show each new test's result before any code change it needs. +4. Full suite on 3.11/3.12/3.13, ruff check, ruff format --check, basedpyright (exit code), + golden hashes unchanged. +5. Push; PR description gets the "Stacked on #290" note; CI + CodeRabbit; worker_done. + +## Progress notes (successor-readable) +- Done: merged origin/main c8efc092 (7ba60eb4; conflict DESIGN_ISSUES.md only: kept main's + "Newcomer papercuts" entry and the ITL-864 entry, ours last). Merged #290 11efe638 + (a2ac8d9e; conflicts in `_refresh_existing_ids_cache`, `get_records_with_column_value`, + `get_records_by_ids`, resolved per the rules above; CHANGELOG/DESIGN_ISSUES auto-merged; + ITL-864 Fixed entry added to `[0.3.0] — unreleased` in the merge commit). +- Done: ba396d41 updates #293's two refresh tests to StoreReadError and adds + `test_unreadable_table_with_sibling_pending_rows_raises_and_drops_nothing` (verified to fail + when #293's old `except Exception: return None` is put back in `get_records_by_ids`). +- ITL-923 ephemeral re-run repro (scratchpad script): main and #290 lose records on both + orchestrators; this branch fixes sync (10 stored, 5 read) but async still stores 5 and reads + None. Out of scope; reported to the coordinator. +- Next: full suite 3.11/3.12/3.13, PR description "Stacked on #290", CI, CodeRabbit, worker_done. diff --git a/superpowers/plans/2026-11-11-itl-919-delta-batched-commits.md b/superpowers/plans/2026-11-11-itl-919-delta-batched-commits.md new file mode 100644 index 000000000..e9cd56aec --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-919-delta-batched-commits.md @@ -0,0 +1,252 @@ +# ITL-919 (part 2): batched Delta commits (D2) — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A `PipelineJob` on a `DeltaTableDatabase` runs in time linear in its rows, close to +the in-memory store, by committing a function node's writes in batches instead of once per +row, and by serving the per-row lookups from indexed in-memory state instead of re-reading +the Delta table. + +**Architecture:** (1) Function nodes running inside `PipelineJob.run` stop committing per row +and commit every `commit_every_rows` rows (a new `run()` keyword, default 100), at node end, +before raising on any failure, and at job end. (2) `DeltaTableDatabase` lookups read an +indexed snapshot of the committed rows (one per cached table handle, extended with only the +new data files after the family's own commit) plus the pending batch, which is bounded by the +commit size. (3) A commit appends when every pending id is new against a fresh table version, +and MERGEs only otherwise. + +**Tech Stack:** deltalake 1.x, PyArrow 23, pytest. + +**Branch / stack:** `eywalker/itl-919-delta-batched-commits`, created at #295's head +(14c1546b = main ebf2511b + #293 + #295), with #304's branch merged in (f84217b2) for its +index helpers and scaling tests. PR against `main`, stacked note first. + +--- + +## Developer decisions (November 11, 2026) — these override the reviewed plan below + +1. **Commit trigger is row count only.** `PipelineJob.run(..., commit_every_rows=100)` + (public; coordinator approved the `run()` keyword placement, documented next to + `materialize_results`; validated as an `int >= 1`, `bool` rejected, else `ValueError`). + No time-based commit (ITL-925), no threads or timers. Commits also at node end, before + raising on any failure, and at job end. +2. **Scope:** only function nodes running inside `PipelineJob.run` batch their commits. + `FunctionJobNode.run()` / `execute()` called directly, and a `CachedFunctionPod` used + directly, keep one commit per row (the node's attribute is `None` outside a job run). +3. **Commit failure:** `StoreWriteError(pipeline_db.base_path, node_label=..., flush=True)` + raised at the commit point, naming the node (no row/tag), chained from the store's error. + The failed batch stays pending (`_flush_by_key` restores it; keep), the job's best-effort + flush retries once, earlier batches are kept, the run fails (PR #287 semantics). A failure + writing one row still names table, node and row tag (unchanged). The node wraps; the store + receives no labels (no message-only parameters). `StoreReadError` passes through, with the + node label added by `add_context`. +4. **Amendments:** (a) on any failure the node flushes its ephemeral store best-effort + itself, never replacing the run error; (b) the crash guarantee states the + results-then-pipeline commit window and the zero-recomputation rerun recovery, with a + subprocess test interrupting between the two commits; (c) append instead of MERGE when + **every** pending id is new, MERGE otherwise; "all ids new" is decided against a FRESH + table version (loaded at flush time), never the cached snapshot; a differential run must + show identical tables. +5. **Also fix:** `side_effects.py` prior-completion `get_record_by_id` lookup raises a store + error without the row's tag (CodeRabbit on #290) — the caller adds node and tag context; + and `compute_base_entry_id` computed 3x per row in `function_node.py` — compute once per + row and pass it down (no hash or id change). +6. **ITL-923:** keep items 1, 2, 4 fixed (#293); fix what falls inside this change; report + precisely what remains (item 3 silent MERGE drop; async ephemeral re-run). + +## Design details (worker, after reading the code) + +### Commit order: results before pipeline records + +Views of one store share one set of pending batches, and `flush()` commits every pending +key of the family in the order keys were first written since their last commit. That order +does not guarantee result table before pipeline table (a batch can start with a cache-hit +row, which writes only a pipeline record; in the async path the pod stores row k+1's result +after row k's pipeline record). The node therefore commits explicitly in this order: + +1. ephemeral store `flush()` (separate database), when the node has one; +2. its result table alone, through `DeltaTableDatabase.flush_batch(record_path)` when the + result database has it (only the Delta store commits durably table by table; the + in-memory store has no crash window). Duck-typed `getattr(db, "flush_batch", None)` on + the result view (the test `CountingDatabase` forwards it); +3. `pipeline_db.flush()` (pipeline records and anything else pending in the family). + +A crash between 2 and 3 leaves results stored without pipeline records; the rerun finds them +in the result cache (no recomputation) and writes the pipeline records. A *failed* flush at +step 3 continues past failed keys (#287 `flush` semantics, unchanged): see "Remaining". + +### Node (`FunctionJobNode`) + +- `_commit_every_rows: int | None = None` set by `PipelineJob.run` on every + `FunctionJobNode` it executes, reset to `None` in a `finally`. While set, the node's + `CachedFunctionPod`s run with `auto_flush=False` (new `CachedFunctionPod.set_auto_flush`, + mirroring `CachedDataFunction.set_auto_flush`), restored afterwards. +- Counter `_rows_since_commit`: incremented after each row that went through + `_process_data_internal` successfully (sync) / each row recorded in `record_and_forward` + (async). At `>= commit_every_rows`: `_commit_pending_writes()`. +- Node end (sync and async DB path): `_commit_pending_writes()` after the last row, before + `on_node_end` (only when batching; outside a job nothing changes). +- `_commit_pending_writes()`: steps 1-3 above; `StoreReadError` gets + `add_context(node_label=...)` and propagates; any other error becomes + `StoreWriteError(self._pipeline_database.base_path, node_label=self.label, flush=True)` + from it. The counter resets only after success. +- On any failure leaving `execute` / `async_execute` while batching: flush the ephemeral + store best-effort (moved helper `flush_best_effort(run_error, *dbs)` from + `pipeline/job.py` to `orcapod/databases/utils.py` so core can use it; job.py imports it). +- `compute_base_entry_id` once per row: `_process_data_internal` / + `_async_process_data_internal` / `add_pipeline_record` accept an optional precomputed + `base_entry_id`; `execute` passes its value; `record_and_forward` reuses it. + +### Store (`DeltaTableDatabase`) + +- Shared helpers moved from `in_memory_databases.py` (#304) to + `src/orcapod/databases/_row_index.py`: `_KeyIndex`, `_is_indexed_lookup`, `_compact_tail`, + plus `build_column_index(table, column)` and `extend_column_index(index, table, offset)`. +- `_CommittedSnapshot(handle, version, table, files, index: _KeyIndex)` per record key, in + a family-shared dict `_snapshots` (passed through `at()` like the other shared caches). + Valid while `snapshot.handle is self._delta_table_cache[key]`. Built lazily from the cached + handle with `delta_to_pyarrow_dataset(handle, as_large_types=True).to_table()` + + `normalize_table_view_types` (exactly today's read). A read failure raises `StoreReadError` + (`_read_failed`); a bad filter raises as the caller's error (split kept). +- `_existing_ids_cache` / `_cache_dirty` / `_refresh_existing_ids_cache` are replaced by + the snapshot's record-id index (no full-table reload per flush). +- Lookups (`get_records_with_column_value`, `get_record_by_id`, `get_records_by_ids`): + committed rows from the snapshot (index when `_is_indexed_lookup`, else filter of the + in-memory snapshot table) + `_pending_not_stored` filtered, combined by #293's + `_combine_stored_and_pending` (unchanged schema rules). Null-id pending rows kept. +- `_flush_by_key`: load a fresh handle (as today). Existing table: get the snapshot for the + fresh version (reuse when the cached snapshot's version equals the fresh version, else + build one from the fresh handle). If no pending id is in it (nulls count as new): append + with `write_deltalake(mode="append", schema_mode="merge" if allow_schema_evolution else + None)`; else MERGE as today. After the commit, reload the handle; if its version is the + fresh version + 1, no data file was removed and the dataset schema is unchanged, extend the + snapshot with only the new fragments (read through `delta_to_pyarrow_dataset`), else drop + it (rebuilt lazily). New table: `overwrite` write as today, snapshot built lazily. + Failure: pending restored (kept), snapshot left for the old handle. +- Pending batch: bounded by `commit_every_rows` in a job (not indexed: YAGNI); its `concat` + per row stays. + +### Remaining (to report, not fixed here) + +- ITL-923 item 3: a pending row (written with `skip_duplicates=False`) whose id is already + stored is still dropped by the commit (MERGE `when_not_matched_insert_all`, and now the + append path decides the same way); the in-memory store instead overwrites. Unifying the + semantics is a behaviour change: out of scope. +- `flush()` keeps attempting later keys after a failed one (#287), so a failed result-table + commit followed by a successful pipeline commit can store pipeline records whose results + are still pending; the job's best-effort retry usually commits them. +- Concurrent writers: the append decision uses the version loaded at flush time; another + writer committing between that load and the append could create a duplicate id (MERGE + would not). Single-writer per store is assumed (as for the existing caches). + +## Tests (TDD order; each must fail first where it encodes new behaviour) + +- [ ] T1 **Delta scaling, sync + async:** `test_job_store_scaling.py` parametrized over the + store (in-memory, Delta); Arrow bytes per row at 150 vs 600 rows flat (`< 2x`). +- [ ] T2 **Commit counts:** 250 rows, `commit_every_rows=100`: result and pipeline tables + each have exactly 3 versions (`DeltaTable(path).history()`), sync and async; history + operations are `WRITE`/`CREATE`... not `MERGE` (append path). +- [ ] T3 **Option validation:** `commit_every_rows=0`, `-1`, `True`, `1.5` raise + `ValueError`/`TypeError` before anything runs. +- [ ] T4 **Outside a job unchanged:** `job.nodes["dbl"].run()` after compile (no `job.run`) + and a `CachedFunctionPod` used directly on a Delta store: one result-table commit per row. +- [ ] T5 **Hard crash mid-run** (subprocess, `os._exit(1)` from the function at row 234 of + 250, `commit_every_rows=100`): reopen: exactly 200 result rows and 200 pipeline records; + rerun without the crash: the function runs exactly 50 times (side file); final tables equal + an uninterrupted run's on stable columns. +- [ ] T6 **Crash between result and pipeline commits** (subprocess; a store wrapper around + `DeltaTableDatabase` whose `flush` on the pipeline view calls `os._exit` on its first + call after the result table committed): reopen: results stored, pipeline records absent; + rerun: zero function calls, pipeline records written, final tables equal an uninterrupted + run's. Plus a cache-hit-first batch case. +- [ ] T7 **Commit failure, sync + async, both error policies:** a store wrapper whose + `flush` raises on its 2nd call: `StoreWriteError`, `flush=True`, `node_label == "dbl"`, + `tag is None`; the first 100 rows committed and readable from a fresh store. +- [ ] T8 **Ephemeral store flushed on failure:** a job with an `is_result_ephemeral=True` + node whose later node raises under `fail_fast`: the ephemeral store's results are committed + (fresh view), the run's own error is the one raised. +- [ ] T9 **Append vs MERGE:** fresh store commits use append; a store another writer + committed to between batches still stores no duplicate id and the tables equal. +- [ ] T10 **Lookup equivalence, Delta:** #304's equivalence suite parametrized over + `DeltaTableDatabase` (committed + pending, many commits, scoped views sharing snapshots, + unindexable lookups, a column missing on one side) vs a filter over `get_all_records`. +- [ ] T11 **ITL-923:** existing #293 regressions stay green; ephemeral re-run on Delta (sync + and async): 10 pipeline records, node reads 5 rows — fix if inside this change, else report. +- [ ] T12 **side_effects:** an unreadable completion table raises `StoreReadError` with + `node_label` and `tag` set. +- [ ] T13 **compute_base_entry_id:** called once per computed row (spy on the method in a + job run) — the one test that touches a non-public entry point, commented. +- [ ] T14 #287, #290, #293, #295 tests unchanged and passing; golden hashes unchanged. +- [ ] T15 **Differential vs main** (scratch script, not a test): joined multi-node pipeline + (two sources, `Join`, two pods, 1,200 rows) on Delta, sync and async; every table dumped: + identical paths, schemas, row counts, record ids, entry ids, content hashes. +- [ ] T16 **Timings:** 100 / 400 / 1,000 rows (5,000 after only), Delta sync + async, + before (main numbers from the brief + this base) and after, in the PR body. No 5,000-row + baseline on main (hours): extrapolate and say so. + +## Tasks + +- [x] 0. Rename branch, merge #304 (f84217b2), ask placement (approved: `run()` kwarg). +- [x] 1. Save this plan; WIP commit + push. +- [x] 2. `_row_index.py`: move helpers out of `in_memory_databases.py` (no behaviour change; + #304 tests green). +- [x] 3. T10 Delta equivalence tests (should pass before and after: guard for the refactor). +- [x] 4. Store snapshot + indexed lookups (T1 store-side part fails first: Delta bytes per + row grow); replace id cache; append path (T9). +- [x] 5. Node batching + `run(commit_every_rows=)` (T2, T3, T4 first); commit order; + failure handling (T7, T8); crash tests (T5, T6). +- [x] 6. side_effects context (T12) and `compute_base_entry_id` reuse (T13). +- [x] 7. ITL-923 ephemeral re-run check (T11). +- [x] 8. Docs: `docs/concepts/function-pods.md` (When the store cannot be written: commit + cadence, crash guarantee, commit failure), pipelines page (`commit_every_rows`), + `CHANGELOG.md` [0.3.0], `DESIGN_ISSUES.md` PQ1 update. +- [ ] 9. Differential (T15), timings (T16), full suite on 3.13 / 3.12 / 3.11 one at a time, + ruff, basedpyright. +- [ ] 10. Merge origin/main, push, PR (stack note, Fixes ITL-919, ITL-923 per result), + `orca linear attach`, `orca worktree set --pr`, CodeRabbit rounds, worker_done. + +## Notes / progress log + +- 2026-11-11: plan copied from the part-1 worker's scratchpad and updated with the + developer's decisions; placement of `commit_every_rows` approved (run() keyword). +- Store: snapshot = stored ids (id-only scan through a one-column schema, so logical list + types cannot break it) + stored rows (indexed, `as_large_types`), extended after own + commits from new data files; pending batch compacted + indexed (the unindexed pending + batch cost ~185 KB per lookup). Append only when all ids are new AND the batch has the + stored schema as the writer converts it (`_stores_as_is`), else MERGE. +- Coordinator approvals: three #293 tests changed (snapshot answers warm reads; own commit + tells ids), with boundary tests added (`test_delta_committed_snapshot.py`); four #290 + tests changed to commit-point semantics + a `commit_every_rows=1` test; both lists go in + the PR body ("Tests of #293 changed", "Tests of #290 changed"). +- ITL-930 (get_all_records short after run): fixed by #293 in the stack; regression test + added. ITL-932 (async run hangs when a node fails with > buffer rows in flight): + pre-existing, fixed in its own commit (channels.close never blocks). ITL-923 async + ephemeral rerun: fixed (route_inputs treated cached EmptyData as a hit). +- Found, not fixed: AsyncPipelineOrchestrator ignores error_policy (documented as ITL-879). +- Timings after (ms/row, Delta, 100/400/1000/5000): sync 2.59/1.28/1.13/1.18, async + 2.36/1.21/1.09/1.16. Main (part-1 worker): sync 80/198/509, async 69/164/359. + +--- + +# Appendix: the reviewed plan (part-1 worker, with three amendments), for reference + +The original plan used `ROWS_PER_COMMIT = 500`, a "drop known ids then append the rest" +commit, and per-key flush order; the decisions above override those. Its user-facing table +otherwise still holds with 100 in place of 500: + +| | Today (`main`) | D2 | +|---|---|---| +| When a job node's rows are committed | after every row (one Delta commit per row per table) | every `commit_every_rows` (100) rows, at node end, before a failed run raises, at job end | +| Hard crash mid-node | loses at most the row in flight | loses at most the last 99 rows (and their pipeline records) of the running node; recomputed on rerun. A crash between a commit's result-table and pipeline-table writes leaves those results stored without pipeline records: joined reads miss them until a rerun, which finds them in the result cache and writes the pipeline records with no recomputation | +| A row whose record cannot be built/added | `StoreWriteError` naming table, node, row tag | unchanged | +| A commit that fails | `StoreWriteError(flush=True)` naming the store (raised at that row) | `StoreWriteError(flush=True)` naming the store and the node (no row); raised at the commit point | +| Rows committed before a failure | kept | kept (every earlier batch); the failed batch stays pending and is retried once by the run's best-effort flush | +| Files per table | one parquet file per row | one per commit | +| Outside a job | flush per row | unchanged | +| Stored rows, ids, hashes, paths, schemas | — | unchanged (differential check) | + +Per-row reads on `main` with the per-row flush disabled: 7.5 / 12.6 / 34 ms per row at +100 / 400 / 1000 rows — still growing — because `get_records_with_column_value` reads the +whole Delta table per call (no predicate pushdown, ENG-601), the pending batch is scanned per +lookup, and `_refresh_existing_ids_cache` re-reads the whole table after every commit. Hence +the snapshot design above. diff --git a/superpowers/plans/2026-11-11-itl-919-in-memory-store-linear.md b/superpowers/plans/2026-11-11-itl-919-in-memory-store-linear.md new file mode 100644 index 000000000..7c26bf1af --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-919-in-memory-store-linear.md @@ -0,0 +1,123 @@ +# ITL-919 (part 1): linear PipelineJob runs on InMemoryArrowDatabase — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make every per-row `InMemoryArrowDatabase` operation a `PipelineJob` performs cost +O(rows touched), not O(rows already stored), so job run time is linear in rows. Stored rows, +row order, record ids, hashes, paths, schemas, public API and write timing stay identical. + +**Architecture:** Two internal changes to `src/orcapod/databases/in_memory_databases.py` only: +(1) the committed table is kept at O(log N) chunks by binary-counter compaction on `flush`; +(2) per-record-key, per-column hash indexes over committed rows (value → sorted row +positions), built lazily, extended on append, dropped when a flush overwrites an existing id. +Lookups (`get_records_with_column_value`, the skip-duplicates check, flush's overwrite check) +use the index and filter only candidate rows. + +**Tech Stack:** PyArrow 23, pytest. + +**Scope:** In-memory store only (coordinator decision, November 11, 2026). Delta is deferred +pending the developer's decision on write timing (options D1/D2/D3 in the ask); do NOT touch +`delta_lake_databases.py`, `result_cache.py` flush timing, `function_node.py` or `job.py`. + +--- + +## Cause (STEP 1 findings, main c8efc092) + +Only function nodes are affected (operator nodes write with one `add_records`). Sync and async +orchestrators both run, per row: `ResultCache.lookup` → `store` (+ auto-flush) → +`add_pipeline_record`. Per row, against the result table and the pipeline table: + +1. `ResultCache.lookup` → `get_records_with_column_value(input hash)`: filters the whole table. +2. `add_pipeline_record` → `get_records_with_column_value(base_entry_id)`: filters the whole + table; then `add_record(skip_duplicates=True)` → `_committed_ids` → `to_pylist` of every id. +3. `ResultCache.store` auto-flushes every row; in-memory `flush` does `is_in` over all committed + ids + `filter` + `concat_tables`. Every flush adds a chunk, so after N rows the table has N + chunks and every Arrow op (`filter`, `take`) costs O(N). + +Delta (deferred): one MERGE commit per row per table (grows with table), lookups re-open and +read all files (one per row), existing-id cache refreshed after every flush; pending batch also +scanned per lookup. `get_records_with_column_value` on Delta returns only pending matches when a +pending batch exists (ignores committed rows) — repro requested by coordinator. + +Timings on main (sync, seconds per row): in-memory 100/400/1000 = 7.2/20.9/64 ms; +Delta 100/200/400 = 78/104/150 ms; standalone pod 0.09 ms flat. + +## Files + +- Modify: `src/orcapod/databases/in_memory_databases.py` +- Create: `tests/test_databases/test_in_memory_database_scaling.py` (DB-level + job-level + scaling tests, deterministic: Arrow bytes allocated per row through a counting wrapper) +- Create: `tests/test_databases/test_in_memory_database_lookup_equivalence.py` (index results + equal a plain filter: multi-column queries, overwrite path, pending + committed, types not + indexable, str vs bytes values) +- Modify: `CHANGELOG.md` (`[0.3.0] — unreleased`), `DESIGN_ISSUES.md` (one appended entry) + +## Task 1: failing scaling tests + +- [ ] Counting wrapper: a delegating `ArrowDatabaseProtocol` wrapper whose `at()` returns a + wrapped view and whose methods add `pa.default_memory_pool().total_bytes_allocated()` deltas + to a shared counter. Deterministic (bytes requested, not time). +- [ ] DB-level test: add N single-row records one by one, each followed by + `get_records_with_column_value` on an indexed binary column, `add_record(skip_duplicates=True)` + and `flush()` (the job's per-row pattern). Arrow bytes per row for the last 100 rows at + N=1600 must be < 2x those at N=200 (main: ~8x, grows linearly). +- [ ] Job-level test: `PipelineJob` with `InMemoryArrowDatabase` wrapped by the counter, one + function pod; run 200 rows and 800 rows; store bytes per row ratio < 2 (main ~4). + Run with both sync and async orchestrators. +- [ ] Run, confirm both fail on main. Commit (test only) as WIP. + +## Task 2: compaction on flush + +```python +def _compact_tail(table: pa.Table) -> pa.Table: + """Merge trailing record batches while the earlier one is not larger (binary counter).""" + batches = [b for b in table.to_batches() if b.num_rows > 0] + while len(batches) >= 2 and batches[-2].num_rows <= batches[-1].num_rows: + merged = pa.Table.from_batches(batches[-2:]).combine_chunks().to_batches() + batches[-2:] = merged + return pa.Table.from_batches(batches, schema=table.schema) +``` + +In `flush`: append path `pa.concat_tables([committed, pending])` (keeps today's schema errors) +then `_compact_tail`; overwrite path (rare) filters then `combine_chunks()`. + +## Task 3: column indexes + +- `_indexes: dict[str, _KeyIndex]` shared across `at()` views (`_shared_indexes`). +- `_KeyIndex(table, columns: dict[str, dict[Any, list[int]]])`; valid only while + `index.table is self._tables[key]` (rebuilt otherwise). +- Indexable: Arrow binary/large_binary/string/large_string/integer/bool columns queried with a + value of the matching Python type (`bytes`, `str`, `int` not `bool`, `bool`). Anything else + falls back to today's filter. +- `get_records_with_column_value`: pick the first indexable pair, `take` sorted candidate + positions from committed, `filter` the full expression on them; pending part filtered as + today; concat parts exactly as `_combined_table` did (same inclusion rule, same errors). +- `_filter_existing_records` and flush overwrite detection use membership in the + `RECORD_ID_COLUMN` index instead of building a full id set. +- Flush append path extends existing column indexes with the pending rows' values at + `old_len + i`. + +## Task 4: verification + +- [ ] New tests pass; `uv run pytest tests/test_databases tests/test_pipeline tests/test_core -q`. +- [ ] Differential check vs main (exported copy of main's `src` on `PYTHONPATH`): multi-node + pipeline with a join (two sources → join → function pod → function pod), in-memory store, + sync and async: compare every stored table (`list_sources` + `get_all_records`) — same + schema, same rows (excluding run-dependent columns: result record UUIDs, timestamps, which + are random/now() on main too), same pipeline record ids, content hashes, entry ids. +- [ ] After timings 100/400/1000/5000 for mem sync/async (+ Delta unchanged numbers). +- [ ] Golden hashes, full suite, ruff, basedpyright. +- [ ] CHANGELOG + DESIGN_ISSUES entry; PR "Fixes"? — No: this PR is part 1 of ITL-919, the + Delta part follows; use "Part of ITL-919" unless the coordinator says otherwise (ask). + +## Notes / progress log + +- 2026-11-11: cause established, coordinator approved in-memory fix; Delta pending developer. +- Main 5000-row timings running in background (scratchpad before_mem.txt / before_delta.txt). +- 2026-11-11: in-memory fix implemented (commit "perf(databases): ..."). Scaling tests pass; + equivalence tests pass on main and on the fix; differential check vs main identical + (scratchpad diff/ scripts: dump_store.py + compare.py). Lookup uses zero-copy row slices + (``take`` on a multi-chunk table copies every chunk) and skips the Arrow filter when every + pair is indexed (acero filter has ~0.35 ms fixed overhead). +- Delta repro of pending-shadow bug reported to coordinator (msg_01ac95ae10d6). +- PR wording: part 1 of ITL-919 (Delta follows) — confirm Fixes vs Part of with coordinator. diff --git a/superpowers/specs/itl-892-exit-hang/FINDINGS.md b/superpowers/specs/itl-892-exit-hang/FINDINGS.md new file mode 100644 index 000000000..744a5f4f8 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/FINDINGS.md @@ -0,0 +1,508 @@ +# ITL-892: process does not exit after reading a Delta table + +November 10, 2026. **Outcome:** the coordinator approved option (c) plus the (d) note. It is +implemented in the same PR (`storage_utils.delta_to_pyarrow_dataset`, two new test modules, +`docs/troubleshooting.md`). The investigation itself changed no `src/` file or `uv.lock`. Scripts, raw results +and native stacks are in this directory; the tables are produced by `summarize.py`. + +## 1. Summary + +- **Reproduced on macOS and Linux, Python 3.11, 3.12 and 3.13, with and without Orcapod.** + A process that scans a Delta table with deltalake's default pyarrow reader can fail at + interpreter exit: it hangs forever, or on Linux with Python 3.11/3.12 it may instead abort + with SIGABRT ("terminate called without an active exception"). The script's work and output + are complete in every failing run. +- **Rate**: one write plus one scan, then exit: about 57% of runs on macOS arm64, 5–38% on + Linux x86_64 (GitHub runners), depending on the Python version and the read call. The ITL-864 script (scan, + four merges with a scan after each) fails 12% on macOS. Through Orcapod, a fresh + `DeltaTableDatabase.get_all_records` fails 53–77% on macOS and 8–23% on Linux, and + `DeltaTableSource` 4–10%. A `PipelineJob` run fails 0–4%. +- **Cause (upstream, not Orcapod)**: `DeltaTable.to_pyarrow_dataset()` (and `to_pyarrow_table()`, + which calls it) reads through `pyarrow.fs.PyFileSystem(DeltaStorageHandler)`, a filesystem + implemented in Python. Arrow's async scan finishes tearing itself down on a thread-pool worker + after `to_table()` returns. When that teardown releases the last reference to a Python-backed + buffer (`PyBuffer`) while the interpreter is finalizing, the worker calls `PyGILState_Ensure` + and CPython stops the thread. It hangs it on 3.13 and exits it on 3.11/3.12, where glibc's + forced unwind then aborts the process. Then `ThreadPool::~ThreadPool`, run from + `__cxa_finalize`, waits forever for a worker that will never finish. This is + [apache/arrow#50482](https://github.com/apache/arrow/issues/50482) and + [apache/arrow#54536](https://github.com/apache/arrow/issues/54536), both open. The fix PR + [apache/arrow#54467](https://github.com/apache/arrow/pull/54467) is open and not merged. No + matching delta-rs issue exists. +- **No dependency version avoids it**: every pyarrow 20.0.0–26.0.1 × deltalake 1.0.2/1.6.6 + combination fails, at 36–68% on macOS. +- **Recommended mitigation, option (c)**: Orcapod scans local Delta tables through a native + pyarrow filesystem (`to_pyarrow_dataset(filesystem=SubTreeFileSystem(table_path, + LocalFileSystem()))`). This takes the failure rate from up to 77% to 0/100 in every Orcapod mode + tested, on macOS and on Linux 3.11/3.12/3.13. The result is schema- and value-identical, including on partitioned + tables with escaped partition values. The full suite passes with it (6834 passed, 96 skipped, + 16 xfailed). No API, hash, stored-schema or dependency change. Cloud stores (`s3://` etc.) + keep deltalake's default reader and stay exposed until Arrow ships a fix. Document that case + (option d). + +## 2. Mechanism and evidence + +Native stacks of hung processes (`samples/`, captured with macOS `sample` and with `gdb` on +Linux): + +- Main thread, every platform: + `exit → __cxa_finalize_ranges → std::shared_ptr::~shared_ptr → + ThreadPool::~ThreadPool → ThreadPool::Shutdown(bool) → condition_variable::wait`. +- Python 3.13, macOS (`samples/dataset_scan_py3.13_pa23.0.1_dl1.0.2.txt`) and Linux + (`linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt`): one pool worker is in + `… MergedGenerator::State::~State → parquet FileReaderImpl::~FileReaderImpl → + SerializedFile::~SerializedFile → ReadRangeCache::~ReadRangeCache → Future result → + arrow::py::PyBuffer::~PyBuffer → PyGILState_Ensure → _PyThreadState_Attach → take_gil → + PyThread_hang_thread`. The `ReadRangeCache` exists because deltalake sets + `ParquetFragmentScanOptions(pre_buffer=True)`. The cached buffers are Python `bytes` returned + by `DeltaStorageHandler`. +- Python 3.11/3.12: the worker is gone. CPython exits a thread that asks for the GIL during + finalization, so it never deregisters from the pool. On macOS the process hangs: the + `samples/orcapod_plain_read_hang_manual.txt` sample has no Arrow worker left and the main + thread still waits in `Shutdown`. On Linux the thread exit is a forced unwind through a + `noexcept` destructor, so `std::terminate` aborts the process (rc −6) in some runs, and others + hang. +- `faulthandler` does not help: the hang is below Python. + +Consistent with this: + +| condition (macOS, py3.12, locked versions, 100 runs) | failed | +|---|---:| +| `write_deltalake` only, or `merge` only, or `to_pyarrow_dataset()` without scanning | 0% | +| pyarrow parquet write + `pyarrow.dataset` scan with deltalake imported | 0% | +| `to_pyarrow_table()` / `to_pyarrow_dataset().to_table()` | 57% / 57% | +| same scan, native filesystem (`filesystem=SubTreeFileSystem(...)`) | 0% | +| same scan, `QueryBuilder` (DataFusion), `pq.read_table(dt.file_uris())`, `pl.read_delta` | 0% each | +| same scan in a `fork` child (exits with `os._exit`, no finalization) | 0% | +| same scan in a `spawn` child (child finalizes normally) | 90% (the parent then waits forever in `join`) | + +## 3. Characterisation + +Full tables: §10. Headline cells, 100 runs each. + +**Operation.** Only scanning (`to_table`, `to_pyarrow_table`) triggers it. Writing and merging +do not. More scans earlier in the process do not raise the rate (ITL-864 script with 4, 12 or +24 merge rounds: 12%, 0%, 0%). What matters is a scan shortly before exit. Orcapod's +`DeltaTableDatabase` scans in `get_all_records`, in `_read_delta_table` (used by +`get_record_by_id`, `get_records_by_ids`, `get_records_with_column_value`), and in +`_refresh_existing_ids_cache` (writes with `skip_duplicates`). `DeltaTableSource.__init__` also +scans. `observability_reader` uses `pl.read_delta`, which is safe (0%). + +**Thread pools and environment** (macOS, one-scan cell, baseline 57%): `pa.set_cpu_count(1)` +33%; `pa.set_io_thread_count(1)` 53%; `OMP_NUM_THREADS=1` 45%; +`RAYON_NUM_THREADS=1 TOKIO_WORKER_THREADS=1` 49%. `to_table(use_threads=False)` gives 0% on +macOS but 8–20% on Linux, so it is not a fix. No thread setting removes it. + +**Two Arrow runtimes.** deltalake's Rust arrow alone (writes, merges, `QueryBuilder`) never +triggers it, and pyarrow alone (`pyarrow_only`) never does. Only pyarrow's C++ pool combined +with the Python-implemented filesystem does. + +**Polars as a third runtime.** Importing polars after pyarrow gives 48%, importing it first 27%. +Polars changes timing, not the outcome. + +**Python version** (macOS): one-scan cell 3.11 57%, 3.12 58%, 3.13 65%; Orcapod plain-read 3.11 77%, 3.12 53%, 3.13 76%. The 3.11/3.13 Orcapod cells ran in an isolated environment with a 30 s deadline and 4 parallel runs; the 3.12 cell ran in the project environment with 6 s and 10. Orcapod `PipelineJob` sync and async: 0% on all three. Linux: see §10. On 3.13 failures are +always hangs. On 3.11/3.12 Linux they are hangs or SIGABRT. + +**Load.** Running 1, 4 or 10 children in parallel gives 3%, 5%, 10% on the ITL-864 script. Load +raises the rate a little; it is not the cause. + +**Platforms.** macOS arm64: measured here. Linux x86_64: measured on GitHub `ubuntu-latest` +through a temporary workflow on this branch (`.github/workflows/itl-892-exit-hang-probe.yml`, +not a required check, since deleted; GitHub Actions runs 42969989607 and 42972298798). Docker is not installed on +this machine. + +## 4. Versions and upstream + +`uv run --no-project --with pyarrow==X --with deltalake==Y` (uv.lock untouched), macOS, +py3.12, one-scan cell: + +| pyarrow \ deltalake | 1.0.2 (locked) | 1.6.6 (latest) | +|---|---:|---:| +| 20.0.0 (project floor) | 67% | 59% | +| 22.0.0 | 55% | 55% | +| 23.0.1 (locked) | 50–58% | 61% | +| 24.0.0 | 53% | 58% | +| 25.0.1 | 36% | 39% | +| 26.0.1 (latest) | 56% | 68% | + +Linux, py3.11/3.12/3.13: pyarrow 22.0.0 + deltalake 1.0.2 fails 6–14%; 26.0.1 + 1.6.6 fails +2–5%. Within what the project allows (`pyarrow>=20`, `deltalake>=1.0.2`) **there is no +combination that does not fail**, so there is no "newest good" version. The oldest failing +combination is the project floor. Upstream reports it on pyarrow 22–26. deltalake 1.6.6 still +defaults to `PyFileSystem(DeltaStorageHandler)` with `pre_buffer=True`. + +Upstream: + +- [apache/arrow#50482](https://github.com/apache/arrow/issues/50482) (open, July 2026): + interpreter hangs at exit after `pq.read_table(python_file_object)`, a thread-pool teardown + deadlock. Reported on pyarrow 22–25, Linux and macOS. +- [apache/arrow#54536](https://github.com/apache/arrow/issues/54536) (open, October 2026): abort + at exit, `~PyReadableFile` takes the GIL on a pool worker during finalization; Linux, + pyarrow 25–26, SIGABRT with the same message we see. +- [apache/arrow#54467](https://github.com/apache/arrow/pull/54467) (open PR, not merged): an + `atexit` callback that releases the GIL and waits for the CPU pool to go idle. +- delta-rs: no matching issue (searched "hang", "hangs", "deadlock", "exit", + "interpreter shutdown", "PyFileSystem"). Proposed: a delta-rs issue pointing at the Arrow + issues, suggesting a native object-store filesystem for local paths. Not filed: this run + does not post to GitHub. + +## 5. Options + +Failure rates are over 100 runs. "Orcapod" cells use `repro_orcapod.py`, which prototypes each +option by wrapping deltalake (`--patch`), so `src/` is unchanged. + +| option | macOS | Linux 3.11 / 3.12 / 3.13 | cost | +|---|---|---|---| +| none (locked) | Orcapod plain-read 53%, source-read 8%, pipeline 0% | two rounds: plain-read 9–12 / 8–19 / 15–23%, source-read 4–8 / 5–10 / 8–10%, pipeline-sync 0 / 0–4 / 0–1% | — | +| (a) pin or bump pyarrow/deltalake | 36–68% for every combination | 2–14% for every tried combination | none possible today. A future pyarrow with #54467 would fix it: bump `uv.lock` then and re-run golden hashes and the suite | +| (b1) `atexit`: `gc.collect()` | Orcapod plain-read 27% | plain-read 21 / 12 / 17%, source-read 5 / 6 / 6%, pipeline-sync 1 / 1 / 1% | trivial, but does not fix it | +| (b2) drop references + `gc.collect()` before exit | 2% (pure) | 0 / 0 / 0% (pure) | needs user code; not a fix on macOS | +| (b3) `atexit`: release the GIL for 50 ms (`time.sleep(0.05)`) | pure 0%; Orcapod 0% in all four modes | pure 0 / 0 / 0% (50 ms and 500 ms); Orcapod 0% in all three modes on each version | +50 ms on every exit of a process that imported Orcapod; timing-based (a busier machine may need longer); a global side effect of importing a library | +| (b4) `os._exit` after flushing | 0% | — | skips `atexit`, buffered I/O and other libraries' cleanup; only acceptable inside our own experiment harnesses, not in the library | +| **(c) native filesystem for local tables** | **0% in all four Orcapod modes; pure 0% on py3.11/3.12/3.13** | **pure 0 / 0 / 0%; Orcapod 0% in all three modes on each version** | ~20 lines in `delta_lake_databases.py` (one helper used by the 4 scan sites) and `delta_table_source.py`. Byte-identical results (`check_native_fs_equivalence.py`, partition edge cases). Full suite passes under the patch. Cloud URIs keep the default path | +| (c') `to_table(use_threads=False)` | 0% | 8–20% (pure) | slower scans; not a fix on Linux | +| (c'') read via `QueryBuilder` / polars / `pq.read_table(file_uris())` | 0% (pure) | — | larger change, and a different reader with its own type mapping (large types, nullability) to verify against the type-strictness rules; no benefit over (c) | +| (d) documentation only | unchanged | unchanged | a troubleshooting entry: "if a script that used a Delta store does not exit, upgrade pyarrow when #54467 ships; meanwhile end the script with `os._exit(0)` after flushing". There is no troubleshooting page on `main` yet | + +Effect on the locked dependencies, golden hashes and the suite: + +- (a) Nothing to lock. A future bump would change `uv.lock` only. +- (b) and (c) change no dependency and no stored value. Option (c) changes the read path only. + Hashes are computed from the tables, which are identical. The suite run with the (c) patch + (`native_fs_pytest_plugin.py`) gave 6834 passed, 96 skipped, 16 xfailed, no failures. The + golden hash tests are part of that run. + +## 6. Recommendation + +Ship **(c)** for v0.3, plus a short **(d)** note for cloud stores: + +1. In `DeltaTableDatabase`, route the four scan sites (`get_all_records`, `_read_delta_table`, + `_refresh_existing_ids_cache`, and through them every read method) through one helper that + passes `filesystem=SubTreeFileSystem(, LocalFileSystem())` when the store is + local (`not self._is_cloud`). Do the same in `DeltaTableSource`, which already requires a + local path. +2. Keep deltalake's default reader for cloud URIs. A native `S3FileSystem` has its own exit + deadlock ([apache/arrow#50188](https://github.com/apache/arrow/issues/50188)), and the + storage-option mapping is not one-to-one. Document the remaining exposure. +3. Revisit when pyarrow ships #54467. Bump the floor then, and decide whether the helper can + go. + +Why (c) over (b3). (b3) also measured 0% everywhere, and it is the only option that also +covers cloud stores. But it does not remove the cause. It only gives the pool time to finish +before finalization starts. The 50 ms is a guess that held on these machines; a loaded machine +or a large scan could outlast it, and the failure would come back silently. It would also run +in every process that imported Orcapod. + +(c) removes the Python objects from the scan, so there is nothing for a worker to release +late. Its results are byte-identical by check. A possible follow-up, not proposed for v0.3 +unless the developer wants cloud covered now: register the (b3) hook lazily, only the first +time a cloud-store scan runs, so local users never pay for it. + +## 7. Regression check + +`regression_check_proto.py` (prototype, not in `tests/` yet): for each of two modes +(`DeltaTableDatabase` write, flush and fresh-handle read; `DeltaTableSource` read), it starts 40 +fresh interpreters, 4 at a time. It fails if any run does not exit within 60 s or exits +non-zero, which covers both the hang and the Linux abort. `ITL892_PATCH` lets the same check +validate a prototype. + +| | macOS | Linux 3.11 / 3.12 / 3.13 | +|---|---|---| +| unpatched (today's `main`) | fails both cases (wall time 497 s, bounded by the 60 s deadlines) | fails both cases on each version (162 / 198 / 511 s) | +| native-fs patch | passes, 10 s | passes on each version, 39 / 50 / 50 s | + +Detection. Measured per-run failure rates without a fix: macOS 53–77% (plain-read) and 8% +(source-read); Linux 8–23% and 4–10%. With 40 runs per case, both cases miss a regression with +probability below 1e-12 on macOS and below 1% per Linux Python job, which is about 1e-6 across +CI's three Python jobs. + +False failures. A clean run takes 1–2 s, so the 60 s deadline leaves a wide margin. Every +patched run passed (one macOS and three Linux runs of the check, plus thousands of patched +harness runs). + +Recommendation: add it to `tests/` as an ordinary test that runs in CI, for example +`tests/test_databases/test_delta_exit.py`, with the child script inline and only the public API. +When it passes it costs about 40–50 s per Python job on GitHub runners (10 s on this Mac) and +needs no infrastructure. If that time is unwelcome, halve `RUNS` to 20: a miss is then about 8% +likely per job and 0.05% across the three jobs. It is reliable enough for CI. An opt-in marker +would be justified only by its run time, not by flakiness. + +## 8. Not verified + +- Linux arm64 and Windows. +- Cloud stores (S3/MinIO): not run. The analysis says they stay exposed with (c), since they + keep the Python filesystem. +- Jupyter kernels: not affected in the same way, since the kernel does not exit. +- The mechanism on 3.11/3.12 macOS is inferred: the worker had already vanished in every + sample, so its last frame was not captured. The 3.13 stacks show it directly. + +## 8a. Out-of-scope findings + +- `delta_lake_databases.py:868`: `return dataset.to_table()(filters=filters)` calls the + returned `Table`, which raises `TypeError` whenever `_read_delta_table` gets a filter + operator other than `=` or `in`. + +## 9. Files + +| file | purpose | +|---|---| +| `harness.py` | runs a command N times with a deadline; counts hangs and exit codes; samples the first hung child's native stack | +| `repro_pure.py` | pyarrow + deltalake only; `--op`, `--read-via`, thread, polars, mp and mitigation switches | +| `repro_orcapod.py` | the same through Orcapod's public API; `--patch` prototypes options (b) and (c) | +| `run_matrix.sh` | the macOS matrix groups | +| `check_native_fs_equivalence.py` | option (c) returns identical schemas and values | +| `native_fs_pytest_plugin.py` | runs the suite with option (c) applied | +| `regression_check_proto.py` | proposed regression check | +| `summarize.py` | renders the tables below | +| `results.jsonl` | macOS raw results (one line per cell) | +| `samples/` | representative macOS stacks: py3.12 (`op=dataset_scan_py3.12_locked.txt`), py3.13 with the worker in `PyGILState_Ensure` (`dataset_scan_py3.13_pa23.0.1_dl1.0.2.txt`), Orcapod with no worker left (`orcapod_plain_read_hang_manual.txt`). The other stacks named in `results.jsonl` are not committed (`*.txt` is gitignored; about 4 MB) | +| `linux/round*/` | Linux raw results from the temporary CI probe, with two py3.13 `gdb` stacks. On 3.11/3.12, `gdb` could not attach because the child aborted first | + +## 10. All results + +Each cell is one `harness.py` run (`results.jsonl`, `linux/*/results.jsonl`). "hung" means the run missed its deadline. In almost every cell it had already printed `done`; the exception is the `spawn` cell, where the parent waits on a hung child. Early exploratory cells (`baseline`, `load`, `amplify`, `itl864 ...`) used the ITL-864 script, which fails less often than the one-scan cell (`dataset_scan`) used afterwards. + +#### macOS arm64 (this machine) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| baseline itl864 py3.12 locked | 40 | 2 | 0 | 0 | **5%** | +| load j=1 itl864 py3.12 locked | 60 | 2 | 0 | 0 | **3%** | +| load j=4 itl864 py3.12 locked | 60 | 3 | 0 | 0 | **5%** | +| amplify rounds=12 itl864 py3.12 locked | 40 | 0 | 0 | 0 | **0%** | +| amplify rounds=24 itl864 py3.12 locked | 40 | 0 | 0 | 0 | **0%** | +| load j=10 itl864 py3.12 locked | 60 | 6 | 0 | 0 | **10%** | +| op=pyarrow_only py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=write py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=merge py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=to_dataset py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=to_table py3.12 locked | 100 | 57 | 0 | 0 | **57%** | +| op=dataset_scan py3.12 locked | 100 | 57 | 0 | 0 | **57%** | +| op=itl864 py3.12 locked | 100 | 12 | 0 | 0 | **12%** | +| itl864 read-via=native_fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=no_threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=querybuilder py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=files py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=polars py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 cpu_threads=1 | 100 | 6 | 0 | 0 | **6%** | +| dataset_scan read-via=native_fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=no_threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=querybuilder py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=files py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=polars py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan cpu_threads=1 | 100 | 33 | 0 | 0 | **33%** | +| dataset_scan io_threads=1 | 100 | 53 | 0 | 0 | **53%** | +| dataset_scan OMP_NUM_THREADS=1 | 100 | 45 | 0 | 0 | **45%** | +| dataset_scan RAYON/TOKIO threads=1 | 100 | 49 | 0 | 0 | **49%** | +| dataset_scan import polars after pyarrow | 100 | 48 | 0 | 0 | **48%** | +| dataset_scan import polars first | 100 | 27 | 0 | 0 | **27%** | +| dataset_scan in fork child | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan in spawn child | 100 | 90 | 0 | 0 | **90%** | +| dataset_scan mitigation=del | 100 | 2 | 0 | 0 | **2%** | +| dataset_scan mitigation=atexit_del | 100 | 1 | 0 | 0 | **1%** | +| dataset_scan mitigation=os_exit | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan mitigation=atexit_sleep | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan mitigation=atexit_sleep_long | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read py3.12 locked | 100 | 53 | 0 | 0 | **53%** | +| orcapod source-read py3.12 locked | 100 | 8 | 0 | 0 | **8%** | +| orcapod pipeline-sync py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod rerun py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.11 pa23.0.1 dl1.0.2 | 100 | 57 | 0 | 0 | **57%** | +| dataset_scan native_fs py3.11 pa23.0.1 dl1.0.2 | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.12 pa23.0.1 dl1.0.2 | 100 | 50 | 0 | 0 | **50%** | +| dataset_scan native_fs py3.12 pa23.0.1 dl1.0.2 | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.13 pa23.0.1 dl1.0.2 | 100 | 65 | 0 | 0 | **65%** | +| dataset_scan native_fs py3.13 pa23.0.1 dl1.0.2 | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.12 pa20.0.0 dl1.6.6 | 100 | 59 | 0 | 0 | **59%** | +| dataset_scan py3.12 pa22.0.0 dl1.0.2 | 100 | 55 | 0 | 0 | **55%** | +| dataset_scan py3.12 pa22.0.0 dl1.6.6 | 100 | 55 | 0 | 0 | **55%** | +| dataset_scan py3.12 pa23.0.1 dl1.6.6 | 100 | 61 | 0 | 0 | **61%** | +| dataset_scan py3.12 pa24.0.0 dl1.0.2 | 100 | 53 | 0 | 0 | **53%** | +| dataset_scan py3.12 pa24.0.0 dl1.6.6 | 100 | 58 | 0 | 0 | **58%** | +| dataset_scan py3.12 pa25.0.1 dl1.0.2 | 100 | 36 | 0 | 0 | **36%** | +| dataset_scan py3.12 pa25.0.1 dl1.6.6 | 100 | 39 | 0 | 0 | **39%** | +| dataset_scan py3.12 pa26.0.1 dl1.0.2 | 100 | 56 | 0 | 0 | **56%** | +| dataset_scan py3.12 pa26.0.1 dl1.6.6 | 100 | 68 | 0 | 0 | **68%** | +| dataset_scan py3.12 pa20.0.0 dl1.0.2 | 100 | 67 | 0 | 0 | **67%** | +| orcapod plain-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod source-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-sync patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod source-read patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-sync patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read patch=atexit-gc py3.12 locked | 100 | 27 | 0 | 0 | **27%** | +| orcapod source-read patch=atexit-gc py3.12 locked | 100 | 5 | 0 | 0 | **5%** | +| orcapod pipeline-sync patch=atexit-gc py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=atexit-gc py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod source-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-sync patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read py3.11 locked | 100 | 77 | 0 | 0 | **77%** | +| orcapod pipeline-sync py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read py3.13 locked | 100 | 76 | 0 | 0 | **76%** | +| orcapod pipeline-sync py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async py3.13 locked | 100 | 0 | 0 | 0 | **0%** | + +#### Linux x86_64, GitHub ubuntu-latest (round1-py3.11) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux op=dataset_scan py3.11 locked | 100 | 8 | 13 | 0 | **21%** | +| linux op=to_table py3.11 locked | 100 | 4 | 19 | 0 | **23%** | +| linux op=itl864 py3.11 locked | 100 | 8 | 0 | 0 | **8%** | +| linux dataset_scan native_fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan no_threads py3.11 locked | 100 | 11 | 0 | 0 | **11%** | +| linux dataset_scan io_threads=1 py3.11 locked | 100 | 18 | 0 | 0 | **18%** | +| linux dataset_scan mitigation=del py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read py3.11 locked | 100 | 12 | 0 | 0 | **12%** | +| linux orcapod source-read py3.11 locked | 100 | 4 | 0 | 0 | **4%** | +| linux orcapod pipeline-sync py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-async py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux op=dataset_scan py3.11 pa22.0.0 dl1.0.2 | 100 | 6 | 0 | 0 | **6%** | +| linux op=dataset_scan py3.11 pa26.0.1 dl1.6.6 | 100 | 2 | 2 | 0 | **4%** | + +#### Linux x86_64, GitHub ubuntu-latest (round1-py3.12) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux op=dataset_scan py3.12 locked | 100 | 10 | 8 | 0 | **18%** | +| linux op=to_table py3.12 locked | 100 | 3 | 17 | 0 | **20%** | +| linux op=itl864 py3.12 locked | 100 | 8 | 0 | 0 | **8%** | +| linux dataset_scan native_fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan no_threads py3.12 locked | 100 | 20 | 1 | 0 | **21%** | +| linux dataset_scan io_threads=1 py3.12 locked | 100 | 18 | 0 | 0 | **18%** | +| linux dataset_scan mitigation=del py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read py3.12 locked | 100 | 8 | 0 | 0 | **8%** | +| linux orcapod source-read py3.12 locked | 100 | 5 | 0 | 0 | **5%** | +| linux orcapod pipeline-sync py3.12 locked | 100 | 4 | 0 | 0 | **4%** | +| linux orcapod pipeline-async py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux op=dataset_scan py3.12 pa22.0.0 dl1.0.2 | 100 | 13 | 0 | 0 | **13%** | +| linux op=dataset_scan py3.12 pa26.0.1 dl1.6.6 | 100 | 2 | 1 | 0 | **3%** | + +#### Linux x86_64, GitHub ubuntu-latest (round1-py3.13) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux op=dataset_scan py3.13 locked | 100 | 5 | 0 | 0 | **5%** | +| linux op=to_table py3.13 locked | 100 | 38 | 0 | 0 | **38%** | +| linux op=itl864 py3.13 locked | 100 | 10 | 0 | 0 | **10%** | +| linux dataset_scan native_fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan no_threads py3.13 locked | 100 | 8 | 0 | 0 | **8%** | +| linux dataset_scan io_threads=1 py3.13 locked | 100 | 32 | 0 | 0 | **32%** | +| linux dataset_scan mitigation=del py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read py3.13 locked | 100 | 15 | 0 | 0 | **15%** | +| linux orcapod source-read py3.13 locked | 100 | 10 | 0 | 0 | **10%** | +| linux orcapod pipeline-sync py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-async py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux op=dataset_scan py3.13 pa22.0.0 dl1.0.2 | 100 | 14 | 0 | 0 | **14%** | +| linux op=dataset_scan py3.13 pa26.0.1 dl1.6.6 | 100 | 5 | 0 | 0 | **5%** | + +#### Linux x86_64, GitHub ubuntu-latest (round2-py3.11) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux dataset_scan mitigation=atexit_sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan mitigation=atexit_sleep_long py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=none py3.11 locked | 100 | 9 | 0 | 0 | **9%** | +| linux orcapod source-read patch=none py3.11 locked | 100 | 8 | 0 | 0 | **8%** | +| linux orcapod pipeline-sync patch=none py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=native-fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=native-fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=native-fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=atexit-sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=atexit-sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-gc py3.11 locked | 100 | 8 | 13 | 0 | **21%** | +| linux orcapod source-read patch=atexit-gc py3.11 locked | 100 | 4 | 1 | 0 | **5%** | +| linux orcapod pipeline-sync patch=atexit-gc py3.11 locked | 100 | 1 | 0 | 0 | **1%** | + +#### Linux x86_64, GitHub ubuntu-latest (round2-py3.12) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux dataset_scan mitigation=atexit_sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan mitigation=atexit_sleep_long py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=none py3.12 locked | 100 | 8 | 11 | 0 | **19%** | +| linux orcapod source-read patch=none py3.12 locked | 100 | 8 | 2 | 0 | **10%** | +| linux orcapod pipeline-sync patch=none py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-gc py3.12 locked | 100 | 11 | 1 | 0 | **12%** | +| linux orcapod source-read patch=atexit-gc py3.12 locked | 100 | 6 | 0 | 0 | **6%** | +| linux orcapod pipeline-sync patch=atexit-gc py3.12 locked | 100 | 1 | 0 | 0 | **1%** | + +#### Linux x86_64, GitHub ubuntu-latest (round2-py3.13) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux dataset_scan mitigation=atexit_sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan mitigation=atexit_sleep_long py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=none py3.13 locked | 100 | 23 | 0 | 0 | **23%** | +| linux orcapod source-read patch=none py3.13 locked | 100 | 8 | 0 | 0 | **8%** | +| linux orcapod pipeline-sync patch=none py3.13 locked | 100 | 1 | 0 | 0 | **1%** | +| linux orcapod plain-read patch=native-fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=native-fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=native-fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=atexit-sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=atexit-sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-gc py3.13 locked | 100 | 17 | 0 | 0 | **17%** | +| linux orcapod source-read patch=atexit-gc py3.13 locked | 100 | 6 | 0 | 0 | **6%** | +| linux orcapod pipeline-sync patch=atexit-gc py3.13 locked | 100 | 1 | 0 | 0 | **1%** | + + +## 11. Proposed delta-rs issue (not filed; for the developer) + +**Title:** `to_pyarrow_dataset()`'s default `PyFileSystem(DeltaStorageHandler)` makes processes hang or abort at interpreter exit + +**Body:** + +> A process that scans a Delta table with `DeltaTable.to_pyarrow_dataset().to_table()` or +> `to_pyarrow_table()` and then exits normally often fails during interpreter shutdown. It +> hangs forever (macOS, any Python; Linux, Python 3.13), or on Linux with Python 3.11/3.12 it +> aborts with `terminate called without an active exception`. All of its work and output are +> done by then. +> +> ```python +> import tempfile, deltalake, pyarrow as pa +> p = tempfile.mkdtemp() +> deltalake.write_deltalake(p, pa.table({"k": [1, 2, 3]})) +> deltalake.DeltaTable(p).to_pyarrow_dataset().to_table() +> print("done") # prints, then the process may never exit +> ``` +> +> Rate over 100 fresh processes: about 57% on macOS arm64 and 5–38% on Linux x86_64. It +> happens on every combination we tried: pyarrow 20.0.0–26.0.1 × deltalake 1.0.2 and 1.6.6, +> Python 3.11–3.13. Writes and merges alone never trigger it. +> +> Native stacks (Python 3.13): the main thread is in `exit → __cxa_finalize → +> arrow::internal::ThreadPool::~ThreadPool → Shutdown`. A pool worker is in +> `parquet::arrow::FileReaderImpl::~FileReaderImpl → ReadRangeCache::~ReadRangeCache → +> arrow::py::PyBuffer::~PyBuffer → PyGILState_Ensure → PyThread_hang_thread`. The cached +> buffers come from `DeltaStorageHandler` (Python `bytes`; `pre_buffer=True`), and Arrow +> releases them on a worker after `to_table()` returns. If that happens during finalization, +> the worker cannot take the GIL. The root bug is in Arrow: apache/arrow#50482, #54536, fix PR +> #54467. +> +> Passing a native filesystem avoids it completely (0/100 on macOS and Linux, identical +> results, partitioned tables included). `table_uri` is percent-encoded (a bare path on 1.0.2 +> and 1.2.1, a `file://` URI on 1.6.6), so it has to be decoded first: +> +> ```python +> uri = dt.table_uri if "://" in dt.table_uri else "file://" + dt.table_uri +> fs, root = pyarrow.fs.FileSystem.from_uri(uri) +> dt.to_pyarrow_dataset(filesystem=pyarrow.fs.SubTreeFileSystem(root, fs)) +> ``` +> +> Suggestion: default to a native pyarrow filesystem for local tables (and possibly the native +> object-store filesystems), or document the issue and the workaround in `to_pyarrow_dataset` +> until Arrow's fix is released. diff --git a/superpowers/specs/itl-892-exit-hang/check_native_fs_equivalence.py b/superpowers/specs/itl-892-exit-hang/check_native_fs_equivalence.py new file mode 100644 index 000000000..068a3befd --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/check_native_fs_equivalence.py @@ -0,0 +1,121 @@ +"""Option (c) check: a native filesystem returns exactly what deltalake's default returns. + +Writes a Delta table with a spread of Arrow types (narrow ints, decimals, timestamps, +nested lists and structs, binary, nulls, non-nullable fields), appends and merges into +it, then reads it with ``to_pyarrow_dataset(as_large_types=True).to_table()`` twice: +once through the default ``PyFileSystem(DeltaStorageHandler)`` and once through +``SubTreeFileSystem(table_path, LocalFileSystem())``. Schemas (with field metadata and +nullability) and values must be identical, with and without a filter expression. +""" + +from __future__ import annotations + +import datetime as dt +import decimal +import tempfile + +import deltalake +import pyarrow as pa +import pyarrow.compute as pc +import pyarrow.fs as pa_fs + + +def make(i: int) -> pa.Table: + """Build 50 rows covering narrow, decimal, timestamp, binary, nested and null columns. + + Args: + i: Batch number, which keeps the ids of each batch distinct. + + Returns: + The table. + """ + schema = pa.schema( + [ + pa.field("id", pa.large_string(), nullable=False), + pa.field("i8", pa.int8()), + pa.field("f32", pa.float32()), + pa.field("dec", pa.decimal128(12, 3)), + pa.field("ts", pa.timestamp("us", tz="UTC")), + pa.field("bin", pa.large_binary()), + pa.field("lst", pa.large_list(pa.int16())), + pa.field( + "st", + pa.struct( + [pa.field("a", pa.int32()), pa.field("b", pa.large_string())] + ), + ), + pa.field("n", pa.large_string()), + ] + ) + rows = 50 + return pa.table( + { + "id": [f"r{i}-{k}" for k in range(rows)], + "i8": [k % 100 for k in range(rows)], + "f32": [k / 3 for k in range(rows)], + "dec": [decimal.Decimal(k * 1001) / 1000 for k in range(rows)], + "ts": [dt.datetime(2026, 1, 1, tzinfo=dt.timezone.utc)] * rows, + "bin": [bytes([k % 256]) * 3 for k in range(rows)], + "lst": [[k, k + 1] if k % 5 else None for k in range(rows)], + "st": [{"a": k, "b": str(k)} for k in range(rows)], + "n": [None] * rows, + }, + schema=schema, + ) + + +def read( + table: deltalake.DeltaTable, native: bool, flt: pc.Expression | None +) -> pa.Table: + """Read ``table`` through the default or the native filesystem, sorted by ``id``. + + Args: + table: The Delta table. + native: Read through a native ``SubTreeFileSystem`` instead of deltalake's. + flt: Optional filter applied by the scan. + + Returns: + The scanned rows. + """ + kwargs = {} + if native: + kwargs["filesystem"] = pa_fs.SubTreeFileSystem( + table.table_uri, pa_fs.LocalFileSystem() + ) + ds = table.to_pyarrow_dataset(as_large_types=True, **kwargs) + return ds.to_table(filter=flt).sort_by("id") + + +def main() -> None: + """Compare default and native reads of one table across versions and filters.""" + path = tempfile.mkdtemp() + deltalake.write_deltalake(path, make(0)) + deltalake.write_deltalake(path, make(1), mode="append") + deltalake.DeltaTable(path).merge( + source=make(2), + predicate="target.id = source.id", + source_alias="source", + target_alias="target", + ).when_not_matched_insert_all().execute() + table = deltalake.DeltaTable(path) + checks = 0 + rows = 0 + for flt in ( + None, + pc.field("i8") > 10, + pc.is_in(pc.field("id"), pa.array(["r1-3", "r2-4"])), + ): + a, b = read(table, False, flt), read(table, True, flt) + assert a.schema.equals(b.schema, check_metadata=True), (a.schema, b.schema) + assert a.equals(b), "values differ" + checks += 1 + rows = a.num_rows + print( + f"files={len(table.file_uris())} last_filter_rows={rows} " + f"checks={checks} identical schema+values" + ) + print("done", flush=True) + + +if __name__ == "__main__": + main() diff --git a/superpowers/specs/itl-892-exit-hang/harness.py b/superpowers/specs/itl-892-exit-hang/harness.py new file mode 100644 index 000000000..a1c215523 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/harness.py @@ -0,0 +1,150 @@ +"""Run a command many times and count the runs that do not exit before a deadline. + +Usage: + harness.py --label LABEL -n 40 -t 20 -j 4 -- + +Each run is a child process in its own session. A run that misses the deadline is +counted as hung; the first hung child of a cell has its native stack sampled with +macOS ``sample`` (or ``eu-stack``/``gdb`` on Linux when present) before it is killed. +A summary line is printed and appended as JSON to ``results.jsonl`` next to this file. +""" + +from __future__ import annotations + +import argparse +import json +import os +import platform +import shutil +import signal +import subprocess +import sys +import time +from collections import Counter +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from typing import Any + +HERE = Path(__file__).resolve().parent +SAMPLES = HERE / "samples" + + +def _sample_stack(pid: int, label: str) -> str | None: + """Capture the native stack of a hung process, returning the file path.""" + SAMPLES.mkdir(exist_ok=True) + out = SAMPLES / f"{label.replace('/', '_').replace(' ', '_')}.txt" + if sys.platform == "darwin" and shutil.which("sample"): + cmd = ["sample", str(pid), "1", "-file", str(out)] + elif shutil.which("gdb"): + cmd = ["gdb", "-p", str(pid), "-batch", "-ex", "thread apply all bt"] + elif shutil.which("eu-stack"): + cmd = ["eu-stack", "-p", str(pid)] + else: + return None + try: + res = subprocess.run(cmd, capture_output=True, text=True, timeout=60) + except Exception: # noqa: BLE001 - diagnostics only + return None + if not out.exists(): + out.write_text(res.stdout + res.stderr) + return str(out.relative_to(HERE)) + + +def _run_one(cmd: list[str], timeout: float, env: dict[str, str]) -> dict[str, Any]: + """Run ``cmd`` once and classify the outcome. + + Args: + cmd: The command to run. + timeout: Seconds before the run counts as hung. + env: Environment for the child. + + Returns: + The outcome. A hung run also carries the live process, for the caller to sample + and kill. + """ + t0 = time.monotonic() + proc = subprocess.Popen( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=True, + env=env, + ) + try: + out, err = proc.communicate(timeout=timeout) + return { + "hung": False, + "rc": proc.returncode, + "done": "done" in out, + "secs": round(time.monotonic() - t0, 2), + "err": err[-400:] if proc.returncode else "", + } + except subprocess.TimeoutExpired: + return {"hung": True, "proc": proc, "secs": timeout} + + +def main() -> int: + """Run one cell of the matrix and append its summary to ``results.jsonl``. + + Returns: + The process exit code. + """ + ap = argparse.ArgumentParser() + ap.add_argument("--label", required=True) + ap.add_argument("-n", type=int, default=40) + ap.add_argument("-t", type=float, default=20.0) + ap.add_argument("-j", type=int, default=4) + ap.add_argument("--no-sample", action="store_true") + ap.add_argument("cmd", nargs=argparse.REMAINDER) + args = ap.parse_args() + cmd = args.cmd[1:] if args.cmd and args.cmd[0] == "--" else args.cmd + env = dict(os.environ) + + results: list[dict[str, Any]] = [] + sampled: str | None = None + with ThreadPoolExecutor(max_workers=args.j) as ex: + for r in ex.map(lambda _: _run_one(cmd, args.t, env), range(args.n)): + if r["hung"]: + proc = r.pop("proc") + if sampled is None and not args.no_sample: + sampled = _sample_stack(proc.pid, args.label) or "unavailable" + try: + os.killpg(proc.pid, signal.SIGKILL) + except (PermissionError, ProcessLookupError): + # macOS refuses killpg once the leader is a zombie; it already exited. + proc.kill() + out, _ = proc.communicate() + r["done"] = "done" in (out or "") + results.append(r) + + hung = sum(r["hung"] for r in results) + rcs = Counter("HUNG" if r["hung"] else r["rc"] for r in results) + hung_after_done = sum(r["hung"] and r["done"] for r in results) + errors = sorted({r["err"] for r in results if r.get("err")})[:2] + summary = { + "label": args.label, + "n": args.n, + "hung": hung, + "hung_after_done": hung_after_done, + "rc": {str(k): v for k, v in rcs.items()}, + "timeout_s": args.t, + "jobs": args.j, + "platform": f"{platform.system()} {platform.machine()}", + "stack": sampled, + "errors": errors, + "cmd": " ".join(cmd), + } + print( + f"{args.label}: hung {hung}/{args.n} (after 'done': {hung_after_done}) rc={dict(rcs)}" + + (f" stack={sampled}" if sampled else "") + + (f" ERR={errors[0][-200:]!r}" if errors else ""), + flush=True, + ) + with open(HERE / "results.jsonl", "a") as fh: + fh.write(json.dumps(summary) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.11/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.11/results.jsonl new file mode 100644 index 000000000..bc48de289 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.11/results.jsonl @@ -0,0 +1,13 @@ +{"label": "linux op=dataset_scan py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 79, "HUNG": 8, "-6": 13}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.11_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=to_table py3.11 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"0": 77, "HUNG": 4, "-6": 19}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=to_table_py3.11_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "linux op=itl864 py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=itl864_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "linux dataset_scan native_fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "linux dataset_scan no_threads py3.11 locked", "n": 100, "hung": 11, "hung_after_done": 11, "rc": {"0": 89, "HUNG": 11}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_no_threads_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "linux dataset_scan io_threads=1 py3.11 locked", "n": 100, "hung": 18, "hung_after_done": 18, "rc": {"0": 82, "HUNG": 18}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_io_threads=1_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "linux dataset_scan mitigation=del py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "linux orcapod plain-read py3.11 locked", "n": 100, "hung": 12, "hung_after_done": 12, "rc": {"0": 88, "HUNG": 12}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "linux orcapod source-read py3.11 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"0": 96, "HUNG": 4}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "linux orcapod pipeline-sync py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "linux orcapod pipeline-async py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "linux op=dataset_scan py3.11 pa22.0.0 dl1.0.2", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.11_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=dataset_scan py3.11 pa26.0.1 dl1.6.6", "n": 100, "hung": 2, "hung_after_done": 2, "rc": {"0": 96, "HUNG": 2, "-6": 2}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.11_pa26.0.1_dl1.6.6.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.12/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.12/results.jsonl new file mode 100644 index 000000000..3de475532 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.12/results.jsonl @@ -0,0 +1,13 @@ +{"label": "linux op=dataset_scan py3.12 locked", "n": 100, "hung": 10, "hung_after_done": 10, "rc": {"0": 82, "HUNG": 10, "-6": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=to_table py3.12 locked", "n": 100, "hung": 3, "hung_after_done": 3, "rc": {"0": 80, "-6": 17, "HUNG": 3}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=to_table_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "linux op=itl864 py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "linux dataset_scan native_fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "linux dataset_scan no_threads py3.12 locked", "n": 100, "hung": 20, "hung_after_done": 20, "rc": {"0": 79, "HUNG": 20, "-6": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_no_threads_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "linux dataset_scan io_threads=1 py3.12 locked", "n": 100, "hung": 18, "hung_after_done": 18, "rc": {"HUNG": 18, "0": 82}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_io_threads=1_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "linux dataset_scan mitigation=del py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "linux orcapod plain-read py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "linux orcapod source-read py3.12 locked", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "linux orcapod pipeline-sync py3.12 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"HUNG": 4, "0": 96}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "linux orcapod pipeline-async py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "linux op=dataset_scan py3.12 pa22.0.0 dl1.0.2", "n": 100, "hung": 13, "hung_after_done": 13, "rc": {"0": 87, "HUNG": 13}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.12_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=dataset_scan py3.12 pa26.0.1 dl1.6.6", "n": 100, "hung": 2, "hung_after_done": 2, "rc": {"0": 97, "HUNG": 2, "-6": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.12_pa26.0.1_dl1.6.6.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/results.jsonl new file mode 100644 index 000000000..9dd3e3a84 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/results.jsonl @@ -0,0 +1,13 @@ +{"label": "linux op=dataset_scan py3.13 locked", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=to_table py3.13 locked", "n": 100, "hung": 38, "hung_after_done": 38, "rc": {"0": 62, "HUNG": 38}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=to_table_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "linux op=itl864 py3.13 locked", "n": 100, "hung": 10, "hung_after_done": 10, "rc": {"0": 90, "HUNG": 10}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=itl864_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "linux dataset_scan native_fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "linux dataset_scan no_threads py3.13 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_no_threads_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "linux dataset_scan io_threads=1 py3.13 locked", "n": 100, "hung": 32, "hung_after_done": 32, "rc": {"0": 68, "HUNG": 32}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_io_threads=1_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "linux dataset_scan mitigation=del py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "linux orcapod plain-read py3.13 locked", "n": 100, "hung": 15, "hung_after_done": 15, "rc": {"0": 85, "HUNG": 15}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "linux orcapod source-read py3.13 locked", "n": 100, "hung": 10, "hung_after_done": 10, "rc": {"0": 90, "HUNG": 10}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "linux orcapod pipeline-sync py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "linux orcapod pipeline-async py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "linux op=dataset_scan py3.13 pa22.0.0 dl1.0.2", "n": 100, "hung": 14, "hung_after_done": 14, "rc": {"0": 86, "HUNG": 14}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.13_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=dataset_scan py3.13 pa26.0.1 dl1.6.6", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.13_pa26.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt new file mode 100644 index 000000000..6db32da99 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt @@ -0,0 +1,348 @@ +[New LWP 4558] +[New LWP 4557] +[New LWP 4556] +[New LWP 4554] +[New LWP 4553] +[New LWP 4552] +[New LWP 4551] +[New LWP 4550] +[New LWP 4549] +[New LWP 4548] +[New LWP 4547] +[New LWP 4546] +[New LWP 4545] +[New LWP 4505] +[New LWP 4501] +[New LWP 4499] +[Thread debugging using libthread_db enabled] +Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". +__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 + +Thread 17 (Thread 0x75909c1ff6c0 (LWP 4499) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263961466408, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x75909c816628, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x75909c816628, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x75909c816628, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x75909c816678, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x75909c816678) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909e497962 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 16 (Thread 0x75908ffff6c0 (LWP 4501) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263923430488, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x75909a3d0458 , expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x75909a3d0458 , expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x75909a3d0458 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x75909a3d0410 , clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x75909a3d0410 ) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000759099230dfb in blas_thread_server () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-56d6093b.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 15 (Thread 0x7590867ff6c0 (LWP 4505) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659116, a2=, a3=a3@entry=2256530360, a4=a4@entry=129263592270784, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=30096, futex_word=0x759086e1662c, expected=2256530360, op=7726105, abstime=0x7590867febc0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e1662c, expected=2256530360, clockid=30096, abstime=0x7590867febc0, private=30096, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e1662c, expected=expected@entry=1, clockid=clockid@entry=0, abstime=abstime@entry=0x7590867feda0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab402 in __pthread_cond_wait_common (cond=0x759086e16608, mutex=0x759086e16678, clockid=, abstime=0x7590867feda0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_timedwait64 (cond=0x759086e16608, mutex=0x759086e16678, abstime=0x7590867feda0) at ./nptl/pthread_cond_wait.c:478 +#7 0x0000759087d26d6e in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 14 (Thread 0x75903e08f6c0 (LWP 4545) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c5240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 13 (Thread 0x75903de8e6c0 (LWP 4546) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x00007590a273f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000075908c52267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c50a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c50b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c5057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 12 (Thread 0x75903dc8d6c0 (LWP 4547) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x00007590a273f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000075908c52267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c50a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c50b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c5057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 11 (Thread 0x75903da8c6c0 (LWP 4548) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c5240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 10 (Thread 0x75903cdff6c0 (LWP 4549) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 9 (Thread 0x759027fff6c0 (LWP 4550) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659324, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x759086e166fc, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e166fc, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e166fc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x759027ffeda0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab402 in __pthread_cond_wait_common (cond=0x759086e166d8, mutex=0x759086e16748, clockid=, abstime=0x759027ffeda0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_timedwait64 (cond=0x759086e166d8, mutex=0x759086e16748, abstime=0x759027ffeda0) at ./nptl/pthread_cond_wait.c:478 +#7 0x0000759087d266ca in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 8 (Thread 0x759026ffe6c0 (LWP 4551) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659528, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x759086e167c8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e167c8, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e167c8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x759086e16818, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x759086e16818) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000759087d267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 7 (Thread 0x75903c5ff6c0 (LWP 4552) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 6 (Thread 0x75903c3fe6c0 (LWP 4553) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 5 (Thread 0x7590255ff6c0 (LWP 4554) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659736, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x759086e16898, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e16898, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e16898, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x759086e168e8, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x759086e168e8) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000759087d267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 4 (Thread 0x75900f1ff6c0 (LWP 4556) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=110043530374456, a2=, a3=a3@entry=2731458932, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x64158189ad38, expected=2731458932, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x64158189ad38, expected=2731458932, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x64158189ad38, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x64158189acf0, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x64158189acf0) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909cac24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x000075909d913e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000075909caf7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 3 (Thread 0x75900e1fe6c0 (LWP 4557) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=110043530374456, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x64158189ad38, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x64158189ad38, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x64158189ad38, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x64158189acf0, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x64158189acf0) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909cac24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x000075909d913e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000075909caf7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 2 (Thread 0x75900d1fd6c0 (LWP 4558) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a838c in __internal_syscall_cancel (a1=0, a2=0, a3=0, a4=0, a5=0, a6=0, nr=34) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=a1@entry=0, a2=a2@entry=0, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=0, nr=34) at ./nptl/cancellation.c:75 +#3 0x00007590a2708301 in __libc_pause () at ../sysdeps/unix/sysv/linux/pause.c:29 +#4 0x00007590a2b63359 in PyThread_hang_thread () at Python/thread_pthread.h:440 +#5 0x00007590a2b43e7b in take_gil (tstate=0x759000005590) at Python/ceval_gil.c:318 +#6 0x00007590a2c46ebf in _PyEval_AcquireLock (tstate=) at Python/ceval_gil.c:597 +#7 0x00007590a2c4dc1b in _PyThreadState_Attach (tstate=0x759000005590) at Python/pystate.c:2100 +#8 0x00007590a2c473f2 in PyEval_RestoreThread (tstate=) at Python/ceval_gil.c:663 +#9 0x00007590a2cec124 in PyGILState_Ensure () at Python/pystate.c:2790 +#10 0x00007590a18ea941 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_python.so.2300 +#11 0x000075909d200417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#12 0x000075909d6bc734 in arrow::Future >::SetResult(arrow::Result >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#13 0x000075909d91dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#14 0x000075909d230baa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#15 0x000075909d82c525 in arrow::io::internal::ReadRangeCache::LazyImpl::~LazyImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#16 0x00007590a0a027b7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#17 0x00007590a0a8426e in parquet::SerializedFile::~SerializedFile() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#18 0x00007590a0b6ecb7 in parquet::arrow::(anonymous namespace)::FileReaderImpl::~FileReaderImpl() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#19 0x00007590a0a027b7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#20 0x00007590a0b6eb20 in std::_Function_handler > ()> > (), parquet::arrow::RowGroupGenerator>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#21 0x00007590a0b8410e in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#22 0x00007590a0a027b7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#23 0x00007590a0b76220 in std::_Function_handler > (), arrow::MergedGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#24 0x00007590a1355b19 in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#25 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#26 0x00007590a1336772 in std::_Function_handler > (), arrow::dataset::SlicingGenerator>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#27 0x00007590a1356e6e in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#28 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#29 0x00007590a133d380 in std::_Function_handler > (), arrow::SerialReadaheadGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#30 0x00007590a1345e46 in arrow::Future > ()> >::SetResult(arrow::Result > ()> >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#31 0x000075909d91dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#32 0x00007590a132bdfa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#33 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#34 0x00007590a133d240 in std::_Function_handler > (), arrow::FutureFirstGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#35 0x00007590a1355a19 in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#36 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#37 0x00007590a13368d2 in std::_Function_handler > (), arrow::DefaultIfEmptyGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#38 0x00007590a13559e9 in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#39 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#40 0x00007590a1336982 in std::_Function_handler > > (), arrow::EnumeratingGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#41 0x00007590a136f9d6 in arrow::Future > > ()> >::SetResult(arrow::Result > > ()> >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#42 0x000075909d91dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#43 0x00007590a132bdfa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#44 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#45 0x00007590a1336a32 in std::_Function_handler > > (), arrow::FutureFirstGenerator > > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#46 0x00007590a135c48d in std::_Sp_counted_ptr_inplace >, arrow::dataset::EnumeratedRecordBatch>::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#47 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#48 0x00007590a1336ae2 in std::_Function_handler (), arrow::MappingGenerator >, arrow::dataset::EnumeratedRecordBatch> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#49 0x00007590a135c506 in std::_Sp_counted_ptr_inplace::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#50 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#51 0x00007590a133cde0 in std::_Function_handler (), arrow::MergedGenerator >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#52 0x00007590a135c39e in std::_Sp_counted_ptr_inplace::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#53 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#54 0x00007590a133ce80 in std::_Function_handler (), arrow::ReadaheadGenerator >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#55 0x00007590a135c35d in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#56 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#57 0x00007590a133cfc0 in std::_Function_handler > (), arrow::MappingGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#58 0x00007590a0732a3b in arrow::acero::(anonymous namespace)::SourceNode::~SourceNode() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#59 0x00007590a079860e in arrow::acero::(anonymous namespace)::ExecPlanImpl::~ExecPlanImpl() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#60 0x00007590a07b0102 in std::_Sp_counted_ptr::_M_dispose() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#61 0x00007590a132bdfa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#62 0x00007590a13459be in std::_Function_handler (), arrow::dataset::(anonymous namespace)::AsyncScanner::ScanBatchesUnorderedAsync(arrow::internal::Executor*, bool, bool)::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#63 0x00007590a135c3ed in std::_Sp_counted_ptr_inplace::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#64 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#65 0x00007590a134d81c in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback::Callback> >::~FnImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#66 0x000075909d9108bc in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#67 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#68 0x00007590a134636f in arrow::Future::MarkFinished(arrow::Result) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#69 0x00007590a1370878 in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback, arrow::Future, false, false> > >::invoke(arrow::FutureImpl const&) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#70 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#71 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#72 0x00007590a134636f in arrow::Future::MarkFinished(arrow::Result) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#73 0x00007590a1365966 in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback::ThenOnComplete::PassthruOnFailure > > >::invoke(arrow::FutureImpl const&) [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#74 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#75 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#76 0x00007590a0744206 in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#77 0x00007590a07b044a in arrow::internal::FnOnce::FnImpl::WrapStatusyOnComplete::Callback >::invoke(arrow::FutureImpl const&) [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#78 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#79 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#80 0x000075909d24665b in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#81 0x000075909d96b4da in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback::ThenOnComplete, arrow::internal::FnOnce, arrow::StopToken)::{lambda()#1}, arrow::Future::PassthruOnFailure, arrow::internal::FnOnce, arrow::StopToken)::{lambda()#1}> > > >::invoke(arrow::FutureImpl const&) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#82 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#83 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#84 0x000075909d24665b in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#85 0x000075909d96dd4e in arrow::internal::FnOnce::FnImpl::WrapStatusyOnComplete::Callback >)::{lambda()#1}::operator()()::{lambda(arrow::Status const&)#1}> >::invoke(arrow::FutureImpl const&) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#86 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#87 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#88 0x00007590a0744206 in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#89 0x00007590a0753cc7 in arrow::internal::FnOnce::FnImpl, std::function)> >::invoke() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#90 0x000075909d9140e4 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#91 0x000075909caf7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#92 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#93 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 1 (Thread 0x7590a2fe4bc0 (LWP 4494) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=110043531371980, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x64158198e5cc, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x64158198e5cc, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x64158198e5cc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x64158198e550, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x64158198e550) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909cac24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x000075909d915f4b in arrow::internal::ThreadPool::Shutdown(bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000075909d9161c2 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#10 0x000075909d200417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#11 0x00007590a264a940 in __run_exit_handlers (status=0, listp=0x7590a2817680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:118 +#12 0x00007590a264aa1e in __GI_exit (status=) at ./stdlib/exit.c:148 +#13 0x00007590a262a8c8 in __libc_start_call_main (main=main@entry=0x641574652060
, argc=argc@entry=4, argv=argv@entry=0x7ffe1b16fd48) at ../sysdeps/nptl/libc_start_call_main.h:83 +#14 0x00007590a262a9d8 in __libc_start_main_impl (main=0x641574652060
, argc=4, argv=0x7ffe1b16fd48, init=, fini=, rtld_fini=, stack_end=0x7ffe1b16fd38) at ../csu/libc-start.c:360 +#15 0x0000641574652095 in _start () +[Inferior 1 (process 4494) detached] + +warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.11/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.11/results.jsonl new file mode 100644 index 000000000..a926510c1 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.11/results.jsonl @@ -0,0 +1,14 @@ +{"label": "linux dataset_scan mitigation=atexit_sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "linux dataset_scan mitigation=atexit_sleep_long py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "linux orcapod plain-read patch=none py3.11 locked", "n": 100, "hung": 9, "hung_after_done": 9, "rc": {"0": 91, "HUNG": 9}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=none_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch none"} +{"label": "linux orcapod source-read patch=none py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=none_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch none"} +{"label": "linux orcapod pipeline-sync patch=none py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch none"} +{"label": "linux orcapod plain-read patch=native-fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "linux orcapod source-read patch=native-fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "linux orcapod pipeline-sync patch=native-fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "linux orcapod plain-read patch=atexit-sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "linux orcapod source-read patch=atexit-sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "linux orcapod pipeline-sync patch=atexit-sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "linux orcapod plain-read patch=atexit-gc py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 79, "HUNG": 8, "-6": 13}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "unavailable", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "linux orcapod source-read patch=atexit-gc py3.11 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"0": 95, "-6": 1, "HUNG": 4}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=atexit-gc_py3.11_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "linux orcapod pipeline-sync patch=atexit-gc py3.11 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=atexit-gc_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.12/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.12/results.jsonl new file mode 100644 index 000000000..87c3fc915 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.12/results.jsonl @@ -0,0 +1,14 @@ +{"label": "linux dataset_scan mitigation=atexit_sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "linux dataset_scan mitigation=atexit_sleep_long py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "linux orcapod plain-read patch=none py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 81, "HUNG": 8, "-6": 11}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "unavailable", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch none"} +{"label": "linux orcapod source-read patch=none py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"HUNG": 8, "0": 90, "-6": 2}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=none_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch none"} +{"label": "linux orcapod pipeline-sync patch=none py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch none"} +{"label": "linux orcapod plain-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "linux orcapod source-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "linux orcapod pipeline-sync patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "linux orcapod plain-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "linux orcapod source-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "linux orcapod pipeline-sync patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "linux orcapod plain-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 11, "hung_after_done": 11, "rc": {"0": 88, "HUNG": 11, "-6": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=atexit-gc_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "linux orcapod source-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "linux orcapod pipeline-sync patch=atexit-gc py3.12 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/results.jsonl new file mode 100644 index 000000000..deeaec2ba --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/results.jsonl @@ -0,0 +1,14 @@ +{"label": "linux dataset_scan mitigation=atexit_sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "linux dataset_scan mitigation=atexit_sleep_long py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "linux orcapod plain-read patch=none py3.13 locked", "n": 100, "hung": 23, "hung_after_done": 23, "rc": {"0": 77, "HUNG": 23}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch none"} +{"label": "linux orcapod source-read patch=none py3.13 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=none_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch none"} +{"label": "linux orcapod pipeline-sync patch=none py3.13 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=none_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch none"} +{"label": "linux orcapod plain-read patch=native-fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "linux orcapod source-read patch=native-fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "linux orcapod pipeline-sync patch=native-fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "linux orcapod plain-read patch=atexit-sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "linux orcapod source-read patch=atexit-sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "linux orcapod pipeline-sync patch=atexit-sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "linux orcapod plain-read patch=atexit-gc py3.13 locked", "n": 100, "hung": 17, "hung_after_done": 17, "rc": {"0": 83, "HUNG": 17}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=atexit-gc_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "linux orcapod source-read patch=atexit-gc py3.13 locked", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=atexit-gc_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "linux orcapod pipeline-sync patch=atexit-gc py3.13 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=atexit-gc_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt new file mode 100644 index 000000000..42e81037b --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt @@ -0,0 +1,276 @@ +[New LWP 6135] +[New LWP 6134] +[New LWP 6133] +[New LWP 6132] +[New LWP 6131] +[New LWP 6130] +[New LWP 6129] +[New LWP 6128] +[New LWP 6127] +[New LWP 6126] +[New LWP 6125] +[New LWP 6124] +[New LWP 6123] +[New LWP 6080] +[New LWP 6074] +[New LWP 6071] +[Thread debugging using libthread_db enabled] +Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". +__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 + +Thread 17 (Thread 0x72005ebff6c0 (LWP 6071) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345921590824, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x72005f216628, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x72005f216628, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x72005f216628, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x72005f216678, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x72005f216678) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000720060e97962 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 16 (Thread 0x7200527ff6c0 (LWP 6074) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345799668824, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720057dd0458 , expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720057dd0458 , expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720057dd0458 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720057dd0410 , clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720057dd0410 ) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000720056c30dfb in blas_thread_server () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-56d6093b.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 15 (Thread 0x720043fff6c0 (LWP 6080) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345472801832, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720044616c28, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720044616c28, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720044616c28, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720044616c80, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720044616c80) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072004d6ae2e7 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/_polars_runtime_32/_polars_runtime.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 14 (Thread 0x71fffbfef6c0 (LWP 6123) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=107518388190316, a2=, a3=a3@entry=4227787776, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x61c99355bc6c, expected=4227787776, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x61c99355bc6c, expected=4227787776, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61c99355bc6c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x61c99355bc20, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x61c99355bc20) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072005f4c24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x0000720060313e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000072005f4f7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 13 (Thread 0x71fffafee6c0 (LWP 6124) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=107518388190316, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x61c99355bc6c, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x61c99355bc6c, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61c99355bc6c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x61c99355bc20, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x61c99355bc20) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072005f4c24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x0000720060313e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000072005f4f7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 12 (Thread 0x71fff99ff6c0 (LWP 6125) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405691432, a2=, a3=a3@entry=3265269968, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720040616628, expected=3265269968, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720040616628, expected=3265269968, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720040616628, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720040616678, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720040616678) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd26f39 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 11 (Thread 0x7200403ff6c0 (LWP 6126) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x0000720064f3f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000071fff152267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff150a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff150b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff15057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 10 (Thread 0x71fff89fe6c0 (LWP 6127) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff15240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 9 (Thread 0x71fff87fd6c0 (LWP 6128) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x0000720064f3f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000071fff152267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff150a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff150b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff15057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 8 (Thread 0x71fff85fc6c0 (LWP 6129) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff15240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 7 (Thread 0x71fff83fb6c0 (LWP 6130) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff1524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 6 (Thread 0x71ffe27ff6c0 (LWP 6131) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405691640, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x7200406166f8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x7200406166f8, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7200406166f8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720040616748, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720040616748) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 5 (Thread 0x71ffe17fe6c0 (LWP 6132) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405691848, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x7200406167c8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x7200406167c8, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7200406167c8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720040616818, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720040616818) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 4 (Thread 0x71ffe07fd6c0 (LWP 6133) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff1524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 3 (Thread 0x71ffc3dff6c0 (LWP 6134) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405692056, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720040616898, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720040616898, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720040616898, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x7200406168e8, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x7200406168e8) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 2 (Thread 0x71ffc217f6c0 (LWP 6135) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea838c in __internal_syscall_cancel (a1=0, a2=0, a3=0, a4=0, a5=0, a6=0, nr=34) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=a1@entry=0, a2=a2@entry=0, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=0, nr=34) at ./nptl/cancellation.c:75 +#3 0x0000720064f08301 in __libc_pause () at ../sysdeps/unix/sysv/linux/pause.c:29 +#4 0x0000720065363359 in PyThread_hang_thread () at Python/thread_pthread.h:440 +#5 0x0000720065343e7b in take_gil (tstate=0x71ffb0002870) at Python/ceval_gil.c:318 +#6 0x0000720065446ebf in _PyEval_AcquireLock (tstate=) at Python/ceval_gil.c:597 +#7 0x000072006544dc1b in _PyThreadState_Attach (tstate=0x71ffb0002870) at Python/pystate.c:2100 +#8 0x00007200654473f2 in PyEval_RestoreThread (tstate=) at Python/ceval_gil.c:663 +#9 0x00007200654ec124 in PyGILState_Ensure () at Python/pystate.c:2790 +#10 0x00007200642ea941 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_python.so.2300 +#11 0x000072005fc00417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#12 0x00007200600bc734 in arrow::Future >::SetResult(arrow::Result >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#13 0x000072006031dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#14 0x000072005fc30baa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#15 0x000072006022583c in arrow::internal::FnOnce::FnImpl >, arrow::io::RandomAccessFile::ReadAsync(arrow::io::IOContext const&, long, long)::{lambda()#1})> >::~FnImpl() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#16 0x00007200603140f1 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#17 0x000072005f4f7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#18 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#19 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 1 (Thread 0x7200658c3bc0 (LWP 6065) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=107518388215708, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x61c993561f9c, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x61c993561f9c, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61c993561f9c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x61c993561f20, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x61c993561f20) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072005f4c24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x0000720060315f4b in arrow::internal::ThreadPool::Shutdown(bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x00007200603161c2 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#10 0x000072005fc00417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#11 0x0000720064e4a940 in __run_exit_handlers (status=0, listp=0x720065017680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:118 +#12 0x0000720064e4aa1e in __GI_exit (status=) at ./stdlib/exit.c:148 +#13 0x0000720064e2a8c8 in __libc_start_call_main (main=main@entry=0x61c991795060
, argc=argc@entry=5, argv=argv@entry=0x7ffcb197ce58) at ../sysdeps/nptl/libc_start_call_main.h:83 +#14 0x0000720064e2a9d8 in __libc_start_main_impl (main=0x61c991795060
, argc=5, argv=0x7ffcb197ce58, init=, fini=, rtld_fini=, stack_end=0x7ffcb197ce48) at ../csu/libc-start.c:360 +#15 0x000061c991795095 in _start () +[Inferior 1 (process 6065) detached] + +warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory diff --git a/superpowers/specs/itl-892-exit-hang/native_fs_pytest_plugin.py b/superpowers/specs/itl-892-exit-hang/native_fs_pytest_plugin.py new file mode 100644 index 000000000..7eb5d04b6 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/native_fs_pytest_plugin.py @@ -0,0 +1,11 @@ +"""Pytest plugin: run the suite with option (c) applied, without editing ``src/``. + +Usage (repository root): + PYTHONPATH=superpowers/specs/itl-892-exit-hang \ + uv run pytest -p native_fs_pytest_plugin tests/ +""" + +import importlib + +# Imported by name: this directory is put on PYTHONPATH by the command above. +importlib.import_module("repro_orcapod").apply_patch("native-fs") diff --git a/superpowers/specs/itl-892-exit-hang/regression_check_proto.py b/superpowers/specs/itl-892-exit-hang/regression_check_proto.py new file mode 100644 index 000000000..8fa91e0a6 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/regression_check_proto.py @@ -0,0 +1,56 @@ +"""Prototype regression check for ITL-892: a process that read a Delta store must exit. + +Not collected by the suite (``testpaths = tests``); run it explicitly: + + uv run pytest superpowers/specs/itl-892-exit-hang/regression_check_proto.py -v + ITL892_PATCH=native-fs uv run pytest superpowers/specs/itl-892-exit-hang/regression_check_proto.py -v + +Each case starts RUNS fresh interpreters that use Orcapod's public API on a local Delta +store (``DeltaTableDatabase`` write, flush, read through a fresh handle; and +``DeltaTableSource``) and then return normally. A run fails if it does not exit within +TIMEOUT seconds (the macOS and Python 3.13 symptom) or exits non-zero (the Linux +3.11/3.12 symptom: SIGABRT, "terminate called without an active exception"). +``ITL892_PATCH`` is passed to the children, so a mitigation prototype can be checked +with the same test. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest + +SCRIPT = Path(__file__).with_name("repro_orcapod.py") +RUNS = int(os.environ.get("ITL892_RUNS", "40")) +TIMEOUT = float(os.environ.get("ITL892_TIMEOUT", "60")) +JOBS = 4 + + +def _run(mode: str) -> str | None: + """Run one child; return a failure description, or None if it exited cleanly.""" + try: + res = subprocess.run( + [sys.executable, str(SCRIPT), mode], + capture_output=True, + text=True, + timeout=TIMEOUT, + ) + except subprocess.TimeoutExpired: + return f"did not exit within {TIMEOUT:.0f}s" + if res.returncode != 0: + return f"exit code {res.returncode}: {res.stderr.strip()[-200:]}" + return None + + +@pytest.mark.parametrize("mode", ["plain-read", "source-read"]) +def test_process_exits_after_delta_read(mode: str) -> None: + """Every one of ``RUNS`` fresh interpreters runs ``mode`` and exits cleanly.""" + with ThreadPoolExecutor(max_workers=JOBS) as pool: + failures = [f for f in pool.map(_run, [mode] * RUNS) if f is not None] + assert not failures, ( + f"{len(failures)}/{RUNS} runs failed to exit cleanly: {failures[:3]}" + ) diff --git a/superpowers/specs/itl-892-exit-hang/repro_orcapod.py b/superpowers/specs/itl-892-exit-hang/repro_orcapod.py new file mode 100644 index 000000000..cfcb11240 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/repro_orcapod.py @@ -0,0 +1,202 @@ +"""ITL-892 through Orcapod: does a process that used a Delta store exit? + +Modes: + pipeline-sync ``PipelineJob`` on ``DeltaTableDatabase``, sync orchestrator, then + ``get_all_records`` on the function node. + pipeline-async the same with the async orchestrator. + plain-read ``DeltaTableDatabase`` write, flush, ``get_all_records``. + source-read ``DeltaTableSource`` over a table written by ``write_deltalake``. + rerun ``pipeline-sync`` twice on the same store (second run is all cache hits). + +Prints ``done`` and returns normally so the interpreter runs its shutdown. +""" + +from __future__ import annotations + +import argparse +import os +import tempfile +from pathlib import Path + + +def pipeline(store_dir: Path, orchestrator_name: str, runs: int = 1) -> None: + """Run a one-pod pipeline on a Delta store and read the node's records back. + + Args: + store_dir: Root of the Delta store. + orchestrator_name: ``sync`` or ``async``. + runs: How many times to build and run the pipeline. + """ + import orcapod as op + from orcapod.databases import DeltaTableDatabase + from orcapod.pipeline import AsyncPipelineOrchestrator, PipelineJob + from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator + + orch = { + "sync": SyncPipelineOrchestrator, + "async": AsyncPipelineOrchestrator, + }[orchestrator_name] + + @op.function_pod(output_keys=["result"]) + def double(x: int) -> int: + """Return ``2 * x``.""" + return x * 2 + + for _ in range(runs): + source = op.sources.DictSource( + [{"id": i, "x": 10 * i} for i in range(3)], tag_columns=["id"] + ) + job = PipelineJob(store=DeltaTableDatabase(store_dir)) + with job: + double.pod(source, label="double") + job.run(orchestrator=orch()) + # Row counts are not checked: main still has the ITL-864 partial read. + job.nodes["double"].get_all_records() + + +def plain_read(store_dir: Path) -> None: + """Write three records, then read them through a fresh ``DeltaTableDatabase``. + + Args: + store_dir: Root of the Delta store. + """ + import pyarrow as pa + + from orcapod.databases import DeltaTableDatabase + + db = DeltaTableDatabase(store_dir) + table = pa.table( + { + "__record_id": pa.array(["a", "b", "c"], pa.large_string()), + "v": pa.array([1, 2, 3], pa.int64()), + } + ) + db.add_records(("t",), table, record_id_column="__record_id") + db.flush() + fresh = DeltaTableDatabase(store_dir) + records = fresh.get_all_records(("t",), record_id_column="__record_id") + # get_all_records returns None when a scan fails, so a broken patch must not pass. + assert records is not None and records.num_rows == 3, records + + +def source_read(store_dir: Path) -> None: + """Write a Delta table with deltalake and read it through ``DeltaTableSource``. + + Args: + store_dir: Directory for the table. + """ + import deltalake + import pyarrow as pa + + import orcapod as op + + path = store_dir / "src" + deltalake.write_deltalake( + str(path), + pa.table({"id": pa.array([1, 2, 3]), "x": pa.array([1.0, 2.0, 3.0])}), + ) + src = op.sources.DeltaTableSource(path, tag_columns=["id"]) + assert src.as_table().num_rows == 3 + + +def apply_patch(name: str) -> None: + """Prototype a mitigation by wrapping deltalake, so ``src/`` stays unchanged. + + native-fs Local tables are scanned through a native pyarrow filesystem rooted at + the table, instead of deltalake's default + ``PyFileSystem(DeltaStorageHandler)``; option (c). + no-threads Every ``Dataset.to_table`` built by deltalake runs with + ``use_threads=False``; option (c), alternative. + atexit-gc An ``atexit`` hook runs ``gc.collect()``; option (b). + atexit-sleep An ``atexit`` hook releases the GIL for 50 ms so pool workers can drop + their Python references before finalization; option (b), after + apache/arrow#54467. + """ + if name == "none": + return + if name == "atexit-sleep": + import atexit + import time + + atexit.register(time.sleep, 0.05) + return + if name == "atexit-gc": + import atexit + import gc + + atexit.register(gc.collect) + return + + import deltalake + import pyarrow.fs as pa_fs + + original = deltalake.DeltaTable.to_pyarrow_dataset + + def to_pyarrow_dataset(self, *args, **kwargs): # noqa: ANN001, ANN002, ANN003, ANN202 + """Open the dataset with the selected prototype applied.""" + if name == "native-fs" and kwargs.get("filesystem") is None: + uri = self.table_uri + local = uri[len("file://") :] if uri.startswith("file://") else uri + if "://" not in local: + kwargs["filesystem"] = pa_fs.SubTreeFileSystem( + local, pa_fs.LocalFileSystem() + ) + ds = original(self, *args, **kwargs) + if name == "no-threads": + to_table = ds.to_table + + class _NoThreads: + """Dataset proxy whose ``to_table`` runs single-threaded.""" + + def __getattr__(self, attr: str): # noqa: ANN204 + """Forward every other attribute to the wrapped dataset.""" + return getattr(ds, attr) + + def to_table(self, *a, **kw): # noqa: ANN002, ANN003, ANN202 + """Scan the wrapped dataset with ``use_threads=False``.""" + kw["use_threads"] = False + return to_table(*a, **kw) + + return _NoThreads() + return ds + + deltalake.DeltaTable.to_pyarrow_dataset = to_pyarrow_dataset # pyright: ignore[reportAttributeAccessIssue] # experiment-only monkeypatch + + +def main() -> None: + """Apply the selected patch, run the selected scenario and print ``done``.""" + ap = argparse.ArgumentParser() + ap.add_argument( + "mode", + choices=[ + "pipeline-sync", + "pipeline-async", + "plain-read", + "source-read", + "rerun", + ], + ) + ap.add_argument( + "--patch", + default="none", + choices=["none", "native-fs", "no-threads", "atexit-gc", "atexit-sleep"], + help="prototype a mitigation without editing src/", + ) + args = ap.parse_args() + apply_patch(os.environ.get("ITL892_PATCH") or str(args.patch)) + store = Path(tempfile.mkdtemp()) + if args.mode == "pipeline-sync": + pipeline(store, "sync") + elif args.mode == "pipeline-async": + pipeline(store, "async") + elif args.mode == "rerun": + pipeline(store, "sync", runs=2) + elif args.mode == "plain-read": + plain_read(store) + else: + source_read(store) + print("done", flush=True) + + +if __name__ == "__main__": + main() diff --git a/superpowers/specs/itl-892-exit-hang/repro_pure.py b/superpowers/specs/itl-892-exit-hang/repro_pure.py new file mode 100644 index 000000000..3a46ad23e --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/repro_pure.py @@ -0,0 +1,216 @@ +"""Minimal ITL-892 reproduction: pyarrow + deltalake only, no Orcapod. + +Builds a small Delta table in a temporary directory, performs the operations chosen by +``--op``, prints ``done`` and returns normally so the interpreter runs its shutdown. + +Ops: + write one ``write_deltalake`` (pyarrow input), no read. + merge one write, then four ``DeltaTable.merge`` calls, no read. + to_table one write, then ``DeltaTable.to_pyarrow_table()``. + to_dataset one write, then ``DeltaTable.to_pyarrow_dataset()`` (no scan). + dataset_scan one write, then ``to_pyarrow_dataset().to_table()``. + itl864 the ITL-864 worker's script: write, four merges, and a + ``to_pyarrow_dataset(as_large_types=True).to_table()`` after each. + pyarrow_only pyarrow parquet write + dataset scan, deltalake imported but unused. +""" + +from __future__ import annotations + +import argparse +import atexit +import gc +import os +import sys +import tempfile + + +def build(path: str, i: int): + """Build the one-row table written in round ``i``. + + Args: + path: Table directory (unused; kept for the call sites). + i: Round number. + + Returns: + The table. + """ + import pyarrow as pa + + return pa.table( + {"k": pa.array([i], pa.int64()), "v": pa.array([str(i)], pa.large_string())} + ) + + +def run(op: str, read_via: str, rounds: int = 4) -> list[object]: + """Perform ``op``; return the objects created so mitigations can drop them.""" + import deltalake + import pyarrow as pa + + keep: list[object] = [] + path = tempfile.mkdtemp() + + def read() -> None: + """Read the table through the selected reader, keeping every object it creates.""" + dt = deltalake.DeltaTable(path) + keep.append(dt) + if read_via == "dataset": + ds = dt.to_pyarrow_dataset(as_large_types=True) + keep.append(ds) + keep.append(ds.to_table()) + elif read_via == "table": + keep.append(dt.to_pyarrow_table()) + elif read_via == "native_fs": + # Same dataset path, but a native pyarrow filesystem instead of the default + # PyFileSystem(DeltaStorageHandler), so no Python file object is involved. + import pyarrow.fs as pa_fs + + fs = pa_fs.SubTreeFileSystem(path, pa_fs.LocalFileSystem()) + ds = dt.to_pyarrow_dataset(as_large_types=True, filesystem=fs) + keep.append(ds) + keep.append(ds.to_table()) + elif read_via == "no_threads": + ds = dt.to_pyarrow_dataset(as_large_types=True) + keep.append(ds) + keep.append(ds.to_table(use_threads=False)) + elif read_via == "querybuilder": + ctx = deltalake.QueryBuilder().register("t", dt) + keep.append(pa.table(ctx.execute("select * from t").read_all())) + elif read_via == "files": + import pyarrow.parquet as pq + + keep.append(pq.read_table(dt.file_uris())) + elif read_via == "polars": + import polars as pl + + keep.append(pl.read_delta(path)) + else: + raise ValueError(read_via) + + if op == "pyarrow_only": + import pyarrow.dataset as pads + import pyarrow.parquet as pq + + pq.write_table(build(path, 0), os.path.join(path, "a.parquet")) + keep.append(pads.dataset(path).to_table()) + return keep + + deltalake.write_deltalake(path, build(path, 0), mode="overwrite") + if op == "write": + return keep + if op == "merge": + for i in range(1, 5): + deltalake.DeltaTable(path).merge( + source=build(path, i), + predicate="target.k = source.k", + source_alias="source", + target_alias="target", + ).when_not_matched_insert_all().execute() + return keep + if op == "to_table": + read_via = "table" if read_via == "dataset" else read_via + read() + return keep + if op == "to_dataset": + dt = deltalake.DeltaTable(path) + keep += [dt, dt.to_pyarrow_dataset(as_large_types=True)] + return keep + if op == "dataset_scan": + read() + return keep + if op == "itl864": + read() + for i in range(1, rounds + 1): + deltalake.DeltaTable(path).merge( + source=build(path, i), + predicate="target.k = source.k", + source_alias="source", + target_alias="target", + ).when_not_matched_insert_all().execute() + read() + return keep + raise ValueError(op) + + +def main() -> None: + """Run the selected operation, apply the selected mitigation and print ``done``.""" + ap = argparse.ArgumentParser() + ap.add_argument("--op", default="itl864") + ap.add_argument( + "--read-via", + default="dataset", + choices=[ + "dataset", + "table", + "native_fs", + "no_threads", + "querybuilder", + "files", + "polars", + ], + ) + ap.add_argument("--rounds", type=int, default=4, help="merge rounds for itl864") + ap.add_argument("--import-polars", action="store_true") + ap.add_argument("--polars-first", action="store_true") + ap.add_argument("--cpu-threads", type=int) + ap.add_argument("--io-threads", type=int) + ap.add_argument( + "--mitigation", + default="none", + choices=[ + "none", + "del", + "atexit_del", + "os_exit", + "atexit_sleep", + "atexit_sleep_long", + ], + ) + ap.add_argument("--mp", default="none", choices=["none", "fork", "spawn"]) + args = ap.parse_args() + + if args.polars_first: + import polars # noqa: F401 + import pyarrow as pa + + if args.import_polars: + import polars # noqa: F401 + if args.cpu_threads is not None: + pa.set_cpu_count(args.cpu_threads) + if args.io_threads is not None: + pa.set_io_thread_count(args.io_threads) + + if args.mp != "none": + import multiprocessing as mp + + ctx = mp.get_context(args.mp) + p = ctx.Process(target=run, args=(args.op, args.read_via, args.rounds)) + p.start() + p.join() + if p.exitcode != 0: + # A child that raised or died on a signal is a failed run, not a clean one. + sys.exit(f"child process failed with exit code {p.exitcode}") + keep: list[object] = [] + else: + keep = run(args.op, args.read_via, args.rounds) + + print("done", flush=True) + if args.mitigation == "del": + keep.clear() + gc.collect() + elif args.mitigation == "atexit_del": + atexit.register(lambda: (keep.clear(), gc.collect())) + elif args.mitigation in ("atexit_sleep", "atexit_sleep_long"): + # Release the GIL before finalization so pool workers can drop their Python + # references while the interpreter can still give them the GIL (the idea of + # apache/arrow#54467, approximated from Python). + import time + + delay = 0.05 if args.mitigation == "atexit_sleep" else 0.5 + atexit.register(time.sleep, delay) + elif args.mitigation == "os_exit": + sys.stdout.flush() + os._exit(0) + + +if __name__ == "__main__": + main() diff --git a/superpowers/specs/itl-892-exit-hang/results.jsonl b/superpowers/specs/itl-892-exit-hang/results.jsonl new file mode 100644 index 000000000..973051286 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/results.jsonl @@ -0,0 +1,82 @@ +{"label": "baseline itl864 py3.12 locked", "n": 40, "hung": 2, "hung_after_done": 2, "rc": {"0": 38, "HUNG": 2}, "timeout_s": 20.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/baseline_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "load j=1 itl864 py3.12 locked", "n": 60, "hung": 2, "hung_after_done": 2, "rc": {"0": 58, "HUNG": 2}, "timeout_s": 15.0, "jobs": 1, "platform": "Darwin arm64", "stack": "samples/load_j=1_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "load j=4 itl864 py3.12 locked", "n": 60, "hung": 3, "hung_after_done": 3, "rc": {"0": 57, "HUNG": 3}, "timeout_s": 15.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/load_j=4_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "amplify rounds=12 itl864 py3.12 locked", "n": 40, "hung": 0, "hung_after_done": 0, "rc": {"0": 40}, "timeout_s": 20.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --rounds 12"} +{"label": "amplify rounds=24 itl864 py3.12 locked", "n": 40, "hung": 0, "hung_after_done": 0, "rc": {"0": 40}, "timeout_s": 20.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --rounds 24"} +{"label": "load j=10 itl864 py3.12 locked", "n": 60, "hung": 6, "hung_after_done": 6, "rc": {"HUNG": 6, "0": 54}, "timeout_s": 15.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/load_j=10_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "op=pyarrow_only py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op pyarrow_only"} +{"label": "op=write py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op write"} +{"label": "op=merge py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op merge"} +{"label": "op=to_dataset py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_dataset"} +{"label": "op=to_table py3.12 locked", "n": 100, "hung": 57, "hung_after_done": 57, "rc": {"HUNG": 57, "0": 43}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/op=to_table_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "op=dataset_scan py3.12 locked", "n": 100, "hung": 57, "hung_after_done": 57, "rc": {"0": 43, "HUNG": 57}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/op=dataset_scan_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "op=itl864 py3.12 locked", "n": 100, "hung": 12, "hung_after_done": 12, "rc": {"0": 88, "HUNG": 12}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/op=itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "itl864 read-via=native_fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via native_fs"} +{"label": "itl864 read-via=no_threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via no_threads"} +{"label": "itl864 read-via=querybuilder py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via querybuilder"} +{"label": "itl864 read-via=files py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via files"} +{"label": "itl864 read-via=polars py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via polars"} +{"label": "itl864 cpu_threads=1", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/itl864_cpu_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --cpu-threads 1"} +{"label": "dataset_scan read-via=native_fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan read-via=no_threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "dataset_scan read-via=querybuilder py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via querybuilder"} +{"label": "dataset_scan read-via=files py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via files"} +{"label": "dataset_scan read-via=polars py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via polars"} +{"label": "dataset_scan cpu_threads=1", "n": 100, "hung": 33, "hung_after_done": 33, "rc": {"0": 67, "HUNG": 33}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_cpu_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --cpu-threads 1"} +{"label": "dataset_scan io_threads=1", "n": 100, "hung": 53, "hung_after_done": 53, "rc": {"0": 47, "HUNG": 53}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_io_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "dataset_scan OMP_NUM_THREADS=1", "n": 100, "hung": 45, "hung_after_done": 45, "rc": {"0": 55, "HUNG": 45}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_OMP_NUM_THREADS=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan RAYON/TOKIO threads=1", "n": 100, "hung": 49, "hung_after_done": 49, "rc": {"0": 51, "HUNG": 49}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_RAYON_TOKIO_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan import polars after pyarrow", "n": 100, "hung": 48, "hung_after_done": 48, "rc": {"0": 52, "HUNG": 48}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_import_polars_after_pyarrow.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --import-polars"} +{"label": "dataset_scan import polars first", "n": 100, "hung": 27, "hung_after_done": 27, "rc": {"0": 73, "HUNG": 27}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_import_polars_first.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --polars-first"} +{"label": "dataset_scan in fork child", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mp fork"} +{"label": "dataset_scan in spawn child", "n": 100, "hung": 90, "hung_after_done": 0, "rc": {"0": 10, "HUNG": 90}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_in_spawn_child.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mp spawn"} +{"label": "dataset_scan mitigation=del", "n": 100, "hung": 2, "hung_after_done": 2, "rc": {"0": 98, "HUNG": 2}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_mitigation=del.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "dataset_scan mitigation=atexit_del", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_mitigation=atexit_del.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_del"} +{"label": "dataset_scan mitigation=os_exit", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation os_exit"} +{"label": "dataset_scan mitigation=atexit_sleep", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "dataset_scan mitigation=atexit_sleep_long", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "orcapod plain-read py3.12 locked", "n": 100, "hung": 53, "hung_after_done": 53, "rc": {"0": 47, "HUNG": 53}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "orcapod source-read py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_source-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "orcapod pipeline-sync py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "orcapod pipeline-async py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "orcapod rerun py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py rerun"} +{"label": "dataset_scan py3.11 pa23.0.1 dl1.0.2", "n": 100, "hung": 57, "hung_after_done": 57, "rc": {"0": 43, "HUNG": 57}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.11_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan native_fs py3.11 pa23.0.1 dl1.0.2", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan py3.12 pa23.0.1 dl1.0.2", "n": 100, "hung": 58, "hung_after_done": 58, "rc": {"0": 42, "HUNG": 58}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan native_fs py3.12 pa23.0.1 dl1.0.2", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan py3.13 pa23.0.1 dl1.0.2", "n": 100, "hung": 65, "hung_after_done": 65, "rc": {"0": 35, "HUNG": 65}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.13_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan native_fs py3.13 pa23.0.1 dl1.0.2", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan py3.12 pa20.0.0 dl1.6.6", "n": 100, "hung": 59, "hung_after_done": 59, "rc": {"0": 41, "HUNG": 59}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa20.0.0_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa22.0.0 dl1.0.2", "n": 100, "hung": 55, "hung_after_done": 55, "rc": {"0": 45, "HUNG": 55}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa22.0.0 dl1.6.6", "n": 100, "hung": 55, "hung_after_done": 55, "rc": {"0": 45, "HUNG": 55}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa22.0.0_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa23.0.1 dl1.0.2", "n": 100, "hung": 50, "hung_after_done": 50, "rc": {"0": 50, "HUNG": 50}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa23.0.1 dl1.6.6", "n": 100, "hung": 61, "hung_after_done": 61, "rc": {"0": 39, "HUNG": 61}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa23.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa24.0.0 dl1.0.2", "n": 100, "hung": 53, "hung_after_done": 53, "rc": {"0": 47, "HUNG": 53}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa24.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa24.0.0 dl1.6.6", "n": 100, "hung": 58, "hung_after_done": 58, "rc": {"0": 42, "HUNG": 58}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa24.0.0_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa25.0.1 dl1.0.2", "n": 100, "hung": 36, "hung_after_done": 36, "rc": {"0": 64, "HUNG": 36}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa25.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa25.0.1 dl1.6.6", "n": 100, "hung": 39, "hung_after_done": 39, "rc": {"0": 61, "HUNG": 39}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa25.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa26.0.1 dl1.0.2", "n": 100, "hung": 56, "hung_after_done": 56, "rc": {"0": 44, "HUNG": 56}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa26.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa26.0.1 dl1.6.6", "n": 100, "hung": 68, "hung_after_done": 68, "rc": {"HUNG": 68, "0": 32}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa26.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa20.0.0 dl1.0.2", "n": 100, "hung": 67, "hung_after_done": 67, "rc": {"0": 33, "HUNG": 67}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa20.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "orcapod plain-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "orcapod source-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "orcapod pipeline-sync patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "orcapod pipeline-async patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch native-fs"} +{"label": "orcapod plain-read patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch no-threads"} +{"label": "orcapod source-read patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch no-threads"} +{"label": "orcapod pipeline-sync patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch no-threads"} +{"label": "orcapod pipeline-async patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch no-threads"} +{"label": "orcapod plain-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 27, "hung_after_done": 27, "rc": {"0": 73, "HUNG": 27}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "orcapod source-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_source-read_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "orcapod pipeline-sync patch=atexit-gc py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} +{"label": "orcapod pipeline-async patch=atexit-gc py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch atexit-gc"} +{"label": "orcapod plain-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "orcapod source-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "orcapod pipeline-sync patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "orcapod pipeline-async patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch atexit-sleep"} +{"label": "orcapod plain-read py3.11 locked", "n": 100, "hung": 77, "hung_after_done": 77, "rc": {"0": 23, "HUNG": 77}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "orcapod pipeline-sync py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "orcapod pipeline-async py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "orcapod plain-read py3.13 locked", "n": 100, "hung": 76, "hung_after_done": 76, "rc": {"0": 24, "HUNG": 76}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "orcapod pipeline-sync py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "orcapod pipeline-async py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} diff --git a/superpowers/specs/itl-892-exit-hang/run_matrix.sh b/superpowers/specs/itl-892-exit-hang/run_matrix.sh new file mode 100644 index 000000000..6f5de99e2 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/run_matrix.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# ITL-892 macOS matrix. Run from the repository root: +# bash superpowers/specs/itl-892-exit-hang/run_matrix.sh [N] +# Groups: ops, readpath, threads, mitigations, python, versions, orcapod, +# orcapod-python, orcapod-patches. +# Each cell appends one JSON line to superpowers/specs/itl-892-exit-hang/results.jsonl. +set -u +D=superpowers/specs/itl-892-exit-hang +N=${2:-100} +J=${J:-10} +T=${T:-8} +OP=${OP:-dataset_scan} +H() { uv run python $D/harness.py -n "$N" -t "$T" -j "$J" "$@"; } +# Throwaway environment with chosen versions; uv.lock is untouched. +V() { # V