diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 616fc31fe..ae2cdee60 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2380,6 +2380,114 @@ newcomer. Re-checked on `b197544e` (v0.2.0 + #291): column ... because it's a binary field` lines on every Delta write still appear. **Open:** awaiting the developer's choice of how to silence them. +--- + +## `src/orcapod/databases/delta_lake_databases.py` — reading back a handle family's own writes + +### DW1 — A read through one `at()` handle could miss rows its family had just written +**Status:** resolved +**Severity:** high +**Issue:** ITL-864 + +Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` +returned 1 of 2 or 3 rows (plain `int -> int` pod, sync and async orchestrators), while +`iter_data` and a fresh handle saw every row. Every `at()` handle shares the root's pending +batches, and any handle's `flush()` writes every pending key, so the result-store handle +flushed the pipeline table's rows. But each handle kept its own loaded-table cache, so the +node's pipeline handle kept reading the table version it had loaded during the run. Every read +method missed the rows a sibling flushed. Separately, the read methods merged pending and stored +rows inconsistently: `get_records_by_ids` ignored pending rows, `get_records_with_column_value` +ignored stored rows whenever a pending batch existed (and returned an empty table instead of +`None`), and a pending row whose record id was already stored was returned as well, although a +flush keeps the stored row and drops the pending one. + +**Fix:** the loaded-table, existing-ids and cache-dirty dicts are shared across the `at()` +family exactly as pending state is. Every read method returns what a flush would leave: the +stored rows plus the pending rows whose record id is not stored (`_pending_not_stored`), and +`None` when there are none. These dicts are touched only from the thread that drives the +pipeline; the async orchestrator runs only user functions in worker threads. Regression tests: +`tests/test_databases/test_delta_read_your_writes.py` and +`tests/test_pipeline/test_delta_node_records_after_run.py`. A failed stored-id refresh leaves +the cache dirty, so the next read retries it instead of treating every stored id as unstored +and returning duplicates; in `get_records_by_ids` the refresh shares the stored read's error +handling. A handle that loaded the table before it had any stored ids (a fresh root over an +existing table) refreshes them on its first read with pending rows. + +When stored and pending rows are read together, the result has the schema a flush would give: +both parts are large-normalized (`null` kept), then combined with +`pa.concat_tables(promote_options="default")`. A column missing on one side is null-filled, and a +`null`-typed column takes the other side's type. Columns whose types differ (`int64` vs `int32`, +a string vs an int, a timestamp unit, a decimal precision) raise `pyarrow.ArrowTypeError`; nothing +is widened. Before this, a read of stored rows next to pending rows with any schema difference, +including a pending `string` column next to the stored `large_string`, raised `ArrowInvalid`; +`get_all_records` already did so on `main`. +A `get_records_with_column_value` query on a column that only the stored or only the pending +rows have skips the side without it (after a flush that side holds null there, and +`column == value` never matches null), so it returns before a flush what it returns after; a +column neither side has still raises `ArrowInvalid`. + +Recorded for the nullability audit (ITL-842) and ITL-890: Delta's merge relaxes nullability (a +non-null pending column into a nullable stored one stays nullable) and evolves struct children +(`struct

` stored + `struct` pending becomes `struct`, stored rows `q = null`). The +read path now mirrors both. Whether Orcapod should allow either at all is for those issues. A +pending all-null `null`-typed column reads back as `null`, but Delta cannot store it: the flush +fails and the batch stays pending (unchanged, pinned by +`test_pending_null_typed_column_reads_back_but_does_not_flush`). + +Still open: a handle does not see writes made by another process, or by a separate +`DeltaTableDatabase` root on the same path, until it reloads the table; this is unchanged. +Reading with a pending batch also refreshes the stored-id set when it is stale, which loads the +whole table (D3). + +--- + +## `src/orcapod/databases/delta_lake_databases.py`, `src/orcapod/core/sources/delta_table_source.py` — exit after a Delta read + +### DX1 — A process that read a Delta table can hang or abort at interpreter exit +**Status:** resolved (local stores); cloud stores remain exposed upstream +**Severity:** high +**Issue:** ITL-892 + +`DeltaTable.to_pyarrow_dataset()` (and `to_pyarrow_table()`) reads through deltalake's +`PyFileSystem(DeltaStorageHandler)`, a filesystem implemented in Python. After a scan returns, +Arrow finishes tearing down the scan on a thread-pool worker. If that worker drops the last +reference to a Python-backed buffer during interpreter finalization, CPython stops the thread, +and `ThreadPool::~ThreadPool` in `__cxa_finalize` waits for it forever. On Linux with Python +3.11/3.12 the process may abort with SIGABRT instead. This is upstream: +apache/arrow#50482 and #54536, with an open fix PR, #54467. The failure follows a scan, not a +write. + +Measured over 100 runs each, locked versions: one write plus one scan fails 57% of runs on +macOS arm64 and 5–38% on Linux x86_64. A fresh `DeltaTableDatabase.get_all_records` fails 53% +on macOS and 8–23% on Linux. `DeltaTableSource` fails 4–10%. A `PipelineJob` run fails 0–4%. +Every pyarrow 20.0.0–26.0.1 × deltalake 1.0.2/1.6.6 combination fails, so no version pin helps. +Scanning local tables through a native pyarrow filesystem (`SubTreeFileSystem(table_path, +LocalFileSystem())`) gives 0/100 in every case. Its results are byte-identical. Cloud stores +keep the Python filesystem and stay exposed until pyarrow ships the fix. Findings, scripts and +raw results: `superpowers/specs/itl-892-exit-hang/FINDINGS.md`. + +**Fix:** `storage_utils.delta_to_pyarrow_dataset` opens local Delta tables on a native pyarrow +filesystem (`SubTreeFileSystem` over `LocalFileSystem`). Every scan in `DeltaTableDatabase` and +`DeltaTableSource` goes through it. Cloud URIs keep deltalake's reader. The results are +schema- and value-identical: `tests/test_databases/test_delta_native_fs_read.py` compares them +with deltalake's reader for logical types (Path, dataclass, numpy, pandas), partitioned tables +and escaped partition values, and for table roots whose names need percent-encoding +(`table_uri` is percent-encoded, so the helper decodes it with `pyarrow.fs.FileSystem.from_uri`; +a root that is neither a `file://` URI nor an absolute POSIX path, such as a Windows drive path, +is untested and keeps deltalake's reader). `tests/test_databases/test_delta_exit.py` runs fresh +interpreters and fails if one does not exit cleanly; it failed before the fix. The remaining +cloud-store exposure is documented in `docs/troubleshooting.md`. + +**Considered alternative, not implemented:** an `atexit` hook that releases the GIL briefly +(`time.sleep(0.05)`) before finalization, the idea of apache/arrow#54467. It measured 0/100 on +macOS and Linux and also covers cloud stores. But it is timing-based, so a loaded machine or a +large scan can outlast it. It would also run in every process that imports Orcapod. If cloud +stores need covering before pyarrow ships the fix, register it lazily on the first cloud-store +scan. When pyarrow ships #54467, bump the floor and reconsider whether the helper is still +needed. + +--- + ## `src/orcapod/core/operators/filters.py` — `PolarsFilter` identity and nullability (ITL-828) ### PF1 — An expression predicate could not be hashed, and a filter broke a join inside a job diff --git a/changelog.d/ITL-864.md b/changelog.d/ITL-864.md new file mode 100644 index 000000000..cfc4bc3dc --- /dev/null +++ b/changelog.d/ITL-864.md @@ -0,0 +1,16 @@ +### Fixed + +#### Delta reads return every row the database's handles wrote (ITL-864) + +Right after a first `PipelineJob` run on a `DeltaTableDatabase`, `node.get_all_records()` could +return only some of the rows (1 of 2 or 3) on both orchestrators. Handles made with `at()` share +their pending writes, and any handle's flush writes all of them, but each handle kept its own +loaded copy of the table, so a sibling kept reading the version it had loaded before the flush. +The loaded tables and the stored-id sets are now shared by every handle of one root. Every read +method (`get_all_records`, `get_record_by_id`, `get_records_by_ids`, +`get_records_with_column_value`) returns what a flush would leave: the stored rows plus the +pending rows whose record id is not stored, or `None` when there are none. Before, +`get_records_by_ids` ignored pending rows, `get_records_with_column_value` ignored stored rows +while a pending batch existed, and a pending row whose id was already stored could be returned +twice or instead of the stored row. Writes made by another process or another root on the same +path are still seen only after the table is reloaded. (ITL-864 #293) diff --git a/changelog.d/ITL-892.md b/changelog.d/ITL-892.md new file mode 100644 index 000000000..f6510cb02 --- /dev/null +++ b/changelog.d/ITL-892.md @@ -0,0 +1,16 @@ +### Fixed + +#### A script that read a local Delta store no longer hangs at exit (ITL-892) + +A process that had read a Delta table through `DeltaTableDatabase` or `DeltaTableSource` could +finish its work and then never exit: it hung in Arrow's thread-pool destructor, or, on Linux with +Python 3.11 or 3.12, aborted with `terminate called without an active exception`. It happened in +up to three runs in four on macOS and about one in ten on Linux, on every supported pyarrow and +deltalake version. The cause is upstream (apache/arrow#50482, apache/arrow#54536). deltalake's +default reader goes through a filesystem written in Python, and Arrow can release one of its +buffers on a worker thread while the interpreter is shutting down. + +Orcapod now reads local Delta tables through a native pyarrow filesystem, which returns the same +schema and values. Tables on cloud storage (`s3://`, `gs://`, `az://`) still use deltalake's +reader and stay exposed until pyarrow ships the fix; see +[Troubleshooting](docs/troubleshooting.md). diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 000000000..b0e5f3d94 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,39 @@ +# Troubleshooting + +## A script finishes its work but the process never exits + +**Symptom.** A Python script that used a Delta store prints all of its output and then hangs, +and has to be killed. On Linux with Python 3.11 or 3.12 the process may instead end with +`terminate called without an active exception` and exit code 134 (SIGABRT). Notebooks are not +affected in the same way, because the kernel keeps running. + +**Cause.** This is a bug in pyarrow, not in your pipeline +([apache/arrow#50482](https://github.com/apache/arrow/issues/50482), +[apache/arrow#54536](https://github.com/apache/arrow/issues/54536)). deltalake's default reader +goes through a filesystem written in Python. After a read returns, Arrow finishes cleaning it up +on a worker thread. If that cleanup runs while the interpreter is shutting down, the worker +cannot take the GIL and never finishes, and Arrow's thread pool waits for it forever. Your data +is not affected: the hang happens after every write has completed. + +**What Orcapod does.** `DeltaTableDatabase` and `DeltaTableSource` read **local** Delta tables +through a native pyarrow filesystem, so they are not affected. Tables on cloud storage +(`s3://`, `gs://`, `az://` and similar) still use deltalake's reader and can still hang at exit. +Reading a Delta table yourself with `deltalake.DeltaTable(...).to_pyarrow_table()` or +`to_pyarrow_dataset()` is affected too, whatever the storage. + +**Workarounds.** + +- Upgrade pyarrow once a release includes the fix + ([apache/arrow#54467](https://github.com/apache/arrow/pull/54467)). +- In a script that reads a cloud Delta store, end it by flushing your outputs and calling + `os._exit(0)`. That skips interpreter shutdown, where the hang happens. It also skips + `atexit` handlers and other cleanup, so close files and flush stores first. +- When reading local Delta tables directly with deltalake, pass a native filesystem. Resolve + `dt.table_uri` first: it is percent-encoded, and newer deltalake versions return a + `file://` URI. + + ```python + uri = dt.table_uri if "://" in dt.table_uri else "file://" + dt.table_uri + fs, root = pyarrow.fs.FileSystem.from_uri(uri) + dataset = dt.to_pyarrow_dataset(filesystem=pyarrow.fs.SubTreeFileSystem(root, fs)) + ``` diff --git a/mkdocs.yml b/mkdocs.yml index dd10922a2..41553db5d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -52,3 +52,4 @@ nav: - Types: api/types.md - Reference: - Hashing: reference/hashing.md + - Troubleshooting: troubleshooting.md diff --git a/src/orcapod/core/sources/delta_table_source.py b/src/orcapod/core/sources/delta_table_source.py index bfc2b0520..86a187932 100644 --- a/src/orcapod/core/sources/delta_table_source.py +++ b/src/orcapod/core/sources/delta_table_source.py @@ -6,6 +6,7 @@ from orcapod.core.sources.base import RootSource from orcapod.core.sources.stream_builder import SourceStreamBuilder +from orcapod.databases.storage_utils import delta_to_pyarrow_dataset from orcapod.types import PathLike from orcapod.utils.lazy_module import LazyModule from orcapod.utils.schema_utils import _normalize_column_list @@ -49,8 +50,8 @@ def __init__( except deltalake.exceptions.TableNotFoundError: raise ValueError(f"Delta table not found at {self._delta_table_path}") - table: pa.Table = self._delta_table.to_pyarrow_dataset( - as_large_types=True + table: pa.Table = delta_to_pyarrow_dataset( + self._delta_table, as_large_types=True ).to_table() # The Delta log stores field-level nullable flags; to_pyarrow_dataset() # preserves them exactly — no inference needed. diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index 8c34c1f04..93ae02e73 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -8,7 +8,11 @@ from orcapod.databases.utils import coerce_record_id from orcapod.errors import InputValidationError, StoreReadError -from orcapod.databases.storage_utils import is_cloud_uri, parse_base_path +from orcapod.databases.storage_utils import ( + delta_to_pyarrow_dataset, + is_cloud_uri, + parse_base_path, +) from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -311,6 +315,9 @@ def __init__( _scoped_path: tuple[str, ...] = (), _shared_pending_batches: "dict[str, pa.Table] | None" = None, _shared_pending_record_ids: "defaultdict[str, set[bytes]] | None" = None, + _shared_delta_table_cache: "dict[str, deltalake.DeltaTable] | None" = None, + _shared_existing_ids_cache: "dict[str, set[bytes]] | None" = None, + _shared_cache_dirty: "dict[str, bool] | None" = None, ): self._root_uri, self._storage_options = parse_base_path( base_path, storage_options @@ -335,9 +342,16 @@ def __init__( ) # For cloud paths: create_base_path is silently ignored (no directory needed). - self._delta_table_cache: dict[str, deltalake.DeltaTable] = {} - # Shared pending state — when constructed via at(), these dicts are shared - # with the root instance so that root.flush() can flush all scoped writes. + # Shared state — when constructed via at(), these dicts are shared with the + # root instance. Any handle's flush() writes every pending key of the family + # and refreshes the loaded table and existing-ids entries for it, so these + # caches must be family-wide too: a per-handle cache would keep serving the + # snapshot a sibling loaded before the flush (ITL-864). All of them are + # touched only from the thread that drives the pipeline (the async + # orchestrator runs only user functions in worker threads). + self._delta_table_cache: dict[str, deltalake.DeltaTable] = ( + _shared_delta_table_cache if _shared_delta_table_cache is not None else {} + ) if _shared_pending_batches is not None: self._pending_batches = _shared_pending_batches else: @@ -346,8 +360,16 @@ def __init__( self._pending_record_ids = _shared_pending_record_ids else: self._pending_record_ids: dict[str, set[bytes]] = defaultdict(set) - self._existing_ids_cache: dict[str, set[str]] = defaultdict(set) - self._cache_dirty: dict[str, bool] = defaultdict(lambda: True) + self._existing_ids_cache: dict[str, set[bytes]] = ( + _shared_existing_ids_cache + if _shared_existing_ids_cache is not None + else defaultdict(set) + ) + self._cache_dirty: dict[str, bool] = ( + _shared_cache_dirty + if _shared_cache_dirty is not None + else defaultdict(lambda: True) + ) def _clear_pending(self): """Clear all pending state.""" @@ -582,9 +604,11 @@ def _refresh_existing_ids_cache(self, record_path: tuple[str, ...]) -> None: try: # Get all existing IDs from Delta table using standard RECORD_ID_COLUMN # TODO: replace this with more targetted loading of only the target column and in batches - arrow_table = delta_table.to_pyarrow_table() + arrow_table = delta_to_pyarrow_dataset(delta_table).to_table() except Exception as e: - # Leave the cache dirty so the next call reads the table again. + # Stay dirty: the loaded table is cached, so a clean cache would never + # be refreshed and stored ids would read as unstored (ITL-864). + self._cache_dirty[record_key] = True raise self._read_failed(record_path) from e if arrow_table.num_rows == 0: @@ -595,7 +619,7 @@ def _refresh_existing_ids_cache(self, record_path: tuple[str, ...]) -> None: self._existing_ids_cache[record_key] = set() else: existing_ids = cast( - set[str], set(arrow_table[self.RECORD_ID_COLUMN].to_pylist()) + set[bytes], set(arrow_table[self.RECORD_ID_COLUMN].to_pylist()) ) self._existing_ids_cache[record_key] = existing_ids @@ -604,12 +628,15 @@ def _refresh_existing_ids_cache(self, record_path: tuple[str, ...]) -> None: f"Refreshed existing IDs cache: {len(self._existing_ids_cache)} IDs" ) - def _get_existing_ids(self, record_path: tuple[str, ...]) -> set[str]: + def _get_existing_ids(self, record_path: tuple[str, ...]) -> set[bytes]: """Get the set of existing IDs in the Delta table, using cache when possible.""" record_key = self._get_record_key(record_path) + # A table loaded by a read is cached before any ids are, so a missing + # id entry needs a refresh as much as a dirty one does. if ( self._cache_dirty.get(record_key) or record_key not in self._delta_table_cache + or record_key not in self._existing_ids_cache ): self._refresh_existing_ids_cache(record_path) return self._existing_ids_cache.get(record_key) or set() @@ -873,7 +900,7 @@ def _add_to_pending_batch(self, record_path: tuple[str, ...], table: pa.Table): else: self._pending_batches[record_key] = pa.concat_tables([pending_batch, table]) - pending_ids = cast(list[str], table[self.RECORD_ID_COLUMN].to_pylist()) + pending_ids = cast(list[bytes], table[self.RECORD_ID_COLUMN].to_pylist()) self._pending_record_ids[record_key].update(pending_ids) def _should_auto_flush(self, record_path: tuple[str, ...]) -> bool: @@ -884,6 +911,97 @@ def _should_auto_flush(self, record_path: tuple[str, ...]) -> bool: and self._pending_batches[record_key].num_rows >= self.batch_size ) + def _pending_not_stored(self, record_path: tuple[str, ...]) -> pa.Table | None: + """Return the pending rows a flush would insert at ``record_path``. + + A flush inserts only rows whose record id is not stored yet (the merge is + ``when_not_matched_insert_all``), so a pending row with a stored id is + never returned by a read: the stored row is. + + Args: + record_path: Path components relative to this database view. + + Returns: + The pending rows whose record id is not stored, or ``None`` if there + are none. + """ + pending = self._pending_batches.get(self._get_record_key(record_path)) + if pending is None or pending.num_rows == 0: + return None + stored_ids = self._get_existing_ids(record_path) + if stored_ids: + # The mask is built from the pending ids against the cached set, so its + # cost follows the pending batch, not the stored table. A null id is + # never matched by the merge (``null = null`` is not true), so its row + # is kept. + keep = [ + record_id is None or record_id not in stored_ids + for record_id in pending[self.RECORD_ID_COLUMN].to_pylist() + ] + pending = pending.filter(pa.array(keep, type=pa.bool_())) + return pending if pending.num_rows > 0 else None + + def _combine_stored_and_pending( + self, + stored: pa.Table | None, + pending: pa.Table | None, + record_id_column: str | None, + ) -> pa.Table | None: + """Combine stored rows with pending rows into one read result. + + Combined rows have the schema a flush would give them: both parts take + their large layouts (a stored table reads back large), a column missing + on one side is null-filled, a ``null``-typed column takes the other + side's type, nullability is merged (nullable if either side is), and + struct children are merged, as Delta's merge evolves the stored schema. + A column whose types still differ raises ``pyarrow.ArrowTypeError``; + nothing is widened. A single part is returned as it is. + + Args: + stored: Rows read from the Delta table, or ``None``. + pending: Pending rows that are not stored (see ``_pending_not_stored``), + already filtered by the read's condition, or ``None``. + record_id_column: Name to give the record id column, or ``None`` to + drop it. + + Returns: + The combined rows, or ``None`` if there are none. + """ + parts = [t for t in (stored, pending) if t is not None and t.num_rows > 0] + if not parts: + return None + if len(parts) == 1: + table = parts[0] + else: + table = pa.concat_tables( + [self._large_layout(t) for t in parts], promote_options="default" + ) + return self._handle_record_id_column(table, record_id_column) + + @staticmethod + def _large_layout(table: pa.Table) -> pa.Table: + """Return ``table`` with every column in its large layout. + + Only layouts change (``string`` to ``large_string``, ``list`` to + ``large_list``, recursively); ``null`` stays ``null``, and names, + nullability and metadata are kept. + + Args: + table: The table to normalize. + + Returns: + The table with large-normalized column types. + """ + schema = table.schema + for index, field in enumerate(table.schema): + schema = schema.set( + index, + field.with_type( + arrow_utils.normalize_to_large_types(field.type, keep_null=True) + ), + ) + return table if schema.equals(table.schema) else table.cast(schema) + def get_all_records( self, record_path: tuple[str, ...], @@ -893,45 +1011,33 @@ def get_all_records( """ Get all records from both pending batch and Delta table. + Every read method returns the rows a flush would leave: the stored rows + plus the pending rows whose record id is not stored yet, as seen through + any handle of this ``at()`` family. + Returns: Combined Arrow table with all records, or None if no records exist Raises: StoreReadError: If the Delta table exists but could not be read. """ - record_key = self._get_record_key(record_path) - tables_to_combine = [] # Add Delta table data if (delta_table := self._get_delta_table(record_path)) is not None: try: - delta_table_data = delta_table.to_pyarrow_dataset( - as_large_types=True + delta_table_data = delta_to_pyarrow_dataset( + delta_table, as_large_types=True ).to_table() except Exception as e: raise self._read_failed(record_path) from e if delta_table_data.num_rows > 0: tables_to_combine.append(delta_table_data) - # Add pending batch data - if ( - retrieve_pending - and (pending_batch := self._pending_batches.get(record_key)) is not None - ): - if pending_batch.num_rows > 0: - tables_to_combine.append(pending_batch) - - if not tables_to_combine: - return None - - if len(tables_to_combine) == 1: - table_to_return = tables_to_combine[0] - else: - table_to_return = pa.concat_tables(tables_to_combine) - - # Handle record_id_column if specified - return self._handle_record_id_column(table_to_return, record_id_column) + # Add the pending rows a flush would insert + pending = self._pending_not_stored(record_path) if retrieve_pending else None + stored = pa.concat_tables(tables_to_combine) if tables_to_combine else None + return self._combine_stored_and_pending(stored, pending, record_id_column) def get_records_with_column_value( self, @@ -942,9 +1048,6 @@ def get_records_with_column_value( ) -> "pa.Table | None": if flush: self.flush_batch(record_path) - # check if record_id is found in pending batches - record_key = self._get_record_key(record_path) - pending_batch = self._pending_batches.get(record_key) if isinstance(column_values, Mapping): # Convert Mapping to list of tuples @@ -958,14 +1061,30 @@ def get_records_with_column_value( for next_expression in expressions[1:]: combined_expression = combined_expression & next_expression - if pending_batch is not None: - filtered_table = pending_batch.filter(combined_expression) - return self._handle_record_id_column(filtered_table, record_id_column) - - # Now check the Delta table + pending = self._pending_not_stored(record_path) delta_table = self._get_delta_table(record_path) + + # A side without a queried column holds null there once flushed, and + # ``column == value`` never matches null, so it contributes no rows. A + # column neither side has is unknown: the filter raises as before. + queried = {c for c, _ in pair_list} + stored_columns = ( + set() + if delta_table is None + else {field.name for field in delta_table.schema().fields} + ) + pending_columns = set() if pending is None else set(pending.column_names) + if queried <= stored_columns | pending_columns: + if not queried <= pending_columns: + pending = None + if not queried <= stored_columns: + delta_table = None + + if pending is not None: + pending = pending.filter(combined_expression) + if delta_table is None: - return None + return self._combine_stored_and_pending(None, pending, record_id_column) # Use schema-preserving read try: @@ -973,11 +1092,7 @@ def get_records_with_column_value( except _DeltaTableReadError as e: raise self._read_failed(record_path) from e.__cause__ - if len(result) == 0: - return None - - # Handle (remove/rename) the record id column before returning - return self._handle_record_id_column(result, record_id_column) + return self._combine_stored_and_pending(result, pending, record_id_column) def get_record_by_id( self, @@ -1001,9 +1116,12 @@ def get_record_by_id( self.flush_batch(record_path) record_id = coerce_record_id(record_id) - # check if record_id is found in pending batches + # Serve a pending record only if its id is not stored: a flush keeps the + # stored row and drops the pending one. record_key = self._get_record_key(record_path) - if record_id in self._pending_record_ids[record_key]: + if record_id in self._pending_record_ids[ + record_key + ] and record_id not in self._get_existing_ids(record_path): # Return the pending record after removing the entry id column pending_batch = self._pending_batches[record_key] assert pending_batch is not None, "Pending batch should not be None" @@ -1070,22 +1188,28 @@ def get_records_by_ids( if len(record_ids) == 0: return None - # check inside the batch delta_table = self._get_delta_table(record_path) - if delta_table is None: - return None - # Use schema-preserving read with filters - filter_expr = self._create_record_ids_filter(record_ids_list) - try: - result = self._read_delta_table(delta_table, filters=filter_expr) - except _DeltaTableReadError as e: - raise self._read_failed(record_path) from e.__cause__ - - if len(result) == 0: - return None - - # Handle record_id column based on parameter - return self._handle_record_id_column(result, record_id_column) + # Reading the stored ids drops the pending rows a flush would not insert; + # it raises StoreReadError when the stored table cannot be read. + pending = self._pending_not_stored(record_path) + result = None + if delta_table is not None: + # Use schema-preserving read with filters + filter_expr = self._create_record_ids_filter(record_ids_list) + try: + result = self._read_delta_table(delta_table, filters=filter_expr) + except _DeltaTableReadError as e: + raise self._read_failed(record_path) from e.__cause__ + + if pending is not None: + id_type = pending.schema.field(self.RECORD_ID_COLUMN).type + pending = pending.filter( + pc.is_in( + pending[self.RECORD_ID_COLUMN], + value_set=pa.array(record_ids_list, type=id_type), + ) + ) + return self._combine_stored_and_pending(result, pending, record_id_column) def _read_delta_table( self, @@ -1112,7 +1236,7 @@ def _read_delta_table( filter_expr = None # Use to_pyarrow_dataset with as_large_types for Polars compatible arrow table loading try: - dataset = delta_table.to_pyarrow_dataset(as_large_types=True) + dataset = delta_to_pyarrow_dataset(delta_table, as_large_types=True) except Exception as e: raise _DeltaTableReadError from e if filters and expression is None: @@ -1128,7 +1252,7 @@ def _read_delta_table( f"Unsupported filter operation: {op}. Falling back to table-level filter application which may be less efficient." ) # Fallback to table-level filtering - return dataset.to_table()(filters=filters) + return dataset.to_table()(filters=filters) # pyright: ignore[reportCallIssue] # pre-existing bug; unreachable, this class builds only "=" and "in" filters if filter_expr is None: filter_expr = expr @@ -1453,6 +1577,9 @@ def at(self, *path_components: str) -> "DeltaTableDatabase": _scoped_path=new_scoped_path, _shared_pending_batches=root_instance._pending_batches, _shared_pending_record_ids=root_instance._pending_record_ids, + _shared_delta_table_cache=root_instance._delta_table_cache, + _shared_existing_ids_cache=root_instance._existing_ids_cache, + _shared_cache_dirty=root_instance._cache_dirty, ) def list_sources(self) -> list[tuple[str, ...]]: diff --git a/src/orcapod/databases/storage_utils.py b/src/orcapod/databases/storage_utils.py index 544d20e69..69410c432 100644 --- a/src/orcapod/databases/storage_utils.py +++ b/src/orcapod/databases/storage_utils.py @@ -1,20 +1,30 @@ """ Storage utility helpers for DeltaTableDatabase. -Provides two public functions: +Provides three public functions: - parse_base_path: Converts any path type (str, Path, UPath) into a URI string and a merged dict of deltalake-compatible storage_options. - is_cloud_uri: Returns True for cloud storage schemes (s3://, gs://, etc.). +- delta_to_pyarrow_dataset: Opens a Delta table as a pyarrow dataset, reading local + tables through a native filesystem. """ from __future__ import annotations import logging -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any + +from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: from pathlib import Path + + import deltalake + import pyarrow.dataset as pa_ds + import pyarrow.fs as pa_fs from upath import UPath +else: + pa_fs = LazyModule("pyarrow.fs") logger = logging.getLogger(__name__) @@ -38,6 +48,42 @@ def is_cloud_uri(uri: str) -> bool: return scheme in _CLOUD_SCHEMES +def delta_to_pyarrow_dataset( + delta_table: deltalake.DeltaTable, **kwargs: Any +) -> pa_ds.Dataset: + """Open a Delta table as a pyarrow dataset, natively for local tables. + + By default ``DeltaTable.to_pyarrow_dataset`` reads through + ``PyFileSystem(DeltaStorageHandler)``, a filesystem implemented in Python. Arrow + finishes tearing down a scan on a thread-pool worker after the scan returns. If that + worker releases a Python-backed buffer while the interpreter is finalizing, the process + hangs at exit in Arrow's thread-pool destructor, or aborts on Linux with Python + 3.11/3.12 (ITL-892; apache/arrow#50482, apache/arrow#54536). + + A local table is therefore read through a native pyarrow filesystem rooted at the + table, which returns the same schema and values without Python file objects. Any other + URI, a cloud store or a ``filesystem`` passed by the caller, keeps deltalake's reader. + + ``table_uri`` is percent-encoded: deltalake 1.0.2 and 1.2.1 return a bare path such as + ``/data/a%20b/``, and 1.6.6 returns ``file:///data/a%20b/``. pyarrow's own URI parser + decodes it to the native path. A root that is neither a ``file://`` URI nor an absolute + POSIX path (a Windows drive path, untested) keeps deltalake's reader. + + Args: + delta_table: The Delta table to open. + **kwargs: Passed to ``DeltaTable.to_pyarrow_dataset`` unchanged. + + Returns: + The pyarrow dataset over the table's current version. + """ + uri = delta_table.table_uri + file_uri = f"file://{uri}" if uri.startswith("/") else uri + if kwargs.get("filesystem") is None and file_uri.startswith("file://"): + filesystem, root = pa_fs.FileSystem.from_uri(file_uri) + kwargs["filesystem"] = pa_fs.SubTreeFileSystem(root, filesystem) + return delta_table.to_pyarrow_dataset(**kwargs) + + def _extract_upath_options(upath: UPath) -> dict[str, str]: """ Extract storage options from a UPath and translate to deltalake's AWS_* format. diff --git a/superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md b/superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md new file mode 100644 index 000000000..546c5e252 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md @@ -0,0 +1,174 @@ +# ITL-864 DeltaTableDatabase read-your-writes Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Every read method of `DeltaTableDatabase` returns exactly the rows a flush would leave, through every handle of an `at()` family, so `node.get_all_records()` returns every row right after the first run on a Delta store. + +**Architecture:** Two defects, one contract. (1) `at()` handles share the root's pending batches, and any handle's `flush()` writes every pending key, but each handle keeps its own `_delta_table_cache`, so a handle that loaded a table earlier keeps serving a stale `DeltaTable` snapshot after a sibling flushed. The fix shares `_delta_table_cache`, `_existing_ids_cache` and `_cache_dirty` across the family, exactly as pending state is already shared. (2) The read methods merge pending and stored rows inconsistently (`get_records_by_ids` ignores pending; `get_records_with_column_value` ignores stored whenever a pending batch exists; overlapping ids show twice). The fix: every read returns stored rows plus pending rows whose record id is not stored (Delta's flush is `when_not_matched_insert_all`, so the stored row wins), `None` when empty. + +**Tech Stack:** Python, pyarrow, deltalake, pytest, `uv run`. + +Approved by the coordinator (option A, Nov 10, 2026). No public API, hash, stored schema or dependency change. Error handling is untouched (PR #290 / ITL-884 rewrites it in the same file; merge main after it lands). + +--- + +## Reproduction (Step 1 of the brief) + +`PipelineJob`, `DictSource` of 2 or 3 ints, `int -> int` pod, `DeltaTableDatabase`: + +| store | orchestrator | n | `get_all_records` | `iter_data` | +|---|---|---|---|---| +| delta | sync | 2 / 3 | 1 / 1 | 2 / 3 | +| delta | async | 2 / 3 | 1 / 2 | 2 / 3 | +| memory | both | 2 / 3 | 2 / 3 | 2 / 3 | + +Trace: the result-store handle (`at("pipeline", "_result")`) flushes the pipeline table's rows; the node's pipeline handle (`at("pipeline")`) cached the table at version 0 and still reads version 0 while the table is at version 2. Its pending batch is empty; a fresh handle reads all rows. + +## Contract and per-method findings (database level) + +Contract: a read through a handle reflects all writes of its `at()` family, pending and flushed together, and equals what a flush would leave: no duplicates, no missing rows. + +| case | `get_all_records` | `get_record_by_id` | `get_records_by_ids` | `get_records_with_column_value` | +|---|---|---|---|---| +| A sibling handle flushed | stale | stale | stale | stale | +| B stored + pending, one handle | ok | ok | misses pending | misses stored when pending exists; empty table instead of `None` | +| C pending id already stored, `skip_duplicates=False` | duplicate | returns pending row, flush keeps stored | (dup) | (dup) | +| D same, `skip_duplicates=True` | ok | ok | ok | ok | +| `record_id_column=` | returns (no hang) | | | | + +Other backends: `InMemoryArrowDatabase` shares `_tables` across `at()` and `ConnectorArrowDatabase` re-reads committed rows on every call, so neither goes stale; both merge pending + committed in every method. Both flush with the NEW row winning on id overlap (Delta keeps the stored row) and show both rows before flush under `skip_duplicates=False`. Report only. + +Thread safety: the async orchestrator runs only the user function in `run_in_executor` (`core/executors/local.py`); every database read and write happens on the event-loop thread, as for the already-shared pending dicts, so sharing the caches adds no cross-thread access. + +## File structure + +- Modify: `src/orcapod/databases/delta_lake_databases.py` — `__init__` / `at()` share caches; new helpers `_pending_not_stored` and `_combine_stored_and_pending`; the four read methods use them. +- Create: `tests/test_databases/test_delta_read_your_writes.py` — database-level regression tests. +- Create: `tests/test_pipeline/test_delta_node_records_after_run.py` — `PipelineJob` regression, both orchestrators. +- Modify: `DESIGN_ISSUES.md` — one entry appended at the end. + +--- + +### Task 1: Failing database-level tests + +**Files:** Create `tests/test_databases/test_delta_read_your_writes.py` + +- [ ] **Step 1: Write the tests** + +Helpers: `_rows(db, ids, vals, groups)` builds a table with `__record_id` (`large_binary`), `v` (`int64`), `g` (`int64`); `_values(table)` returns sorted `v` or `None`. A `READS` parametrization maps each read method to a call and its expected values: + +```python +READS = { + "get_all_records": (lambda db: db.get_all_records(PATH), [1, 2]), + "get_record_by_id": (lambda db: db.get_record_by_id(PATH, b"b"), [2]), + "get_records_by_ids": (lambda db: db.get_records_by_ids(PATH, [b"a", b"b"]), [1, 2]), + "get_records_with_column_value": ( + lambda db: db.get_records_with_column_value(PATH, {"g": 0}), [1, 2]), +} +``` + +Tests: +1. `test_sibling_handle_sees_rows_flushed_by_another_handle[method][flusher]`: `root = DeltaTableDatabase(tmp_path)`, `h1, h2 = root.at("s"), root.at("s")`; `h1` writes `a=1` with `flush=True` and reads once (loads its snapshot); `h2` writes `b=2`; read through `h1` and through `root` (path `("s",) + PATH`) before flush; flush through `flusher` in (`h2`, `root`, `h1`); read through `h1`, `root` and a new `root.at("s")` after flush. All equal the expected values. +2. `test_pending_and_stored_rows_are_both_returned[method]`: one handle, stored `a=1 (g=0), b=2 (g=1)`, pending `c=3 (g=1)`: `get_records_by_ids([a, c]) == [1, 3]`, `get_records_with_column_value({"g": 1}) == [2, 3]`, `{"g": 0} == [1]`, `get_all_records == [1, 2, 3]`, `get_record_by_id(c) == [3]`. +3. `test_no_match_returns_none`: stored + pending, `get_records_with_column_value({"g": 9})` and `get_records_by_ids([b"zz"])` are `None`. +4. `test_pending_row_with_stored_id_reads_as_flush_keeps_it[skip_duplicates]`: stored `a=1`; pending add of `a=10, d=4`; before flush each method returns `a=1` and `d=4` once (`get_all_records == [1, 4]`, `get_record_by_id(a) == [1]`, `get_records_by_ids([a, d]) == [1, 4]`, `get_records_with_column_value({"g": 0}) == [1, 4]`); after `flush()` the same values. +5. `test_record_id_column_is_returned_for_merged_rows`: stored + pending, `get_all_records(PATH, record_id_column="rid")` has column `rid` with all three ids and no `__record_id`; same for `get_records_by_ids` and `get_records_with_column_value`. + +- [ ] **Step 2: Run, verify the expected failures** + +Run: `uv run pytest tests/test_databases/test_delta_read_your_writes.py -v` +Expected: test 1 fails for every method after flush; test 2 fails for `get_records_by_ids` / `get_records_with_column_value`; test 3 fails (empty table, not `None`); test 4 fails for `skip_duplicates=False`. + +### Task 2: Failing PipelineJob test + +**Files:** Create `tests/test_pipeline/test_delta_node_records_after_run.py` + +- [ ] **Step 1: Write the test** + +```python +@pytest.mark.parametrize("orchestrator", [SyncPipelineOrchestrator, AsyncPipelineOrchestrator]) +def test_node_returns_every_row_right_after_first_run(tmp_path, orchestrator): + source = op.sources.DictSource([{"id": i, "x": 10 * i} for i in range(3)], tag_columns=["id"]) + + @op.function_pod(output_keys=["result"]) + def double(x: int) -> int: + return x * 2 + + job = PipelineJob(store=DeltaTableDatabase(tmp_path)) + with job: + double.pod(source, label="double") + job.run(orchestrator=orchestrator()) + + records = job.nodes["double"].get_all_records() + assert records is not None + assert sorted(records.column("result").to_pylist()) == [0, 20, 40] +``` + +- [ ] **Step 2: Run, verify it fails** (`1 != 3` rows) for both orchestrators. + +- [ ] **Step 3: Commit the failing tests** + +```bash +git add tests/test_databases/test_delta_read_your_writes.py tests/test_pipeline/test_delta_node_records_after_run.py +git commit -m "test(databases): regression tests for Delta reads missing own writes (ITL-864)" +``` + +### Task 3: Share the snapshot caches across the `at()` family + +**Files:** Modify `src/orcapod/databases/delta_lake_databases.py` (`__init__`, `at()`) + +- [ ] **Step 1:** Add keyword-only private constructor parameters `_shared_delta_table_cache`, `_shared_existing_ids_cache`, `_shared_cache_dirty`, used when given (same pattern as `_shared_pending_batches`); `at()` passes the root instance's dicts. Comment why: any family member's `flush()` writes every pending key and refreshes the cache entry for it, so the snapshot cache must be family-wide or a sibling keeps reading the old version. +- [ ] **Step 2:** Run Task 1 test 1 and the Task 2 test: expected PASS for test 1 and Task 2. + +### Task 4: Merge stored and pending rows the way flush does + +**Files:** Modify `src/orcapod/databases/delta_lake_databases.py` (read methods) + +- [ ] **Step 1:** Add helpers: + +```python +def _pending_not_stored(self, record_path): + """Pending rows whose record id is not stored; flush inserts only these.""" + pending = self._pending_batches.get(self._get_record_key(record_path)) + if pending is None or pending.num_rows == 0: + return None + stored_ids = self._get_existing_ids(record_path) + if stored_ids: + id_type = pending.schema.field(self.RECORD_ID_COLUMN).type + pending = pending.filter(pc.invert(pc.is_in( + pending[self.RECORD_ID_COLUMN], value_set=pa.array(list(stored_ids), type=id_type)))) + return pending if pending.num_rows > 0 else None + +def _combine_stored_and_pending(self, stored, pending, record_id_column): + parts = [t for t in (stored, pending) if t is not None and t.num_rows > 0] + if not parts: + return None + table = parts[0] if len(parts) == 1 else pa.concat_tables(parts) + return self._handle_record_id_column(table, record_id_column) +``` + +- [ ] **Step 2:** `get_all_records`: pending part uses `_pending_not_stored`. `get_record_by_id`: serve from pending only when the id is pending and not stored. `get_records_by_ids`: stored read (existing try/except unchanged) plus pending filtered by the ids. `get_records_with_column_value`: stored read (existing try/except unchanged) plus pending filtered by the expression; `None` when empty. +- [ ] **Step 3:** Run `uv run pytest tests/test_databases tests/test_pipeline -q`: all pass. +- [ ] **Step 4:** Commit `fix(databases): Delta reads return every row the handle family wrote (ITL-864)`. + +### Task 5: Design issue entry, full verification, PR + +- [ ] Append one `DESIGN_ISSUES.md` entry (resolved, with **Fix:**; note that writes from other processes are still not seen until a handle reloads the table — follow-up for the coordinator to file). +- [ ] `uv run --with ruff ruff check .`, `ruff format --check .`, `uv run --with basedpyright basedpyright` (exit code), `uv run pytest tests/` (golden hashes unchanged). +- [ ] Merge `origin/main`, rerun if it moved, push, PR "Fixes ITL-864", `orca linear attach`, `orca worktree set --pr`. + +--- + +## Review rounds and continuation notes (successor-readable) + +Done and pushed (verify, do not redo): +- Round 1 (e9c78b5c): a failed ID refresh keeps the cache dirty; `get_records_by_ids` refresh shares the stored-read handler (8f9adf8b). +- Round 2 (8f9adf8b): refresh when the record key has no ID-cache entry; stored + pending combined with `normalize_table_to_large_types` + `pa.concat_tables(promote_options="default")` (b8bcc817). Tests: `test_fresh_handle_drops_pending_rows_with_stored_ids`, evolved-schema tests. + +Round 3 (CodeRabbit review of b8bcc817, outside-diff, `get_records_with_column_value`), approved fix: +- A side (stored schema, read without loading data; pending column names) that lacks a queried column contributes no rows; after a flush its rows hold null there and `col == value` never matches null. A column on neither side keeps today's `ArrowInvalid` ("No match for FieldRef"). +- Tests: `test_query_on_column_one_side_lacks_matches_read_after_flush[column_only_pending|column_only_stored]`, `test_query_on_column_neither_side_has_raises[flushed]`. +- Also the round-1 outside-diff trivial: `_add_to_pending_batch` cast `list[str]` -> `list[bytes]`. +- Reply: signed top-level PR comment (no thread for outside-diff findings), and one for the round-1 trivial. + +Then: merge origin/main (b197544e); wait for PR #290 (ITL-884, StoreReadError) to merge; merge main again, take #290's error handling around `_pending_not_stored` / `_combine_stored_and_pending`; a failed ID refresh must raise `StoreReadError` on every read method; full suite; worker_done only after the post-#290 merge is green. diff --git a/superpowers/plans/2026-11-10-itl-892-exit-hang.md b/superpowers/plans/2026-11-10-itl-892-exit-hang.md new file mode 100644 index 000000000..2cdd8d916 --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-892-exit-hang.md @@ -0,0 +1,295 @@ +# ITL-892 Exit Hang Investigation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Reproduce and characterise the interpreter-exit hang after a Delta read (ITL-892), evaluate +mitigations with hang rates over at least 40 runs each, and report to the coordinator before any +`src/` or `uv.lock` change. + +**Architecture:** One timeout harness (`harness.py`) runs a target script N times as child processes, +counts exits that miss the deadline, and samples the native stack of the first hung child (`sample` +on macOS, `py-spy`/`gdb` not required). Target scripts are small and parameterised by argv so one +harness drives every matrix cell. Version cells run in throwaway environments +(`uv run --no-project --with pyarrow==X --with deltalake==Y`) so `uv.lock` never changes. + +**Tech Stack:** Python 3.11/3.12/3.13 (uv-managed), pyarrow, deltalake, polars, Orcapod +`PipelineJob` on `DeltaTableDatabase`, GitHub Actions (temporary, non-required job) for Linux. + +Constraints (from the dispatch brief): no change to `uv.lock` or `src/` before the coordinator +approves a mitigation. All experiments live in `superpowers/specs/itl-892-exit-hang/`. Draft PR body +says `Refs ITL-892`. + +--- + +### Task 1: Harness and minimal pure reproduction + +**Files:** +- Create: `superpowers/specs/itl-892-exit-hang/harness.py` +- Create: `superpowers/specs/itl-892-exit-hang/repro_pure.py` + +- [ ] Write `repro_pure.py`: builds a small Delta table in a temp dir and performs one operation + chosen by `--op` (`write`, `merge`, `to_table`, `to_dataset`, `dataset_scan`, `write_and_read` + — the last is the ITL-864 worker's script: 1 write, 4 merges, a `to_pyarrow_dataset().to_table()` + after each). Flags: `--import-polars`, `--cpu-threads N` (`pa.set_cpu_count`), `--io-threads N` + (`pa.set_io_thread_count`), `--mitigation {none,del,gc,os_exit,pool_shutdown}`, `--mp {none,fork,spawn}`. + Prints `done` with flush then returns. +- [ ] Write `harness.py`: `harness.py N TIMEOUT -- `; runs N children with up to `-j` in + parallel, each in its own session; a child that misses TIMEOUT is sampled once (first hang only, + `sample 2`) then killed; prints `hung/N`, exit-code histogram, whether `done` was printed + before the hang, and writes a JSON line to `results.jsonl` with the cell label. +- [ ] Run the ITL-864 script 40 times on Python 3.12 with the locked versions. Expected: a hang rate + near 50% and a stack in `ThreadPool::~ThreadPool`. If 0/40, vary parallelism (-j1 vs -j8) and + table size before concluding. + +### Task 2: Characterise the trigger + +- [ ] Run each `--op` alone, 40 runs each: which operation is necessary. +- [ ] With the triggering op: `--cpu-threads 1`, `--io-threads 1`, env `OMP_NUM_THREADS=1`, + `ARROW_IO_THREADS=1`, `RAYON_NUM_THREADS=1`; `--import-polars`; deltalake without pyarrow + imported where the API allows (`DeltaTable.to_pyarrow_*` needs pyarrow; use `write_deltalake` + with an arro3 table / `QueryBuilder` to separate the runtimes); `--mp fork/spawn`. +- [ ] Python 3.11 and 3.13 at the triggering cell. + +### Task 3: Orcapod paths + +**Files:** +- Create: `superpowers/specs/itl-892-exit-hang/repro_orcapod.py` + +- [ ] Modes: `pipeline-sync`, `pipeline-async` (a two-source `PipelineJob` with one function pod on + `DeltaTableDatabase`, then `get_all_records`), `plain-read` (`DeltaTableDatabase` write, flush, + read). 40 runs each on 3.12 via the project environment; 3.11/3.13 via `uv run --isolated --python`. + +### Task 4: Version bisect and upstream search + +- [ ] Grid over pyarrow {20.0.0, 21, 22, 23.0.1, 24, 25.0.1, 26.0.1} × deltalake {1.0.2, 1.1.x, + 1.2.x, 1.3.x, 1.4.2, 1.5.1, 1.6.6} at the triggering cell, 40 runs each (narrow by halving once a + boundary is visible); record newest non-hanging and oldest hanging combinations. +- [ ] Search apache/arrow and delta-io/delta-rs issues (WebSearch, `gh search issues`) for + `ThreadPool` destructor / exit hang / `__cxa_finalize`; link matches. + +### Task 5: Mitigations + +- [ ] (a) Best version combination from Task 4: copy the worktree to a scratch dir, `uv lock + --upgrade-package`, run golden hash tests and the full suite there (never in this worktree). +- [ ] (b) Orcapod-side shutdown, prototyped in the repro scripts only: drop `DeltaTable`/dataset + references + `gc.collect()`; an `atexit` hook; pyarrow pool shutdown if exposed; `os._exit` after + flush (for comparison only). +- [ ] (c) Alternative read path: `DeltaTable.to_pyarrow_table`, `QueryBuilder`/arro3, polars + `scan_delta`/`read_delta`, reading the parquet files listed by `DeltaTable.file_uris()` with + pyarrow directly. +- [ ] (d) Documentation workaround text. +- [ ] 40 runs per option; record hang rate. + +### Task 6: Linux + +- [ ] Docker is not installed on this machine. Push a temporary, non-required workflow + `.github/workflows/itl-892-exit-hang-probe.yml` (`workflow_dispatch` + push on this branch only) + running the harness on ubuntu-latest for 3.11/3.12/3.13. Remove it before the PR leaves draft. + +### Task 7: Report + +**Files:** +- Create: `superpowers/specs/itl-892-exit-hang/FINDINGS.md` +- Modify: `DESIGN_ISSUES.md` (one appended entry for ITL-892) + +- [ ] Findings: hang-rate tables, stack, trigger, versions, upstream links, options (a)-(d) with + evidence and cost, recommendation, proposed regression check and its reliability. +- [ ] Commit (Conventional Commits, Co-Authored-By trailer), push, open draft PR "Refs ITL-892". +- [ ] `ask` the coordinator with results and options; do not implement a mitigation before approval. + +--- + +## Phase 2: approved mitigation, option (c) plus (d) + +Approved by the coordinator after reading FINDINGS.md (November 10, 2026). The scope is the +read path only. Do not implement (b3); record it in DESIGN_ISSUES. Keep the `src/` change to +one helper plus one-line call-site changes, because PR #290 (ITL-884) and PR #293 (ITL-864) +rewrite the same read methods and merge first. PR body: `Fixes ITL-892`. + +### Task 8: Failing regression test + +**Files:** +- Create: `tests/test_databases/test_delta_exit.py` + +- [ ] Each case starts `RUNS = 20` fresh interpreters. The child script is inline and uses + only the public API: + - `DeltaTableDatabase` `add_records` + `flush`, then `get_all_records` on a new handle. + - `DeltaTableSource` over a table written with `write_deltalake`. +- [ ] The deadline is 120 s; run 4 children at a time. +- [ ] Fail if any run misses the deadline or exits non-zero. The message names the defect: + "process did not exit after a Delta read (ITL-892)". +- [ ] Run it: expect FAIL on the current code (macOS: about 53% / 8% per run). + +### Task 9: Equivalence test (type strictness) + +**Files:** +- Create: `tests/test_databases/test_delta_native_fs_read.py` + +- [ ] Write stores through Orcapod's public API: a `PipelineJob` on `DeltaTableDatabase` with + function pods returning `int`, `Path`, a dataclass, `numpy.ndarray`, `pandas.DataFrame`, + `list[int]` and a nullable value. For every table in `db.list_sources()`, assert that + `db.get_all_records(path)` equals deltalake's default reader + (`DeltaTable(uri).to_pyarrow_dataset(as_large_types=True).to_table()`): schema equality with + `check_metadata=True`, then values. +- [ ] `DeltaTableSource` over a hand-written table: narrow ints, decimal, tz timestamp, nested + list and struct, non-nullable field, partitioned by a column with values `"a b"`, `"x/y"`, + `"é%"` and `None`. Its `as_table()` must equal the same `DeltaTableSource` built under a + monkeypatched default reader. +- [ ] Passes before and after the change; the guard is that it keeps passing after. + +### Task 10: Helper and call sites + +**Files:** +- Modify: `src/orcapod/databases/storage_utils.py` (new `delta_to_pyarrow_dataset`) +- Modify: `src/orcapod/databases/delta_lake_databases.py` (3 call sites: + `_refresh_existing_ids_cache`, `get_all_records`, `_read_delta_table`) +- Modify: `src/orcapod/core/sources/delta_table_source.py` (1 call site) + +- [ ] `delta_to_pyarrow_dataset(delta_table, **kwargs)`: if `delta_table.table_uri` is a + local path (no scheme, or `file://`), pass + `filesystem=pa.fs.SubTreeFileSystem(, pa.fs.LocalFileSystem())`. Otherwise use + deltalake's default reader. Forward every other keyword unchanged. The docstring names + ITL-892 and the upstream issues. +- [ ] `to_pyarrow_table()` becomes `delta_to_pyarrow_dataset(dt).to_table()`; deltalake's own + `to_pyarrow_table` does exactly that, so behaviour is identical. +- [ ] Tasks 8 and 9 pass. The full suite and the golden hash tests are unchanged. + +### Task 11: Docs, changelog, design issues, cleanup + +- [ ] `docs/troubleshooting.md` (new; in the mkdocs nav): the process-exit note, the remaining + exposure for cloud stores, apache/arrow#50482, #54536 and PR #54467, and the workaround. +- [ ] `CHANGELOG.md` `[0.3.0]` entry. +- [ ] DESIGN_ISSUES DX1: status resolved, with a **Fix:** note, and (b3) recorded as the + considered alternative and a possible lazy hook for cloud stores. +- [ ] Delete `.github/workflows/itl-892-exit-hang-probe.yml`. +- [ ] Run ruff, basedpyright and the full suite. Merge `main` after #290 and #293 land. Mark + the PR ready, take it through CodeRabbit, and send worker_done with the delta-rs issue text. + +--- + +## Phase 3: CodeRabbit round 1 (resumed after a machine restart, November 11, 2026) + +State on arrival: Tasks 8-11 were done, committed and pushed (head 561f5999), CI was green and +the PR was ready for review. CodeRabbit round 1 (review 5855612064: 8 inline findings and 5 +nitpicks in the review body) was unanswered. The coordinator approved this plan. + +### Task 12: Percent-encoded table roots (behaviour fix) + +`DeltaTable.table_uri` is percent-encoded on every supported deltalake: 1.0.2 and 1.2.1 return +`/x/a%20b/t/`, and 1.6.6 returns `file:///x/a%20b/`. The helper passed it undecoded to +`SubTreeFileSystem`, so a store under `a b`, `pct%` or `é ü` was unreadable. `get_all_records` +swallowed the error, so the store looked empty. + +- [x] Failing test first, in `test_delta_native_fs_read.py`: for store roots `a b`, `pct%`, + `p%20q`, `é ü`, `x#y`, `q?r`, `semi;c` and `plus+` (skip a name the installed deltalake + itself cannot write), `DeltaTableDatabase.get_all_records` and `DeltaTableSource.as_table()` + equal the default reader. Assert a correct read, never an empty result. After #290 an + unreadable table raises `StoreReadError`. +- [x] Fix: `pa_fs.FileSystem.from_uri(uri if "://" in uri else "file://" + uri)`, and use the + result only when it is a `LocalFileSystem`. Windows drive paths are untested; record a note. +- [x] Fix the troubleshooting snippet the same way. + +### Task 13: Docstrings and research-script checks + +- [x] Docstrings in both new test files and in the `superpowers/specs/itl-892-exit-hang/*.py` + scripts. +- [x] `repro_orcapod.py`: assert that the fresh read returns 3 rows. `repro_pure.py`: exit + non-zero when the child process fails. Spot-check the patched mode. +- [x] Decline the `harness.py` worker-slot finding. The committed results came from that + harness, and the effect exists only after a hang, so it cannot touch the 0/100 fixed rates. + +### Task 14: Merge, replies, next round + +- [x] Merge origin/main (b197544e), run the full suite (6908 passed, 100 skipped, 15 xfailed), push, and reply on every thread, plus a signed + top-level comment for the nitpicks (done: replies on all 8 threads and issuecomment-6791908147). +- [ ] Next CodeRabbit round. Merge main again after #290 (watch for `StoreReadError`: the + tests must assert a raise or a correct read) and after #293. + +--- + +## Phase 4: rebuild on the stack #290 -> #293 -> #295 (November 11, 2026) + +Brief from the coordinator (task_5f115155397b): merge the rebuilt #293 +(`eywalker/itl-864-deltatabledatabase-reading-through-the`, head 06ecbd5c) into this branch, +keep all three behaviours, prove it by test, push, refresh the PR body. Brainstorming skipped: +the design was approved in Phase 2; this phase is a merge with the coordinator's DO list as +the spec. State on arrival: #290 has since been squash-merged to main as 1425b840, whose tree +equals #290's head 11efe638, so the stack is now main -> #293 -> #295. + +Rules to keep after the merge: +- #290: an unreadable table raises `StoreReadError`, never "no rows"; `_read_delta_table` + keeps main's signature (no message-only parameter) and raises `_DeltaTableReadError`; the + caller wraps it with the path. +- #293: reads return stored plus pending-not-stored rows; `_refresh_existing_ids_cache` + stays dirty on failure. +- #295: every scan opens through `delta_to_pyarrow_dataset`. +- Hazard: the filter must stay OUTSIDE the `_DeltaTableReadError` try blocks; a bad filter + (unknown column, type mismatch) is a caller bug and must not become `StoreReadError`. + +### Task 15: Merge #293's branch, then origin/main + +- [x] `git merge origin/eywalker/itl-864-deltatabledatabase-reading-through-the` (merge + commit). Resolve `delta_lake_databases.py`: take #293's code and swap each + `to_pyarrow_dataset`/`to_pyarrow_table` scan for `delta_to_pyarrow_dataset(...)` inside + the existing try blocks. CHANGELOG: one `[0.3.0] — unreleased` section with every entry. + DESIGN_ISSUES: all entries. +- [x] Fix the stale comment on the `dataset.to_table()(filters=...)` ignore: #290 did not + remove that line (still on main). Report it out of scope. +- [x] `git merge origin/main` (expected no content change; both conflicts took this side). +- [x] #293's `fail_id_refresh_once` fixture patched `to_pyarrow_table`; the refresh now opens + through `to_pyarrow_dataset` (no `as_large_types`), so the fixture fails that call instead. +- [ ] Merge the NEW #293 head (after #293 merges main 1425b840); the coordinator sends it. + +### Task 16: Interaction tests (`tests/test_databases/test_delta_native_fs_read.py`) + +- [x] Filtered reads (`get_record_by_id`, `get_records_by_ids`, + `get_records_with_column_value`) of an unreadable stored table raise `StoreReadError`. +- [x] A bad filter (`get_records_with_column_value` on an unknown column, or a value whose + type does not compare) raises the same non-`StoreReadError` error before and after a flush. +- [x] Filtered reads return stored plus pending rows equal to `get_all_records` filtered in + memory, with the native reader in use. +- [x] `test_delta_exit.py` still passes. +- [x] Done in `tests/test_databases/test_delta_native_fs_filtered_reads.py` (36 cases); moving the + filter inside the read's try block fails the 2 bad-filter cases (mutation check). + +### Task 17: Verify, push, PR body, CodeRabbit, worker_done + +- [ ] Full suite on 3.11/3.12/3.13, ruff check, ruff format --check, basedpyright (exit + code), golden tests unchanged (`git diff --stat origin/main -- tests/test_hashing/golden`). +- [ ] Push; PR body: stacked note at the top, merge notes, fresh counts. +- [ ] CI green, CodeRabbit round on the new head incl. review bodies. + +--- + +## Phase 5: merge the rebuilt #293 head 48678ca5 (November 11, 2026, task_8f9c4a65455b) + +The previous worker was killed by OOM after pushing 1f73fd10 (Tasks 15-16 done, clean tree). +The new #293 head 48678ca5 differs from the merged 06ecbd5c only by main's #298 (ITL-828, +merge commits 5aa47946 and 48678ca5): no #293 code change. Brainstorming skipped: the design +was approved in Phase 2 and the coordinator's DO list is the spec. + +### Task 18: Merge + +- [x] `git merge origin/eywalker/itl-864-deltatabledatabase-reading-through-the` (merge commit): f2c45c1d, conflicts only CHANGELOG + DESIGN_ISSUES (kept both sides). + Expected conflicts: `CHANGELOG.md` (keep one `[0.3.0] — unreleased` section with every entry), + `DESIGN_ISSUES.md` (keep all entries). No conflict expected in `delta_lake_databases.py`. +- [x] Check `git merge-base --is-ancestor origin/main HEAD`. +- [x] Golden: `git diff origin/main -- tests/test_hashing/golden` must be empty. +- [x] Commit and push straight away (WIP safety). + +### Task 19: Verify (ONE suite at a time, no xdist; memory is short) + +- [x] `uv run pytest tests/test_databases -q` first. +- [x] (7548 passed, 100 skipped, 19 xfailed each; use UV_PROJECT_ENVIRONMENT per interpreter so other uv calls cannot swap .venv mid-run) Full suite on 3.13, 3.12, 3.11 in turn: `uv run --python 3.1x pytest tests/ -q -p no:cacheprovider`. +- [x] (all clean, basedpyright exit 0) `uv run --with ruff ruff check .`, `ruff format --check .`, `uv run --with basedpyright basedpyright; echo $?`. + +### Task 20: PR, CI, CodeRabbit, report + +- [x] PR body: first line `Stacked on #293: until it merges, this PR's diff also shows its changes; review after it.`; update merge notes and counts. +- [ ] CI: wait at most 20 minutes (Actions outage), report state. +- [ ] CodeRabbit on the new head: inline threads and review bodies. #293-line findings: reply + they belong there and report; mechanical on this PR's lines: fix; others: `ask`. +- [ ] worker_done with: head, conflicts/resolutions, counts, golden diff, CI/CodeRabbit state, + ITL-919 watch-outs. + +Progress note: CodeRabbit review 5865082626 (on 1f73fd10): 3 inline findings, all on #293's lines; replied on each thread (r4543139876, r4543140113, r4543140365) that they belong to #293. diff --git a/superpowers/plans/2026-11-10-pr-293-coderabbit-id-cache-refresh.md b/superpowers/plans/2026-11-10-pr-293-coderabbit-id-cache-refresh.md new file mode 100644 index 000000000..6aea99fca --- /dev/null +++ b/superpowers/plans/2026-11-10-pr-293-coderabbit-id-cache-refresh.md @@ -0,0 +1,59 @@ +# PR #293 CodeRabbit follow-up: stored-ID cache refresh failures (ITL-864) + +Revision plan approved by the coordinator (task_355ce670f6bb), provided each finding +reproduces. Branch `eywalker/itl-864-deltatabledatabase-reading-through-the`. + +## Findings + +1. **Comment 4534785488 (Major), `delta_lake_databases.py:658`.** When + `_refresh_existing_ids_cache` fails after the Delta table loaded, it stores an empty ID + set and marks the cache **clean**. The loaded table is cached, so `_get_existing_ids` + never refreshes again: a pending row whose id is already stored is served next to the + stored row (a duplicate) by every read method. + *Fix:* the failure path marks the cache dirty, so the next read retries the refresh. + +2. **Comment 4534785497 (Minor), `delta_lake_databases.py:891`.** In + `get_records_by_ids`, `_pending_not_stored` refreshes the ID cache before the + method's stored-read `try`, so a refresh failure escapes while a stored-read failure is + logged and returns `None`. + *Fix:* run the refresh inside the same `try` as the stored read (one handler, no second + `return None`). After PR #290 (ITL-884) merges, that handler is gone and both failures + raise `StoreReadError` naming the table. + +## Steps (TDD) + +1. Regression tests in `tests/test_databases/test_delta_read_your_writes.py`, public + read methods only, with the backend's `DeltaTable.to_pyarrow_table` (used only by the + ID refresh) patched to fail once: + - stored `a=1`, flushed; pending `a=10`, `d=4`; one read fails in the refresh; every + read method then returns `a=1`, `d=4` and never `a=10`. + - stored `a=1`, flushed; pending `c=3`; `get_records_by_ids` with the refresh failing + follows the method's stored-read error behaviour (now `None`), and the retry returns + `[1, 3]`. +2. Run them, confirm both fail for the reasons above. +3. Apply the two fixes; rerun the file, then the full suite, ruff, basedpyright. +4. Commit, push, signed replies on both threads, wait for CI and CodeRabbit. + +## After PR #290 merges + +Merge `main`; in the read methods take #290's error handling (`StoreReadError`, no +`try`/`return None`) around this branch's merge helpers. #290's +`_refresh_existing_ids_cache` leaves `_cache_dirty` untouched on failure, which is still +clean when the refresh ran only because the table was not cached yet, so keep an explicit +`self._cache_dirty[record_key] = True` there. Change the test of finding 2 to expect +`StoreReadError` and extend it to every read method. Rerun the full suite (3.11, 3.12, +3.13), golden tests unchanged. + +## Round 2 (CodeRabbit review of 8f9adf8b; coordinator approved both behaviour fixes) + +- **Outside-diff (Major), `_get_existing_ids`:** a handle that loaded the table before any + stored ids were read (a fresh root over an existing table) never refreshed them, so pending + duplicates were served. Refresh also when the record key has no id entry. Test: + `test_fresh_handle_drops_pending_rows_with_stored_ids` (four read methods). +- **4535100450 (Major), `_combine_stored_and_pending`:** stored and pending rows with any + schema difference raised `ArrowInvalid`. Large-normalize both parts (`keep_null=True`), + then `pa.concat_tables(promote_options="default")`. Contract tests assert the read before + flush equals the read after flush (missing column either way, nullability merge, struct + child added, `string`/`list` layouts); type differences keep raising `ArrowTypeError`; a + `null`-typed new column is pinned (reads back, flush fails). +- **4535100459 (nit):** Google docstrings on the five tests and the wrapper. diff --git a/superpowers/plans/2026-11-11-itl-864-stack-on-290.md b/superpowers/plans/2026-11-11-itl-864-stack-on-290.md new file mode 100644 index 000000000..0e6a703b8 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-864-stack-on-290.md @@ -0,0 +1,60 @@ +# ITL-864 / PR #293: rebuild on top of PR #290 (ITL-884) and main + +Date: 2026-11-11. Coordinator task task_0f94cf0766e2. Stack approved by the developer: +#290 -> #293 -> #295 -> Delta half of ITL-919. Previous plan: +`superpowers/plans/2026-11-10-itl-864-delta-read-your-writes.md`. + +**Goal:** the branch contains main (c8efc092) and #290 (head 11efe638) by merge commits, +with both behaviours intact: +- #290: an existing table that cannot be loaded or read raises `StoreReadError`, never + "no rows"; `_read_delta_table(delta_table, filters=None, expression=None)` raises the + private `_DeltaTableReadError` and the caller wraps it with `self._read_failed(record_path)`. +- #293: every read method returns stored rows plus pending rows whose id is not stored, + through any handle of the `at()` family (shared table / ids / dirty caches). + +No public API, hash, stored-schema or dependency change. + +## Tasks + +1. `git merge origin/main` (conflicts expected only where #293 touches the same code as + main; resolve, commit, push WIP). +2. `git merge origin/eywalker/itl-884-delta-read-failures-are-returned-as-no-rows`. + Resolution rules in `delta_lake_databases.py`: + - `_refresh_existing_ids_cache`: #290's shape (try only around the load/scan, raise + `self._read_failed(record_path) from e`), with #293's `set[bytes]` typing and the + cache left dirty on failure. + - `get_all_records`: #290's raise on a failed scan + #293's `_pending_not_stored` / + `_combine_stored_and_pending`. + - `get_records_with_column_value`: #293's one-side-lacks-the-column logic; the stored + read uses #290's `except _DeltaTableReadError -> StoreReadError`; filter errors + propagate as is. + - `get_record_by_id`: #293's "pending and not stored" check + #290's wrapping. + - `get_records_by_ids`: drop #293's `except Exception: return None` (that is exactly + what #290 forbids); `_pending_not_stored` propagates `StoreReadError`; the read uses + #290's wrapping. + - CHANGELOG: one `[0.3.0] — unreleased` section; DESIGN_ISSUES: all entries. +3. Tests: run #290's and #293's tests together. Add interaction tests (public interface, + in `tests/test_databases/test_delta_read_your_writes.py`): + - unreadable stored table with pending rows queued through a sibling handle: every read + method raises `StoreReadError` (pending rows are not returned alone as if the table + were empty), and the pending rows stay queued (nothing dropped); + - a failed stored-id refresh raises `StoreReadError` on every read method, and a later + read after the table is readable again returns stored + pending. + Show each new test's result before any code change it needs. +4. Full suite on 3.11/3.12/3.13, ruff check, ruff format --check, basedpyright (exit code), + golden hashes unchanged. +5. Push; PR description gets the "Stacked on #290" note; CI + CodeRabbit; worker_done. + +## Progress notes (successor-readable) +- Done: merged origin/main c8efc092 (7ba60eb4; conflict DESIGN_ISSUES.md only: kept main's + "Newcomer papercuts" entry and the ITL-864 entry, ours last). Merged #290 11efe638 + (a2ac8d9e; conflicts in `_refresh_existing_ids_cache`, `get_records_with_column_value`, + `get_records_by_ids`, resolved per the rules above; CHANGELOG/DESIGN_ISSUES auto-merged; + ITL-864 Fixed entry added to `[0.3.0] — unreleased` in the merge commit). +- Done: ba396d41 updates #293's two refresh tests to StoreReadError and adds + `test_unreadable_table_with_sibling_pending_rows_raises_and_drops_nothing` (verified to fail + when #293's old `except Exception: return None` is put back in `get_records_by_ids`). +- ITL-923 ephemeral re-run repro (scratchpad script): main and #290 lose records on both + orchestrators; this branch fixes sync (10 stored, 5 read) but async still stores 5 and reads + None. Out of scope; reported to the coordinator. +- Next: full suite 3.11/3.12/3.13, PR description "Stacked on #290", CI, CodeRabbit, worker_done. diff --git a/superpowers/specs/itl-892-exit-hang/FINDINGS.md b/superpowers/specs/itl-892-exit-hang/FINDINGS.md new file mode 100644 index 000000000..744a5f4f8 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/FINDINGS.md @@ -0,0 +1,508 @@ +# ITL-892: process does not exit after reading a Delta table + +November 10, 2026. **Outcome:** the coordinator approved option (c) plus the (d) note. It is +implemented in the same PR (`storage_utils.delta_to_pyarrow_dataset`, two new test modules, +`docs/troubleshooting.md`). The investigation itself changed no `src/` file or `uv.lock`. Scripts, raw results +and native stacks are in this directory; the tables are produced by `summarize.py`. + +## 1. Summary + +- **Reproduced on macOS and Linux, Python 3.11, 3.12 and 3.13, with and without Orcapod.** + A process that scans a Delta table with deltalake's default pyarrow reader can fail at + interpreter exit: it hangs forever, or on Linux with Python 3.11/3.12 it may instead abort + with SIGABRT ("terminate called without an active exception"). The script's work and output + are complete in every failing run. +- **Rate**: one write plus one scan, then exit: about 57% of runs on macOS arm64, 5–38% on + Linux x86_64 (GitHub runners), depending on the Python version and the read call. The ITL-864 script (scan, + four merges with a scan after each) fails 12% on macOS. Through Orcapod, a fresh + `DeltaTableDatabase.get_all_records` fails 53–77% on macOS and 8–23% on Linux, and + `DeltaTableSource` 4–10%. A `PipelineJob` run fails 0–4%. +- **Cause (upstream, not Orcapod)**: `DeltaTable.to_pyarrow_dataset()` (and `to_pyarrow_table()`, + which calls it) reads through `pyarrow.fs.PyFileSystem(DeltaStorageHandler)`, a filesystem + implemented in Python. Arrow's async scan finishes tearing itself down on a thread-pool worker + after `to_table()` returns. When that teardown releases the last reference to a Python-backed + buffer (`PyBuffer`) while the interpreter is finalizing, the worker calls `PyGILState_Ensure` + and CPython stops the thread. It hangs it on 3.13 and exits it on 3.11/3.12, where glibc's + forced unwind then aborts the process. Then `ThreadPool::~ThreadPool`, run from + `__cxa_finalize`, waits forever for a worker that will never finish. This is + [apache/arrow#50482](https://github.com/apache/arrow/issues/50482) and + [apache/arrow#54536](https://github.com/apache/arrow/issues/54536), both open. The fix PR + [apache/arrow#54467](https://github.com/apache/arrow/pull/54467) is open and not merged. No + matching delta-rs issue exists. +- **No dependency version avoids it**: every pyarrow 20.0.0–26.0.1 × deltalake 1.0.2/1.6.6 + combination fails, at 36–68% on macOS. +- **Recommended mitigation, option (c)**: Orcapod scans local Delta tables through a native + pyarrow filesystem (`to_pyarrow_dataset(filesystem=SubTreeFileSystem(table_path, + LocalFileSystem()))`). This takes the failure rate from up to 77% to 0/100 in every Orcapod mode + tested, on macOS and on Linux 3.11/3.12/3.13. The result is schema- and value-identical, including on partitioned + tables with escaped partition values. The full suite passes with it (6834 passed, 96 skipped, + 16 xfailed). No API, hash, stored-schema or dependency change. Cloud stores (`s3://` etc.) + keep deltalake's default reader and stay exposed until Arrow ships a fix. Document that case + (option d). + +## 2. Mechanism and evidence + +Native stacks of hung processes (`samples/`, captured with macOS `sample` and with `gdb` on +Linux): + +- Main thread, every platform: + `exit → __cxa_finalize_ranges → std::shared_ptr::~shared_ptr → + ThreadPool::~ThreadPool → ThreadPool::Shutdown(bool) → condition_variable::wait`. +- Python 3.13, macOS (`samples/dataset_scan_py3.13_pa23.0.1_dl1.0.2.txt`) and Linux + (`linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt`): one pool worker is in + `… MergedGenerator::State::~State → parquet FileReaderImpl::~FileReaderImpl → + SerializedFile::~SerializedFile → ReadRangeCache::~ReadRangeCache → Future result → + arrow::py::PyBuffer::~PyBuffer → PyGILState_Ensure → _PyThreadState_Attach → take_gil → + PyThread_hang_thread`. The `ReadRangeCache` exists because deltalake sets + `ParquetFragmentScanOptions(pre_buffer=True)`. The cached buffers are Python `bytes` returned + by `DeltaStorageHandler`. +- Python 3.11/3.12: the worker is gone. CPython exits a thread that asks for the GIL during + finalization, so it never deregisters from the pool. On macOS the process hangs: the + `samples/orcapod_plain_read_hang_manual.txt` sample has no Arrow worker left and the main + thread still waits in `Shutdown`. On Linux the thread exit is a forced unwind through a + `noexcept` destructor, so `std::terminate` aborts the process (rc −6) in some runs, and others + hang. +- `faulthandler` does not help: the hang is below Python. + +Consistent with this: + +| condition (macOS, py3.12, locked versions, 100 runs) | failed | +|---|---:| +| `write_deltalake` only, or `merge` only, or `to_pyarrow_dataset()` without scanning | 0% | +| pyarrow parquet write + `pyarrow.dataset` scan with deltalake imported | 0% | +| `to_pyarrow_table()` / `to_pyarrow_dataset().to_table()` | 57% / 57% | +| same scan, native filesystem (`filesystem=SubTreeFileSystem(...)`) | 0% | +| same scan, `QueryBuilder` (DataFusion), `pq.read_table(dt.file_uris())`, `pl.read_delta` | 0% each | +| same scan in a `fork` child (exits with `os._exit`, no finalization) | 0% | +| same scan in a `spawn` child (child finalizes normally) | 90% (the parent then waits forever in `join`) | + +## 3. Characterisation + +Full tables: §10. Headline cells, 100 runs each. + +**Operation.** Only scanning (`to_table`, `to_pyarrow_table`) triggers it. Writing and merging +do not. More scans earlier in the process do not raise the rate (ITL-864 script with 4, 12 or +24 merge rounds: 12%, 0%, 0%). What matters is a scan shortly before exit. Orcapod's +`DeltaTableDatabase` scans in `get_all_records`, in `_read_delta_table` (used by +`get_record_by_id`, `get_records_by_ids`, `get_records_with_column_value`), and in +`_refresh_existing_ids_cache` (writes with `skip_duplicates`). `DeltaTableSource.__init__` also +scans. `observability_reader` uses `pl.read_delta`, which is safe (0%). + +**Thread pools and environment** (macOS, one-scan cell, baseline 57%): `pa.set_cpu_count(1)` +33%; `pa.set_io_thread_count(1)` 53%; `OMP_NUM_THREADS=1` 45%; +`RAYON_NUM_THREADS=1 TOKIO_WORKER_THREADS=1` 49%. `to_table(use_threads=False)` gives 0% on +macOS but 8–20% on Linux, so it is not a fix. No thread setting removes it. + +**Two Arrow runtimes.** deltalake's Rust arrow alone (writes, merges, `QueryBuilder`) never +triggers it, and pyarrow alone (`pyarrow_only`) never does. Only pyarrow's C++ pool combined +with the Python-implemented filesystem does. + +**Polars as a third runtime.** Importing polars after pyarrow gives 48%, importing it first 27%. +Polars changes timing, not the outcome. + +**Python version** (macOS): one-scan cell 3.11 57%, 3.12 58%, 3.13 65%; Orcapod plain-read 3.11 77%, 3.12 53%, 3.13 76%. The 3.11/3.13 Orcapod cells ran in an isolated environment with a 30 s deadline and 4 parallel runs; the 3.12 cell ran in the project environment with 6 s and 10. Orcapod `PipelineJob` sync and async: 0% on all three. Linux: see §10. On 3.13 failures are +always hangs. On 3.11/3.12 Linux they are hangs or SIGABRT. + +**Load.** Running 1, 4 or 10 children in parallel gives 3%, 5%, 10% on the ITL-864 script. Load +raises the rate a little; it is not the cause. + +**Platforms.** macOS arm64: measured here. Linux x86_64: measured on GitHub `ubuntu-latest` +through a temporary workflow on this branch (`.github/workflows/itl-892-exit-hang-probe.yml`, +not a required check, since deleted; GitHub Actions runs 42969989607 and 42972298798). Docker is not installed on +this machine. + +## 4. Versions and upstream + +`uv run --no-project --with pyarrow==X --with deltalake==Y` (uv.lock untouched), macOS, +py3.12, one-scan cell: + +| pyarrow \ deltalake | 1.0.2 (locked) | 1.6.6 (latest) | +|---|---:|---:| +| 20.0.0 (project floor) | 67% | 59% | +| 22.0.0 | 55% | 55% | +| 23.0.1 (locked) | 50–58% | 61% | +| 24.0.0 | 53% | 58% | +| 25.0.1 | 36% | 39% | +| 26.0.1 (latest) | 56% | 68% | + +Linux, py3.11/3.12/3.13: pyarrow 22.0.0 + deltalake 1.0.2 fails 6–14%; 26.0.1 + 1.6.6 fails +2–5%. Within what the project allows (`pyarrow>=20`, `deltalake>=1.0.2`) **there is no +combination that does not fail**, so there is no "newest good" version. The oldest failing +combination is the project floor. Upstream reports it on pyarrow 22–26. deltalake 1.6.6 still +defaults to `PyFileSystem(DeltaStorageHandler)` with `pre_buffer=True`. + +Upstream: + +- [apache/arrow#50482](https://github.com/apache/arrow/issues/50482) (open, July 2026): + interpreter hangs at exit after `pq.read_table(python_file_object)`, a thread-pool teardown + deadlock. Reported on pyarrow 22–25, Linux and macOS. +- [apache/arrow#54536](https://github.com/apache/arrow/issues/54536) (open, October 2026): abort + at exit, `~PyReadableFile` takes the GIL on a pool worker during finalization; Linux, + pyarrow 25–26, SIGABRT with the same message we see. +- [apache/arrow#54467](https://github.com/apache/arrow/pull/54467) (open PR, not merged): an + `atexit` callback that releases the GIL and waits for the CPU pool to go idle. +- delta-rs: no matching issue (searched "hang", "hangs", "deadlock", "exit", + "interpreter shutdown", "PyFileSystem"). Proposed: a delta-rs issue pointing at the Arrow + issues, suggesting a native object-store filesystem for local paths. Not filed: this run + does not post to GitHub. + +## 5. Options + +Failure rates are over 100 runs. "Orcapod" cells use `repro_orcapod.py`, which prototypes each +option by wrapping deltalake (`--patch`), so `src/` is unchanged. + +| option | macOS | Linux 3.11 / 3.12 / 3.13 | cost | +|---|---|---|---| +| none (locked) | Orcapod plain-read 53%, source-read 8%, pipeline 0% | two rounds: plain-read 9–12 / 8–19 / 15–23%, source-read 4–8 / 5–10 / 8–10%, pipeline-sync 0 / 0–4 / 0–1% | — | +| (a) pin or bump pyarrow/deltalake | 36–68% for every combination | 2–14% for every tried combination | none possible today. A future pyarrow with #54467 would fix it: bump `uv.lock` then and re-run golden hashes and the suite | +| (b1) `atexit`: `gc.collect()` | Orcapod plain-read 27% | plain-read 21 / 12 / 17%, source-read 5 / 6 / 6%, pipeline-sync 1 / 1 / 1% | trivial, but does not fix it | +| (b2) drop references + `gc.collect()` before exit | 2% (pure) | 0 / 0 / 0% (pure) | needs user code; not a fix on macOS | +| (b3) `atexit`: release the GIL for 50 ms (`time.sleep(0.05)`) | pure 0%; Orcapod 0% in all four modes | pure 0 / 0 / 0% (50 ms and 500 ms); Orcapod 0% in all three modes on each version | +50 ms on every exit of a process that imported Orcapod; timing-based (a busier machine may need longer); a global side effect of importing a library | +| (b4) `os._exit` after flushing | 0% | — | skips `atexit`, buffered I/O and other libraries' cleanup; only acceptable inside our own experiment harnesses, not in the library | +| **(c) native filesystem for local tables** | **0% in all four Orcapod modes; pure 0% on py3.11/3.12/3.13** | **pure 0 / 0 / 0%; Orcapod 0% in all three modes on each version** | ~20 lines in `delta_lake_databases.py` (one helper used by the 4 scan sites) and `delta_table_source.py`. Byte-identical results (`check_native_fs_equivalence.py`, partition edge cases). Full suite passes under the patch. Cloud URIs keep the default path | +| (c') `to_table(use_threads=False)` | 0% | 8–20% (pure) | slower scans; not a fix on Linux | +| (c'') read via `QueryBuilder` / polars / `pq.read_table(file_uris())` | 0% (pure) | — | larger change, and a different reader with its own type mapping (large types, nullability) to verify against the type-strictness rules; no benefit over (c) | +| (d) documentation only | unchanged | unchanged | a troubleshooting entry: "if a script that used a Delta store does not exit, upgrade pyarrow when #54467 ships; meanwhile end the script with `os._exit(0)` after flushing". There is no troubleshooting page on `main` yet | + +Effect on the locked dependencies, golden hashes and the suite: + +- (a) Nothing to lock. A future bump would change `uv.lock` only. +- (b) and (c) change no dependency and no stored value. Option (c) changes the read path only. + Hashes are computed from the tables, which are identical. The suite run with the (c) patch + (`native_fs_pytest_plugin.py`) gave 6834 passed, 96 skipped, 16 xfailed, no failures. The + golden hash tests are part of that run. + +## 6. Recommendation + +Ship **(c)** for v0.3, plus a short **(d)** note for cloud stores: + +1. In `DeltaTableDatabase`, route the four scan sites (`get_all_records`, `_read_delta_table`, + `_refresh_existing_ids_cache`, and through them every read method) through one helper that + passes `filesystem=SubTreeFileSystem(, LocalFileSystem())` when the store is + local (`not self._is_cloud`). Do the same in `DeltaTableSource`, which already requires a + local path. +2. Keep deltalake's default reader for cloud URIs. A native `S3FileSystem` has its own exit + deadlock ([apache/arrow#50188](https://github.com/apache/arrow/issues/50188)), and the + storage-option mapping is not one-to-one. Document the remaining exposure. +3. Revisit when pyarrow ships #54467. Bump the floor then, and decide whether the helper can + go. + +Why (c) over (b3). (b3) also measured 0% everywhere, and it is the only option that also +covers cloud stores. But it does not remove the cause. It only gives the pool time to finish +before finalization starts. The 50 ms is a guess that held on these machines; a loaded machine +or a large scan could outlast it, and the failure would come back silently. It would also run +in every process that imported Orcapod. + +(c) removes the Python objects from the scan, so there is nothing for a worker to release +late. Its results are byte-identical by check. A possible follow-up, not proposed for v0.3 +unless the developer wants cloud covered now: register the (b3) hook lazily, only the first +time a cloud-store scan runs, so local users never pay for it. + +## 7. Regression check + +`regression_check_proto.py` (prototype, not in `tests/` yet): for each of two modes +(`DeltaTableDatabase` write, flush and fresh-handle read; `DeltaTableSource` read), it starts 40 +fresh interpreters, 4 at a time. It fails if any run does not exit within 60 s or exits +non-zero, which covers both the hang and the Linux abort. `ITL892_PATCH` lets the same check +validate a prototype. + +| | macOS | Linux 3.11 / 3.12 / 3.13 | +|---|---|---| +| unpatched (today's `main`) | fails both cases (wall time 497 s, bounded by the 60 s deadlines) | fails both cases on each version (162 / 198 / 511 s) | +| native-fs patch | passes, 10 s | passes on each version, 39 / 50 / 50 s | + +Detection. Measured per-run failure rates without a fix: macOS 53–77% (plain-read) and 8% +(source-read); Linux 8–23% and 4–10%. With 40 runs per case, both cases miss a regression with +probability below 1e-12 on macOS and below 1% per Linux Python job, which is about 1e-6 across +CI's three Python jobs. + +False failures. A clean run takes 1–2 s, so the 60 s deadline leaves a wide margin. Every +patched run passed (one macOS and three Linux runs of the check, plus thousands of patched +harness runs). + +Recommendation: add it to `tests/` as an ordinary test that runs in CI, for example +`tests/test_databases/test_delta_exit.py`, with the child script inline and only the public API. +When it passes it costs about 40–50 s per Python job on GitHub runners (10 s on this Mac) and +needs no infrastructure. If that time is unwelcome, halve `RUNS` to 20: a miss is then about 8% +likely per job and 0.05% across the three jobs. It is reliable enough for CI. An opt-in marker +would be justified only by its run time, not by flakiness. + +## 8. Not verified + +- Linux arm64 and Windows. +- Cloud stores (S3/MinIO): not run. The analysis says they stay exposed with (c), since they + keep the Python filesystem. +- Jupyter kernels: not affected in the same way, since the kernel does not exit. +- The mechanism on 3.11/3.12 macOS is inferred: the worker had already vanished in every + sample, so its last frame was not captured. The 3.13 stacks show it directly. + +## 8a. Out-of-scope findings + +- `delta_lake_databases.py:868`: `return dataset.to_table()(filters=filters)` calls the + returned `Table`, which raises `TypeError` whenever `_read_delta_table` gets a filter + operator other than `=` or `in`. + +## 9. Files + +| file | purpose | +|---|---| +| `harness.py` | runs a command N times with a deadline; counts hangs and exit codes; samples the first hung child's native stack | +| `repro_pure.py` | pyarrow + deltalake only; `--op`, `--read-via`, thread, polars, mp and mitigation switches | +| `repro_orcapod.py` | the same through Orcapod's public API; `--patch` prototypes options (b) and (c) | +| `run_matrix.sh` | the macOS matrix groups | +| `check_native_fs_equivalence.py` | option (c) returns identical schemas and values | +| `native_fs_pytest_plugin.py` | runs the suite with option (c) applied | +| `regression_check_proto.py` | proposed regression check | +| `summarize.py` | renders the tables below | +| `results.jsonl` | macOS raw results (one line per cell) | +| `samples/` | representative macOS stacks: py3.12 (`op=dataset_scan_py3.12_locked.txt`), py3.13 with the worker in `PyGILState_Ensure` (`dataset_scan_py3.13_pa23.0.1_dl1.0.2.txt`), Orcapod with no worker left (`orcapod_plain_read_hang_manual.txt`). The other stacks named in `results.jsonl` are not committed (`*.txt` is gitignored; about 4 MB) | +| `linux/round*/` | Linux raw results from the temporary CI probe, with two py3.13 `gdb` stacks. On 3.11/3.12, `gdb` could not attach because the child aborted first | + +## 10. All results + +Each cell is one `harness.py` run (`results.jsonl`, `linux/*/results.jsonl`). "hung" means the run missed its deadline. In almost every cell it had already printed `done`; the exception is the `spawn` cell, where the parent waits on a hung child. Early exploratory cells (`baseline`, `load`, `amplify`, `itl864 ...`) used the ITL-864 script, which fails less often than the one-scan cell (`dataset_scan`) used afterwards. + +#### macOS arm64 (this machine) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| baseline itl864 py3.12 locked | 40 | 2 | 0 | 0 | **5%** | +| load j=1 itl864 py3.12 locked | 60 | 2 | 0 | 0 | **3%** | +| load j=4 itl864 py3.12 locked | 60 | 3 | 0 | 0 | **5%** | +| amplify rounds=12 itl864 py3.12 locked | 40 | 0 | 0 | 0 | **0%** | +| amplify rounds=24 itl864 py3.12 locked | 40 | 0 | 0 | 0 | **0%** | +| load j=10 itl864 py3.12 locked | 60 | 6 | 0 | 0 | **10%** | +| op=pyarrow_only py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=write py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=merge py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=to_dataset py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| op=to_table py3.12 locked | 100 | 57 | 0 | 0 | **57%** | +| op=dataset_scan py3.12 locked | 100 | 57 | 0 | 0 | **57%** | +| op=itl864 py3.12 locked | 100 | 12 | 0 | 0 | **12%** | +| itl864 read-via=native_fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=no_threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=querybuilder py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=files py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 read-via=polars py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| itl864 cpu_threads=1 | 100 | 6 | 0 | 0 | **6%** | +| dataset_scan read-via=native_fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=no_threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=querybuilder py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=files py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan read-via=polars py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan cpu_threads=1 | 100 | 33 | 0 | 0 | **33%** | +| dataset_scan io_threads=1 | 100 | 53 | 0 | 0 | **53%** | +| dataset_scan OMP_NUM_THREADS=1 | 100 | 45 | 0 | 0 | **45%** | +| dataset_scan RAYON/TOKIO threads=1 | 100 | 49 | 0 | 0 | **49%** | +| dataset_scan import polars after pyarrow | 100 | 48 | 0 | 0 | **48%** | +| dataset_scan import polars first | 100 | 27 | 0 | 0 | **27%** | +| dataset_scan in fork child | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan in spawn child | 100 | 90 | 0 | 0 | **90%** | +| dataset_scan mitigation=del | 100 | 2 | 0 | 0 | **2%** | +| dataset_scan mitigation=atexit_del | 100 | 1 | 0 | 0 | **1%** | +| dataset_scan mitigation=os_exit | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan mitigation=atexit_sleep | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan mitigation=atexit_sleep_long | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read py3.12 locked | 100 | 53 | 0 | 0 | **53%** | +| orcapod source-read py3.12 locked | 100 | 8 | 0 | 0 | **8%** | +| orcapod pipeline-sync py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod rerun py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.11 pa23.0.1 dl1.0.2 | 100 | 57 | 0 | 0 | **57%** | +| dataset_scan native_fs py3.11 pa23.0.1 dl1.0.2 | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.12 pa23.0.1 dl1.0.2 | 100 | 50 | 0 | 0 | **50%** | +| dataset_scan native_fs py3.12 pa23.0.1 dl1.0.2 | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.13 pa23.0.1 dl1.0.2 | 100 | 65 | 0 | 0 | **65%** | +| dataset_scan native_fs py3.13 pa23.0.1 dl1.0.2 | 100 | 0 | 0 | 0 | **0%** | +| dataset_scan py3.12 pa20.0.0 dl1.6.6 | 100 | 59 | 0 | 0 | **59%** | +| dataset_scan py3.12 pa22.0.0 dl1.0.2 | 100 | 55 | 0 | 0 | **55%** | +| dataset_scan py3.12 pa22.0.0 dl1.6.6 | 100 | 55 | 0 | 0 | **55%** | +| dataset_scan py3.12 pa23.0.1 dl1.6.6 | 100 | 61 | 0 | 0 | **61%** | +| dataset_scan py3.12 pa24.0.0 dl1.0.2 | 100 | 53 | 0 | 0 | **53%** | +| dataset_scan py3.12 pa24.0.0 dl1.6.6 | 100 | 58 | 0 | 0 | **58%** | +| dataset_scan py3.12 pa25.0.1 dl1.0.2 | 100 | 36 | 0 | 0 | **36%** | +| dataset_scan py3.12 pa25.0.1 dl1.6.6 | 100 | 39 | 0 | 0 | **39%** | +| dataset_scan py3.12 pa26.0.1 dl1.0.2 | 100 | 56 | 0 | 0 | **56%** | +| dataset_scan py3.12 pa26.0.1 dl1.6.6 | 100 | 68 | 0 | 0 | **68%** | +| dataset_scan py3.12 pa20.0.0 dl1.0.2 | 100 | 67 | 0 | 0 | **67%** | +| orcapod plain-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod source-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-sync patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod source-read patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-sync patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=no-threads py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read patch=atexit-gc py3.12 locked | 100 | 27 | 0 | 0 | **27%** | +| orcapod source-read patch=atexit-gc py3.12 locked | 100 | 5 | 0 | 0 | **5%** | +| orcapod pipeline-sync patch=atexit-gc py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=atexit-gc py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod source-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-sync patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read py3.11 locked | 100 | 77 | 0 | 0 | **77%** | +| orcapod pipeline-sync py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod plain-read py3.13 locked | 100 | 76 | 0 | 0 | **76%** | +| orcapod pipeline-sync py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| orcapod pipeline-async py3.13 locked | 100 | 0 | 0 | 0 | **0%** | + +#### Linux x86_64, GitHub ubuntu-latest (round1-py3.11) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux op=dataset_scan py3.11 locked | 100 | 8 | 13 | 0 | **21%** | +| linux op=to_table py3.11 locked | 100 | 4 | 19 | 0 | **23%** | +| linux op=itl864 py3.11 locked | 100 | 8 | 0 | 0 | **8%** | +| linux dataset_scan native_fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan no_threads py3.11 locked | 100 | 11 | 0 | 0 | **11%** | +| linux dataset_scan io_threads=1 py3.11 locked | 100 | 18 | 0 | 0 | **18%** | +| linux dataset_scan mitigation=del py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read py3.11 locked | 100 | 12 | 0 | 0 | **12%** | +| linux orcapod source-read py3.11 locked | 100 | 4 | 0 | 0 | **4%** | +| linux orcapod pipeline-sync py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-async py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux op=dataset_scan py3.11 pa22.0.0 dl1.0.2 | 100 | 6 | 0 | 0 | **6%** | +| linux op=dataset_scan py3.11 pa26.0.1 dl1.6.6 | 100 | 2 | 2 | 0 | **4%** | + +#### Linux x86_64, GitHub ubuntu-latest (round1-py3.12) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux op=dataset_scan py3.12 locked | 100 | 10 | 8 | 0 | **18%** | +| linux op=to_table py3.12 locked | 100 | 3 | 17 | 0 | **20%** | +| linux op=itl864 py3.12 locked | 100 | 8 | 0 | 0 | **8%** | +| linux dataset_scan native_fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan no_threads py3.12 locked | 100 | 20 | 1 | 0 | **21%** | +| linux dataset_scan io_threads=1 py3.12 locked | 100 | 18 | 0 | 0 | **18%** | +| linux dataset_scan mitigation=del py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read py3.12 locked | 100 | 8 | 0 | 0 | **8%** | +| linux orcapod source-read py3.12 locked | 100 | 5 | 0 | 0 | **5%** | +| linux orcapod pipeline-sync py3.12 locked | 100 | 4 | 0 | 0 | **4%** | +| linux orcapod pipeline-async py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux op=dataset_scan py3.12 pa22.0.0 dl1.0.2 | 100 | 13 | 0 | 0 | **13%** | +| linux op=dataset_scan py3.12 pa26.0.1 dl1.6.6 | 100 | 2 | 1 | 0 | **3%** | + +#### Linux x86_64, GitHub ubuntu-latest (round1-py3.13) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux op=dataset_scan py3.13 locked | 100 | 5 | 0 | 0 | **5%** | +| linux op=to_table py3.13 locked | 100 | 38 | 0 | 0 | **38%** | +| linux op=itl864 py3.13 locked | 100 | 10 | 0 | 0 | **10%** | +| linux dataset_scan native_fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan no_threads py3.13 locked | 100 | 8 | 0 | 0 | **8%** | +| linux dataset_scan io_threads=1 py3.13 locked | 100 | 32 | 0 | 0 | **32%** | +| linux dataset_scan mitigation=del py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read py3.13 locked | 100 | 15 | 0 | 0 | **15%** | +| linux orcapod source-read py3.13 locked | 100 | 10 | 0 | 0 | **10%** | +| linux orcapod pipeline-sync py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-async py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux op=dataset_scan py3.13 pa22.0.0 dl1.0.2 | 100 | 14 | 0 | 0 | **14%** | +| linux op=dataset_scan py3.13 pa26.0.1 dl1.6.6 | 100 | 5 | 0 | 0 | **5%** | + +#### Linux x86_64, GitHub ubuntu-latest (round2-py3.11) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux dataset_scan mitigation=atexit_sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan mitigation=atexit_sleep_long py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=none py3.11 locked | 100 | 9 | 0 | 0 | **9%** | +| linux orcapod source-read patch=none py3.11 locked | 100 | 8 | 0 | 0 | **8%** | +| linux orcapod pipeline-sync patch=none py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=native-fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=native-fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=native-fs py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=atexit-sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=atexit-sleep py3.11 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-gc py3.11 locked | 100 | 8 | 13 | 0 | **21%** | +| linux orcapod source-read patch=atexit-gc py3.11 locked | 100 | 4 | 1 | 0 | **5%** | +| linux orcapod pipeline-sync patch=atexit-gc py3.11 locked | 100 | 1 | 0 | 0 | **1%** | + +#### Linux x86_64, GitHub ubuntu-latest (round2-py3.12) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux dataset_scan mitigation=atexit_sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan mitigation=atexit_sleep_long py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=none py3.12 locked | 100 | 8 | 11 | 0 | **19%** | +| linux orcapod source-read patch=none py3.12 locked | 100 | 8 | 2 | 0 | **10%** | +| linux orcapod pipeline-sync patch=none py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=native-fs py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=atexit-sleep py3.12 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-gc py3.12 locked | 100 | 11 | 1 | 0 | **12%** | +| linux orcapod source-read patch=atexit-gc py3.12 locked | 100 | 6 | 0 | 0 | **6%** | +| linux orcapod pipeline-sync patch=atexit-gc py3.12 locked | 100 | 1 | 0 | 0 | **1%** | + +#### Linux x86_64, GitHub ubuntu-latest (round2-py3.13) + +| cell | runs | hung | aborted (rc<0) | other rc!=0 | failed | +|---|---:|---:|---:|---:|---:| +| linux dataset_scan mitigation=atexit_sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux dataset_scan mitigation=atexit_sleep_long py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=none py3.13 locked | 100 | 23 | 0 | 0 | **23%** | +| linux orcapod source-read patch=none py3.13 locked | 100 | 8 | 0 | 0 | **8%** | +| linux orcapod pipeline-sync patch=none py3.13 locked | 100 | 1 | 0 | 0 | **1%** | +| linux orcapod plain-read patch=native-fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=native-fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=native-fs py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod source-read patch=atexit-sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod pipeline-sync patch=atexit-sleep py3.13 locked | 100 | 0 | 0 | 0 | **0%** | +| linux orcapod plain-read patch=atexit-gc py3.13 locked | 100 | 17 | 0 | 0 | **17%** | +| linux orcapod source-read patch=atexit-gc py3.13 locked | 100 | 6 | 0 | 0 | **6%** | +| linux orcapod pipeline-sync patch=atexit-gc py3.13 locked | 100 | 1 | 0 | 0 | **1%** | + + +## 11. Proposed delta-rs issue (not filed; for the developer) + +**Title:** `to_pyarrow_dataset()`'s default `PyFileSystem(DeltaStorageHandler)` makes processes hang or abort at interpreter exit + +**Body:** + +> A process that scans a Delta table with `DeltaTable.to_pyarrow_dataset().to_table()` or +> `to_pyarrow_table()` and then exits normally often fails during interpreter shutdown. It +> hangs forever (macOS, any Python; Linux, Python 3.13), or on Linux with Python 3.11/3.12 it +> aborts with `terminate called without an active exception`. All of its work and output are +> done by then. +> +> ```python +> import tempfile, deltalake, pyarrow as pa +> p = tempfile.mkdtemp() +> deltalake.write_deltalake(p, pa.table({"k": [1, 2, 3]})) +> deltalake.DeltaTable(p).to_pyarrow_dataset().to_table() +> print("done") # prints, then the process may never exit +> ``` +> +> Rate over 100 fresh processes: about 57% on macOS arm64 and 5–38% on Linux x86_64. It +> happens on every combination we tried: pyarrow 20.0.0–26.0.1 × deltalake 1.0.2 and 1.6.6, +> Python 3.11–3.13. Writes and merges alone never trigger it. +> +> Native stacks (Python 3.13): the main thread is in `exit → __cxa_finalize → +> arrow::internal::ThreadPool::~ThreadPool → Shutdown`. A pool worker is in +> `parquet::arrow::FileReaderImpl::~FileReaderImpl → ReadRangeCache::~ReadRangeCache → +> arrow::py::PyBuffer::~PyBuffer → PyGILState_Ensure → PyThread_hang_thread`. The cached +> buffers come from `DeltaStorageHandler` (Python `bytes`; `pre_buffer=True`), and Arrow +> releases them on a worker after `to_table()` returns. If that happens during finalization, +> the worker cannot take the GIL. The root bug is in Arrow: apache/arrow#50482, #54536, fix PR +> #54467. +> +> Passing a native filesystem avoids it completely (0/100 on macOS and Linux, identical +> results, partitioned tables included). `table_uri` is percent-encoded (a bare path on 1.0.2 +> and 1.2.1, a `file://` URI on 1.6.6), so it has to be decoded first: +> +> ```python +> uri = dt.table_uri if "://" in dt.table_uri else "file://" + dt.table_uri +> fs, root = pyarrow.fs.FileSystem.from_uri(uri) +> dt.to_pyarrow_dataset(filesystem=pyarrow.fs.SubTreeFileSystem(root, fs)) +> ``` +> +> Suggestion: default to a native pyarrow filesystem for local tables (and possibly the native +> object-store filesystems), or document the issue and the workaround in `to_pyarrow_dataset` +> until Arrow's fix is released. diff --git a/superpowers/specs/itl-892-exit-hang/check_native_fs_equivalence.py b/superpowers/specs/itl-892-exit-hang/check_native_fs_equivalence.py new file mode 100644 index 000000000..068a3befd --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/check_native_fs_equivalence.py @@ -0,0 +1,121 @@ +"""Option (c) check: a native filesystem returns exactly what deltalake's default returns. + +Writes a Delta table with a spread of Arrow types (narrow ints, decimals, timestamps, +nested lists and structs, binary, nulls, non-nullable fields), appends and merges into +it, then reads it with ``to_pyarrow_dataset(as_large_types=True).to_table()`` twice: +once through the default ``PyFileSystem(DeltaStorageHandler)`` and once through +``SubTreeFileSystem(table_path, LocalFileSystem())``. Schemas (with field metadata and +nullability) and values must be identical, with and without a filter expression. +""" + +from __future__ import annotations + +import datetime as dt +import decimal +import tempfile + +import deltalake +import pyarrow as pa +import pyarrow.compute as pc +import pyarrow.fs as pa_fs + + +def make(i: int) -> pa.Table: + """Build 50 rows covering narrow, decimal, timestamp, binary, nested and null columns. + + Args: + i: Batch number, which keeps the ids of each batch distinct. + + Returns: + The table. + """ + schema = pa.schema( + [ + pa.field("id", pa.large_string(), nullable=False), + pa.field("i8", pa.int8()), + pa.field("f32", pa.float32()), + pa.field("dec", pa.decimal128(12, 3)), + pa.field("ts", pa.timestamp("us", tz="UTC")), + pa.field("bin", pa.large_binary()), + pa.field("lst", pa.large_list(pa.int16())), + pa.field( + "st", + pa.struct( + [pa.field("a", pa.int32()), pa.field("b", pa.large_string())] + ), + ), + pa.field("n", pa.large_string()), + ] + ) + rows = 50 + return pa.table( + { + "id": [f"r{i}-{k}" for k in range(rows)], + "i8": [k % 100 for k in range(rows)], + "f32": [k / 3 for k in range(rows)], + "dec": [decimal.Decimal(k * 1001) / 1000 for k in range(rows)], + "ts": [dt.datetime(2026, 1, 1, tzinfo=dt.timezone.utc)] * rows, + "bin": [bytes([k % 256]) * 3 for k in range(rows)], + "lst": [[k, k + 1] if k % 5 else None for k in range(rows)], + "st": [{"a": k, "b": str(k)} for k in range(rows)], + "n": [None] * rows, + }, + schema=schema, + ) + + +def read( + table: deltalake.DeltaTable, native: bool, flt: pc.Expression | None +) -> pa.Table: + """Read ``table`` through the default or the native filesystem, sorted by ``id``. + + Args: + table: The Delta table. + native: Read through a native ``SubTreeFileSystem`` instead of deltalake's. + flt: Optional filter applied by the scan. + + Returns: + The scanned rows. + """ + kwargs = {} + if native: + kwargs["filesystem"] = pa_fs.SubTreeFileSystem( + table.table_uri, pa_fs.LocalFileSystem() + ) + ds = table.to_pyarrow_dataset(as_large_types=True, **kwargs) + return ds.to_table(filter=flt).sort_by("id") + + +def main() -> None: + """Compare default and native reads of one table across versions and filters.""" + path = tempfile.mkdtemp() + deltalake.write_deltalake(path, make(0)) + deltalake.write_deltalake(path, make(1), mode="append") + deltalake.DeltaTable(path).merge( + source=make(2), + predicate="target.id = source.id", + source_alias="source", + target_alias="target", + ).when_not_matched_insert_all().execute() + table = deltalake.DeltaTable(path) + checks = 0 + rows = 0 + for flt in ( + None, + pc.field("i8") > 10, + pc.is_in(pc.field("id"), pa.array(["r1-3", "r2-4"])), + ): + a, b = read(table, False, flt), read(table, True, flt) + assert a.schema.equals(b.schema, check_metadata=True), (a.schema, b.schema) + assert a.equals(b), "values differ" + checks += 1 + rows = a.num_rows + print( + f"files={len(table.file_uris())} last_filter_rows={rows} " + f"checks={checks} identical schema+values" + ) + print("done", flush=True) + + +if __name__ == "__main__": + main() diff --git a/superpowers/specs/itl-892-exit-hang/harness.py b/superpowers/specs/itl-892-exit-hang/harness.py new file mode 100644 index 000000000..a1c215523 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/harness.py @@ -0,0 +1,150 @@ +"""Run a command many times and count the runs that do not exit before a deadline. + +Usage: + harness.py --label LABEL -n 40 -t 20 -j 4 -- + +Each run is a child process in its own session. A run that misses the deadline is +counted as hung; the first hung child of a cell has its native stack sampled with +macOS ``sample`` (or ``eu-stack``/``gdb`` on Linux when present) before it is killed. +A summary line is printed and appended as JSON to ``results.jsonl`` next to this file. +""" + +from __future__ import annotations + +import argparse +import json +import os +import platform +import shutil +import signal +import subprocess +import sys +import time +from collections import Counter +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from typing import Any + +HERE = Path(__file__).resolve().parent +SAMPLES = HERE / "samples" + + +def _sample_stack(pid: int, label: str) -> str | None: + """Capture the native stack of a hung process, returning the file path.""" + SAMPLES.mkdir(exist_ok=True) + out = SAMPLES / f"{label.replace('/', '_').replace(' ', '_')}.txt" + if sys.platform == "darwin" and shutil.which("sample"): + cmd = ["sample", str(pid), "1", "-file", str(out)] + elif shutil.which("gdb"): + cmd = ["gdb", "-p", str(pid), "-batch", "-ex", "thread apply all bt"] + elif shutil.which("eu-stack"): + cmd = ["eu-stack", "-p", str(pid)] + else: + return None + try: + res = subprocess.run(cmd, capture_output=True, text=True, timeout=60) + except Exception: # noqa: BLE001 - diagnostics only + return None + if not out.exists(): + out.write_text(res.stdout + res.stderr) + return str(out.relative_to(HERE)) + + +def _run_one(cmd: list[str], timeout: float, env: dict[str, str]) -> dict[str, Any]: + """Run ``cmd`` once and classify the outcome. + + Args: + cmd: The command to run. + timeout: Seconds before the run counts as hung. + env: Environment for the child. + + Returns: + The outcome. A hung run also carries the live process, for the caller to sample + and kill. + """ + t0 = time.monotonic() + proc = subprocess.Popen( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=True, + env=env, + ) + try: + out, err = proc.communicate(timeout=timeout) + return { + "hung": False, + "rc": proc.returncode, + "done": "done" in out, + "secs": round(time.monotonic() - t0, 2), + "err": err[-400:] if proc.returncode else "", + } + except subprocess.TimeoutExpired: + return {"hung": True, "proc": proc, "secs": timeout} + + +def main() -> int: + """Run one cell of the matrix and append its summary to ``results.jsonl``. + + Returns: + The process exit code. + """ + ap = argparse.ArgumentParser() + ap.add_argument("--label", required=True) + ap.add_argument("-n", type=int, default=40) + ap.add_argument("-t", type=float, default=20.0) + ap.add_argument("-j", type=int, default=4) + ap.add_argument("--no-sample", action="store_true") + ap.add_argument("cmd", nargs=argparse.REMAINDER) + args = ap.parse_args() + cmd = args.cmd[1:] if args.cmd and args.cmd[0] == "--" else args.cmd + env = dict(os.environ) + + results: list[dict[str, Any]] = [] + sampled: str | None = None + with ThreadPoolExecutor(max_workers=args.j) as ex: + for r in ex.map(lambda _: _run_one(cmd, args.t, env), range(args.n)): + if r["hung"]: + proc = r.pop("proc") + if sampled is None and not args.no_sample: + sampled = _sample_stack(proc.pid, args.label) or "unavailable" + try: + os.killpg(proc.pid, signal.SIGKILL) + except (PermissionError, ProcessLookupError): + # macOS refuses killpg once the leader is a zombie; it already exited. + proc.kill() + out, _ = proc.communicate() + r["done"] = "done" in (out or "") + results.append(r) + + hung = sum(r["hung"] for r in results) + rcs = Counter("HUNG" if r["hung"] else r["rc"] for r in results) + hung_after_done = sum(r["hung"] and r["done"] for r in results) + errors = sorted({r["err"] for r in results if r.get("err")})[:2] + summary = { + "label": args.label, + "n": args.n, + "hung": hung, + "hung_after_done": hung_after_done, + "rc": {str(k): v for k, v in rcs.items()}, + "timeout_s": args.t, + "jobs": args.j, + "platform": f"{platform.system()} {platform.machine()}", + "stack": sampled, + "errors": errors, + "cmd": " ".join(cmd), + } + print( + f"{args.label}: hung {hung}/{args.n} (after 'done': {hung_after_done}) rc={dict(rcs)}" + + (f" stack={sampled}" if sampled else "") + + (f" ERR={errors[0][-200:]!r}" if errors else ""), + flush=True, + ) + with open(HERE / "results.jsonl", "a") as fh: + fh.write(json.dumps(summary) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.11/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.11/results.jsonl new file mode 100644 index 000000000..bc48de289 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.11/results.jsonl @@ -0,0 +1,13 @@ +{"label": "linux op=dataset_scan py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 79, "HUNG": 8, "-6": 13}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.11_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=to_table py3.11 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"0": 77, "HUNG": 4, "-6": 19}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=to_table_py3.11_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "linux op=itl864 py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=itl864_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "linux dataset_scan native_fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "linux dataset_scan no_threads py3.11 locked", "n": 100, "hung": 11, "hung_after_done": 11, "rc": {"0": 89, "HUNG": 11}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_no_threads_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "linux dataset_scan io_threads=1 py3.11 locked", "n": 100, "hung": 18, "hung_after_done": 18, "rc": {"0": 82, "HUNG": 18}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_io_threads=1_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "linux dataset_scan mitigation=del py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "linux orcapod plain-read py3.11 locked", "n": 100, "hung": 12, "hung_after_done": 12, "rc": {"0": 88, "HUNG": 12}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "linux orcapod source-read py3.11 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"0": 96, "HUNG": 4}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "linux orcapod pipeline-sync py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "linux orcapod pipeline-async py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "linux op=dataset_scan py3.11 pa22.0.0 dl1.0.2", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.11_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=dataset_scan py3.11 pa26.0.1 dl1.6.6", "n": 100, "hung": 2, "hung_after_done": 2, "rc": {"0": 96, "HUNG": 2, "-6": 2}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.11_pa26.0.1_dl1.6.6.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.12/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.12/results.jsonl new file mode 100644 index 000000000..3de475532 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.12/results.jsonl @@ -0,0 +1,13 @@ +{"label": "linux op=dataset_scan py3.12 locked", "n": 100, "hung": 10, "hung_after_done": 10, "rc": {"0": 82, "HUNG": 10, "-6": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=to_table py3.12 locked", "n": 100, "hung": 3, "hung_after_done": 3, "rc": {"0": 80, "-6": 17, "HUNG": 3}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=to_table_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "linux op=itl864 py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "linux dataset_scan native_fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "linux dataset_scan no_threads py3.12 locked", "n": 100, "hung": 20, "hung_after_done": 20, "rc": {"0": 79, "HUNG": 20, "-6": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_no_threads_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "linux dataset_scan io_threads=1 py3.12 locked", "n": 100, "hung": 18, "hung_after_done": 18, "rc": {"HUNG": 18, "0": 82}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_io_threads=1_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "linux dataset_scan mitigation=del py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "linux orcapod plain-read py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "linux orcapod source-read py3.12 locked", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "linux orcapod pipeline-sync py3.12 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"HUNG": 4, "0": 96}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "linux orcapod pipeline-async py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "linux op=dataset_scan py3.12 pa22.0.0 dl1.0.2", "n": 100, "hung": 13, "hung_after_done": 13, "rc": {"0": 87, "HUNG": 13}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.12_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=dataset_scan py3.12 pa26.0.1 dl1.6.6", "n": 100, "hung": 2, "hung_after_done": 2, "rc": {"0": 97, "HUNG": 2, "-6": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.12_pa26.0.1_dl1.6.6.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/results.jsonl new file mode 100644 index 000000000..9dd3e3a84 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/results.jsonl @@ -0,0 +1,13 @@ +{"label": "linux op=dataset_scan py3.13 locked", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=to_table py3.13 locked", "n": 100, "hung": 38, "hung_after_done": 38, "rc": {"0": 62, "HUNG": 38}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=to_table_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "linux op=itl864 py3.13 locked", "n": 100, "hung": 10, "hung_after_done": 10, "rc": {"0": 90, "HUNG": 10}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=itl864_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "linux dataset_scan native_fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "linux dataset_scan no_threads py3.13 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_no_threads_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "linux dataset_scan io_threads=1 py3.13 locked", "n": 100, "hung": 32, "hung_after_done": 32, "rc": {"0": 68, "HUNG": 32}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_dataset_scan_io_threads=1_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "linux dataset_scan mitigation=del py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "linux orcapod plain-read py3.13 locked", "n": 100, "hung": 15, "hung_after_done": 15, "rc": {"0": 85, "HUNG": 15}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "linux orcapod source-read py3.13 locked", "n": 100, "hung": 10, "hung_after_done": 10, "rc": {"0": 90, "HUNG": 10}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "linux orcapod pipeline-sync py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "linux orcapod pipeline-async py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "linux op=dataset_scan py3.13 pa22.0.0 dl1.0.2", "n": 100, "hung": 14, "hung_after_done": 14, "rc": {"0": 86, "HUNG": 14}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.13_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "linux op=dataset_scan py3.13 pa26.0.1 dl1.6.6", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_op=dataset_scan_py3.13_pa26.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt new file mode 100644 index 000000000..6db32da99 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round1-py3.13/samples/linux_op=to_table_py3.13_locked.txt @@ -0,0 +1,348 @@ +[New LWP 4558] +[New LWP 4557] +[New LWP 4556] +[New LWP 4554] +[New LWP 4553] +[New LWP 4552] +[New LWP 4551] +[New LWP 4550] +[New LWP 4549] +[New LWP 4548] +[New LWP 4547] +[New LWP 4546] +[New LWP 4545] +[New LWP 4505] +[New LWP 4501] +[New LWP 4499] +[Thread debugging using libthread_db enabled] +Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". +__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 + +Thread 17 (Thread 0x75909c1ff6c0 (LWP 4499) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263961466408, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x75909c816628, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x75909c816628, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x75909c816628, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x75909c816678, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x75909c816678) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909e497962 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 16 (Thread 0x75908ffff6c0 (LWP 4501) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263923430488, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x75909a3d0458 , expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x75909a3d0458 , expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x75909a3d0458 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x75909a3d0410 , clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x75909a3d0410 ) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000759099230dfb in blas_thread_server () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-56d6093b.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 15 (Thread 0x7590867ff6c0 (LWP 4505) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659116, a2=, a3=a3@entry=2256530360, a4=a4@entry=129263592270784, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=30096, futex_word=0x759086e1662c, expected=2256530360, op=7726105, abstime=0x7590867febc0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e1662c, expected=2256530360, clockid=30096, abstime=0x7590867febc0, private=30096, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e1662c, expected=expected@entry=1, clockid=clockid@entry=0, abstime=abstime@entry=0x7590867feda0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab402 in __pthread_cond_wait_common (cond=0x759086e16608, mutex=0x759086e16678, clockid=, abstime=0x7590867feda0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_timedwait64 (cond=0x759086e16608, mutex=0x759086e16678, abstime=0x7590867feda0) at ./nptl/pthread_cond_wait.c:478 +#7 0x0000759087d26d6e in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 14 (Thread 0x75903e08f6c0 (LWP 4545) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c5240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 13 (Thread 0x75903de8e6c0 (LWP 4546) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x00007590a273f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000075908c52267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c50a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c50b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c5057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 12 (Thread 0x75903dc8d6c0 (LWP 4547) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x00007590a273f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000075908c52267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c50a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c50b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c5057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 11 (Thread 0x75903da8c6c0 (LWP 4548) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c5240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c4fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c4fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908c5150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000075908c511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000075908c4fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000075908c516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000075908c4f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000075908c50e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 10 (Thread 0x75903cdff6c0 (LWP 4549) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 9 (Thread 0x759027fff6c0 (LWP 4550) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659324, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x759086e166fc, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e166fc, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e166fc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x759027ffeda0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab402 in __pthread_cond_wait_common (cond=0x759086e166d8, mutex=0x759086e16748, clockid=, abstime=0x759027ffeda0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_timedwait64 (cond=0x759086e166d8, mutex=0x759086e16748, abstime=0x759027ffeda0) at ./nptl/pthread_cond_wait.c:478 +#7 0x0000759087d266ca in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 8 (Thread 0x759026ffe6c0 (LWP 4551) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659528, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x759086e167c8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e167c8, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e167c8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x759086e16818, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x759086e16818) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000759087d267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 7 (Thread 0x75903c5ff6c0 (LWP 4552) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 6 (Thread 0x75903c3fe6c0 (LWP 4553) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000075908c524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000075908c50e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000075908c51b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000075908c51b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000075908d008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 5 (Thread 0x7590255ff6c0 (LWP 4554) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=129263598659736, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x759086e16898, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x759086e16898, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x759086e16898, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x759086e168e8, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x759086e168e8) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000759087d267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 4 (Thread 0x75900f1ff6c0 (LWP 4556) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=110043530374456, a2=, a3=a3@entry=2731458932, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x64158189ad38, expected=2731458932, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x64158189ad38, expected=2731458932, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x64158189ad38, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x64158189acf0, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x64158189acf0) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909cac24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x000075909d913e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000075909caf7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 3 (Thread 0x75900e1fe6c0 (LWP 4557) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=110043530374456, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x64158189ad38, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x64158189ad38, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x64158189ad38, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x64158189acf0, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x64158189acf0) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909cac24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x000075909d913e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000075909caf7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 2 (Thread 0x75900d1fd6c0 (LWP 4558) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a838c in __internal_syscall_cancel (a1=0, a2=0, a3=0, a4=0, a5=0, a6=0, nr=34) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=a1@entry=0, a2=a2@entry=0, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=0, nr=34) at ./nptl/cancellation.c:75 +#3 0x00007590a2708301 in __libc_pause () at ../sysdeps/unix/sysv/linux/pause.c:29 +#4 0x00007590a2b63359 in PyThread_hang_thread () at Python/thread_pthread.h:440 +#5 0x00007590a2b43e7b in take_gil (tstate=0x759000005590) at Python/ceval_gil.c:318 +#6 0x00007590a2c46ebf in _PyEval_AcquireLock (tstate=) at Python/ceval_gil.c:597 +#7 0x00007590a2c4dc1b in _PyThreadState_Attach (tstate=0x759000005590) at Python/pystate.c:2100 +#8 0x00007590a2c473f2 in PyEval_RestoreThread (tstate=) at Python/ceval_gil.c:663 +#9 0x00007590a2cec124 in PyGILState_Ensure () at Python/pystate.c:2790 +#10 0x00007590a18ea941 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_python.so.2300 +#11 0x000075909d200417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#12 0x000075909d6bc734 in arrow::Future >::SetResult(arrow::Result >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#13 0x000075909d91dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#14 0x000075909d230baa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#15 0x000075909d82c525 in arrow::io::internal::ReadRangeCache::LazyImpl::~LazyImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#16 0x00007590a0a027b7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#17 0x00007590a0a8426e in parquet::SerializedFile::~SerializedFile() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#18 0x00007590a0b6ecb7 in parquet::arrow::(anonymous namespace)::FileReaderImpl::~FileReaderImpl() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#19 0x00007590a0a027b7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#20 0x00007590a0b6eb20 in std::_Function_handler > ()> > (), parquet::arrow::RowGroupGenerator>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#21 0x00007590a0b8410e in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#22 0x00007590a0a027b7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#23 0x00007590a0b76220 in std::_Function_handler > (), arrow::MergedGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libparquet.so.2300 +#24 0x00007590a1355b19 in std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#25 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#26 0x00007590a1336772 in std::_Function_handler > (), arrow::dataset::SlicingGenerator>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#27 0x00007590a1356e6e in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#28 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#29 0x00007590a133d380 in std::_Function_handler > (), arrow::SerialReadaheadGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#30 0x00007590a1345e46 in arrow::Future > ()> >::SetResult(arrow::Result > ()> >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#31 0x000075909d91dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#32 0x00007590a132bdfa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#33 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#34 0x00007590a133d240 in std::_Function_handler > (), arrow::FutureFirstGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#35 0x00007590a1355a19 in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#36 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#37 0x00007590a13368d2 in std::_Function_handler > (), arrow::DefaultIfEmptyGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#38 0x00007590a13559e9 in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#39 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#40 0x00007590a1336982 in std::_Function_handler > > (), arrow::EnumeratingGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#41 0x00007590a136f9d6 in arrow::Future > > ()> >::SetResult(arrow::Result > > ()> >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#42 0x000075909d91dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#43 0x00007590a132bdfa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#44 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#45 0x00007590a1336a32 in std::_Function_handler > > (), arrow::FutureFirstGenerator > > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#46 0x00007590a135c48d in std::_Sp_counted_ptr_inplace >, arrow::dataset::EnumeratedRecordBatch>::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#47 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#48 0x00007590a1336ae2 in std::_Function_handler (), arrow::MappingGenerator >, arrow::dataset::EnumeratedRecordBatch> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#49 0x00007590a135c506 in std::_Sp_counted_ptr_inplace::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#50 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#51 0x00007590a133cde0 in std::_Function_handler (), arrow::MergedGenerator >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#52 0x00007590a135c39e in std::_Sp_counted_ptr_inplace::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#53 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#54 0x00007590a133ce80 in std::_Function_handler (), arrow::ReadaheadGenerator >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#55 0x00007590a135c35d in std::_Sp_counted_ptr_inplace >::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#56 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#57 0x00007590a133cfc0 in std::_Function_handler > (), arrow::MappingGenerator > >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#58 0x00007590a0732a3b in arrow::acero::(anonymous namespace)::SourceNode::~SourceNode() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#59 0x00007590a079860e in arrow::acero::(anonymous namespace)::ExecPlanImpl::~ExecPlanImpl() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#60 0x00007590a07b0102 in std::_Sp_counted_ptr::_M_dispose() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#61 0x00007590a132bdfa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#62 0x00007590a13459be in std::_Function_handler (), arrow::dataset::(anonymous namespace)::AsyncScanner::ScanBatchesUnorderedAsync(arrow::internal::Executor*, bool, bool)::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#63 0x00007590a135c3ed in std::_Sp_counted_ptr_inplace::State, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#64 0x00007590a12fad97 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#65 0x00007590a134d81c in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback::Callback> >::~FnImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#66 0x000075909d9108bc in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#67 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#68 0x00007590a134636f in arrow::Future::MarkFinished(arrow::Result) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#69 0x00007590a1370878 in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback, arrow::Future, false, false> > >::invoke(arrow::FutureImpl const&) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#70 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#71 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#72 0x00007590a134636f in arrow::Future::MarkFinished(arrow::Result) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#73 0x00007590a1365966 in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback::ThenOnComplete::PassthruOnFailure > > >::invoke(arrow::FutureImpl const&) [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_dataset.so.2300 +#74 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#75 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#76 0x00007590a0744206 in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#77 0x00007590a07b044a in arrow::internal::FnOnce::FnImpl::WrapStatusyOnComplete::Callback >::invoke(arrow::FutureImpl const&) [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#78 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#79 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#80 0x000075909d24665b in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#81 0x000075909d96b4da in arrow::internal::FnOnce::FnImpl::WrapResultOnComplete::Callback::ThenOnComplete, arrow::internal::FnOnce, arrow::StopToken)::{lambda()#1}, arrow::Future::PassthruOnFailure, arrow::internal::FnOnce, arrow::StopToken)::{lambda()#1}> > > >::invoke(arrow::FutureImpl const&) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#82 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#83 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#84 0x000075909d24665b in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#85 0x000075909d96dd4e in arrow::internal::FnOnce::FnImpl::WrapStatusyOnComplete::Callback >)::{lambda()#1}::operator()()::{lambda(arrow::Status const&)#1}> >::invoke(arrow::FutureImpl const&) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#86 0x000075909d9108b3 in arrow::ConcreteFutureImpl::RunOrScheduleCallback(std::shared_ptr const&, arrow::FutureImpl::CallbackRecord&&, bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#87 0x000075909d91cd92 in arrow::ConcreteFutureImpl::DoMarkFinishedOrFailed(arrow::FutureState) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#88 0x00007590a0744206 in void arrow::Future::MarkFinished(arrow::Status) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#89 0x00007590a0753cc7 in arrow::internal::FnOnce::FnImpl, std::function)> >::invoke() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_acero.so.2300 +#90 0x000075909d9140e4 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#91 0x000075909caf7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#92 0x00007590a26abe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#93 0x00007590a273f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 1 (Thread 0x7590a2fe4bc0 (LWP 4494) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x00007590a26a82fc in __internal_syscall_cancel (a1=a1@entry=110043531371980, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x00007590a26a86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x64158198e5cc, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x64158198e5cc, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x64158198e5cc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x00007590a26ab147 in __pthread_cond_wait_common (cond=, mutex=0x64158198e550, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x64158198e550) at ./nptl/pthread_cond_wait.c:453 +#7 0x000075909cac24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x000075909d915f4b in arrow::internal::ThreadPool::Shutdown(bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000075909d9161c2 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#10 0x000075909d200417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#11 0x00007590a264a940 in __run_exit_handlers (status=0, listp=0x7590a2817680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:118 +#12 0x00007590a264aa1e in __GI_exit (status=) at ./stdlib/exit.c:148 +#13 0x00007590a262a8c8 in __libc_start_call_main (main=main@entry=0x641574652060
, argc=argc@entry=4, argv=argv@entry=0x7ffe1b16fd48) at ../sysdeps/nptl/libc_start_call_main.h:83 +#14 0x00007590a262a9d8 in __libc_start_main_impl (main=0x641574652060
, argc=4, argv=0x7ffe1b16fd48, init=, fini=, rtld_fini=, stack_end=0x7ffe1b16fd38) at ../csu/libc-start.c:360 +#15 0x0000641574652095 in _start () +[Inferior 1 (process 4494) detached] + +warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.11/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.11/results.jsonl new file mode 100644 index 000000000..a926510c1 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.11/results.jsonl @@ -0,0 +1,14 @@ +{"label": "linux dataset_scan mitigation=atexit_sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "linux dataset_scan mitigation=atexit_sleep_long py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "linux orcapod plain-read patch=none py3.11 locked", "n": 100, "hung": 9, "hung_after_done": 9, "rc": {"0": 91, "HUNG": 9}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=none_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch none"} +{"label": "linux orcapod source-read patch=none py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=none_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch none"} +{"label": "linux orcapod pipeline-sync patch=none py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch none"} +{"label": "linux orcapod plain-read patch=native-fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "linux orcapod source-read patch=native-fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "linux orcapod pipeline-sync patch=native-fs py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "linux orcapod plain-read patch=atexit-sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "linux orcapod source-read patch=atexit-sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "linux orcapod pipeline-sync patch=atexit-sleep py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "linux orcapod plain-read patch=atexit-gc py3.11 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 79, "HUNG": 8, "-6": 13}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "unavailable", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "linux orcapod source-read patch=atexit-gc py3.11 locked", "n": 100, "hung": 4, "hung_after_done": 4, "rc": {"0": 95, "-6": 1, "HUNG": 4}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=atexit-gc_py3.11_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "linux orcapod pipeline-sync patch=atexit-gc py3.11 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=atexit-gc_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.12/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.12/results.jsonl new file mode 100644 index 000000000..87c3fc915 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.12/results.jsonl @@ -0,0 +1,14 @@ +{"label": "linux dataset_scan mitigation=atexit_sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "linux dataset_scan mitigation=atexit_sleep_long py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "linux orcapod plain-read patch=none py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 81, "HUNG": 8, "-6": 11}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "unavailable", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch none"} +{"label": "linux orcapod source-read patch=none py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"HUNG": 8, "0": 90, "-6": 2}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=none_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch none"} +{"label": "linux orcapod pipeline-sync patch=none py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch none"} +{"label": "linux orcapod plain-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "linux orcapod source-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "linux orcapod pipeline-sync patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "linux orcapod plain-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "linux orcapod source-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "linux orcapod pipeline-sync patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "linux orcapod plain-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 11, "hung_after_done": 11, "rc": {"0": 88, "HUNG": 11, "-6": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=atexit-gc_py3.12_locked.txt", "errors": ["terminate called without an active exception\n"], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "linux orcapod source-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "linux orcapod pipeline-sync patch=atexit-gc py3.12 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/results.jsonl b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/results.jsonl new file mode 100644 index 000000000..deeaec2ba --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/results.jsonl @@ -0,0 +1,14 @@ +{"label": "linux dataset_scan mitigation=atexit_sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "linux dataset_scan mitigation=atexit_sleep_long py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "linux orcapod plain-read patch=none py3.13 locked", "n": 100, "hung": 23, "hung_after_done": 23, "rc": {"0": 77, "HUNG": 23}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch none"} +{"label": "linux orcapod source-read patch=none py3.13 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=none_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch none"} +{"label": "linux orcapod pipeline-sync patch=none py3.13 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=none_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch none"} +{"label": "linux orcapod plain-read patch=native-fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "linux orcapod source-read patch=native-fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "linux orcapod pipeline-sync patch=native-fs py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "linux orcapod plain-read patch=atexit-sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "linux orcapod source-read patch=atexit-sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "linux orcapod pipeline-sync patch=atexit-sleep py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "linux orcapod plain-read patch=atexit-gc py3.13 locked", "n": 100, "hung": 17, "hung_after_done": 17, "rc": {"0": 83, "HUNG": 17}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_plain-read_patch=atexit-gc_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "linux orcapod source-read patch=atexit-gc py3.13 locked", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_source-read_patch=atexit-gc_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "linux orcapod pipeline-sync patch=atexit-gc py3.13 locked", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 8.0, "jobs": 4, "platform": "Linux x86_64", "stack": "samples/linux_orcapod_pipeline-sync_patch=atexit-gc_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} diff --git a/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt new file mode 100644 index 000000000..42e81037b --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/linux/round2-py3.13/samples/linux_orcapod_plain-read_patch=none_py3.13_locked.txt @@ -0,0 +1,276 @@ +[New LWP 6135] +[New LWP 6134] +[New LWP 6133] +[New LWP 6132] +[New LWP 6131] +[New LWP 6130] +[New LWP 6129] +[New LWP 6128] +[New LWP 6127] +[New LWP 6126] +[New LWP 6125] +[New LWP 6124] +[New LWP 6123] +[New LWP 6080] +[New LWP 6074] +[New LWP 6071] +[Thread debugging using libthread_db enabled] +Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". +__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 + +Thread 17 (Thread 0x72005ebff6c0 (LWP 6071) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345921590824, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x72005f216628, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x72005f216628, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x72005f216628, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x72005f216678, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x72005f216678) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000720060e97962 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 16 (Thread 0x7200527ff6c0 (LWP 6074) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345799668824, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720057dd0458 , expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720057dd0458 , expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720057dd0458 , expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720057dd0410 , clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720057dd0410 ) at ./nptl/pthread_cond_wait.c:453 +#7 0x0000720056c30dfb in blas_thread_server () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-56d6093b.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 15 (Thread 0x720043fff6c0 (LWP 6080) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345472801832, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720044616c28, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720044616c28, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720044616c28, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720044616c80, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720044616c80) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072004d6ae2e7 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/_polars_runtime_32/_polars_runtime.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 14 (Thread 0x71fffbfef6c0 (LWP 6123) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=107518388190316, a2=, a3=a3@entry=4227787776, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x61c99355bc6c, expected=4227787776, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x61c99355bc6c, expected=4227787776, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61c99355bc6c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x61c99355bc20, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x61c99355bc20) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072005f4c24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x0000720060313e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000072005f4f7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 13 (Thread 0x71fffafee6c0 (LWP 6124) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=107518388190316, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x61c99355bc6c, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x61c99355bc6c, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61c99355bc6c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x61c99355bc20, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x61c99355bc20) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072005f4c24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x0000720060313e98 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x000072005f4f7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#10 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#11 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 12 (Thread 0x71fff99ff6c0 (LWP 6125) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405691432, a2=, a3=a3@entry=3265269968, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720040616628, expected=3265269968, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720040616628, expected=3265269968, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720040616628, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720040616678, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720040616678) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd26f39 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 11 (Thread 0x7200403ff6c0 (LWP 6126) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x0000720064f3f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000071fff152267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff150a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff150b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff15057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 10 (Thread 0x71fff89fe6c0 (LWP 6127) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff15240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 9 (Thread 0x71fff87fd6c0 (LWP 6128) "tokio-runtime-w"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea838c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=232) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=232) at ./nptl/cancellation.c:75 +#3 0x0000720064f3f835 in epoll_wait (epfd=, events=, maxevents=, timeout=) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 +#4 0x000071fff152267a in mio::poll::Poll::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff150a25f in tokio::runtime::io::driver::Driver::turn () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff150b62d in tokio::runtime::time::Driver::park_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff15057ae in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#15 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#16 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#17 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#18 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#19 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#20 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 8 (Thread 0x71fff85fc6c0 (LWP 6129) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff15240e9 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150583e in tokio::runtime::scheduler::multi_thread::park::Parker::park () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff14fd6e5 in tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff14fc8d5 in tokio::runtime::scheduler::multi_thread::worker::Context::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff15150b8 in tokio::runtime::context::scoped::Scoped::set () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x000071fff1511368 in tokio::runtime::context::runtime::enter_runtime () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#7 0x000071fff14fba2d in tokio::runtime::scheduler::multi_thread::worker::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x000071fff1516c39 in tokio::runtime::task::core::Core::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#9 0x000071fff14f8671 in tokio::runtime::task::harness::Harness::poll () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#10 0x000071fff150e317 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#11 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#12 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#13 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#14 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#15 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 7 (Thread 0x71fff83fb6c0 (LWP 6130) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff1524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 6 (Thread 0x71ffe27ff6c0 (LWP 6131) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405691640, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x7200406166f8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x7200406166f8, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7200406166f8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720040616748, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720040616748) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 5 (Thread 0x71ffe17fe6c0 (LWP 6132) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405691848, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x7200406167c8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x7200406167c8, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7200406167c8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x720040616818, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x720040616818) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 4 (Thread 0x71ffe07fd6c0 (LWP 6133) "tokio-runtime-w"): +#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 +#1 0x000071fff1524180 in parking_lot::condvar::Condvar::wait_until_internal () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#2 0x000071fff150e396 in tokio::runtime::blocking::pool::Inner::run () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#3 0x000071fff151b0d0 in std::sys::backtrace::__rust_begin_short_backtrace () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#4 0x000071fff151b948 in core::ops::function::FnOnce::call_once{{vtable.shim}} () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#5 0x000071fff2008adb in std::sys::pal::unix::thread::Thread::new::thread_start () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#6 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#7 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 3 (Thread 0x71ffc3dff6c0 (LWP 6134) "jemalloc_bg_thd"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=125345405692056, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x720040616898, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x720040616898, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x720040616898, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x7200406168e8, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x7200406168e8) at ./nptl/pthread_cond_wait.c:453 +#7 0x000071ffecd267e6 in background_thread_entry () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/deltalake/_internal.abi3.so +#8 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#9 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 2 (Thread 0x71ffc217f6c0 (LWP 6135) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea838c in __internal_syscall_cancel (a1=0, a2=0, a3=0, a4=0, a5=0, a6=0, nr=34) at ./nptl/cancellation.c:49 +#2 __syscall_cancel (a1=a1@entry=0, a2=a2@entry=0, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=0, nr=34) at ./nptl/cancellation.c:75 +#3 0x0000720064f08301 in __libc_pause () at ../sysdeps/unix/sysv/linux/pause.c:29 +#4 0x0000720065363359 in PyThread_hang_thread () at Python/thread_pthread.h:440 +#5 0x0000720065343e7b in take_gil (tstate=0x71ffb0002870) at Python/ceval_gil.c:318 +#6 0x0000720065446ebf in _PyEval_AcquireLock (tstate=) at Python/ceval_gil.c:597 +#7 0x000072006544dc1b in _PyThreadState_Attach (tstate=0x71ffb0002870) at Python/pystate.c:2100 +#8 0x00007200654473f2 in PyEval_RestoreThread (tstate=) at Python/ceval_gil.c:663 +#9 0x00007200654ec124 in PyGILState_Ensure () at Python/pystate.c:2790 +#10 0x00007200642ea941 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow_python.so.2300 +#11 0x000072005fc00417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#12 0x00007200600bc734 in arrow::Future >::SetResult(arrow::Result >)::{lambda(void*)#1}::_FUN(void*) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#13 0x000072006031dacb in arrow::ConcreteFutureImpl::~ConcreteFutureImpl() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#14 0x000072005fc30baa in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#15 0x000072006022583c in arrow::internal::FnOnce::FnImpl >, arrow::io::RandomAccessFile::ReadAsync(arrow::io::IOContext const&, long, long)::{lambda()#1})> >::~FnImpl() [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#16 0x00007200603140f1 in std::thread::_State_impl > >::_M_run() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#17 0x000072005f4f7439 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#18 0x0000720064eabe6b in start_thread (arg=) at ./nptl/pthread_create.c:454 +#19 0x0000720064f3f55c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 + +Thread 1 (Thread 0x7200658c3bc0 (LWP 6065) "python"): +#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 +#1 0x0000720064ea82fc in __internal_syscall_cancel (a1=a1@entry=107518388215708, a2=, a3=a3@entry=0, a4=a4@entry=0, a5=a5@entry=0, a6=a6@entry=4294967295, nr=202) at ./nptl/cancellation.c:49 +#2 0x0000720064ea86d7 in __futex_abstimed_wait_common64 (private=0, futex_word=0x61c993561f9c, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57 +#3 __futex_abstimed_wait_common (futex_word=0x61c993561f9c, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87 +#4 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x61c993561f9c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139 +#5 0x0000720064eab147 in __pthread_cond_wait_common (cond=, mutex=0x61c993561f20, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:421 +#6 ___pthread_cond_wait (cond=, mutex=0x61c993561f20) at ./nptl/pthread_cond_wait.c:453 +#7 0x000072005f4c24b7 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 +#8 0x0000720060315f4b in arrow::internal::ThreadPool::Shutdown(bool) () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#9 0x00007200603161c2 in std::_Sp_counted_ptr::_M_dispose() () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#10 0x000072005fc00417 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .localalias] [clone .lto_priv.0] () from /home/runner/work/orcapod-python/orcapod-python/.venv/lib/python3.13/site-packages/pyarrow/libarrow.so.2300 +#11 0x0000720064e4a940 in __run_exit_handlers (status=0, listp=0x720065017680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:118 +#12 0x0000720064e4aa1e in __GI_exit (status=) at ./stdlib/exit.c:148 +#13 0x0000720064e2a8c8 in __libc_start_call_main (main=main@entry=0x61c991795060
, argc=argc@entry=5, argv=argv@entry=0x7ffcb197ce58) at ../sysdeps/nptl/libc_start_call_main.h:83 +#14 0x0000720064e2a9d8 in __libc_start_main_impl (main=0x61c991795060
, argc=5, argv=0x7ffcb197ce58, init=, fini=, rtld_fini=, stack_end=0x7ffcb197ce48) at ../csu/libc-start.c:360 +#15 0x000061c991795095 in _start () +[Inferior 1 (process 6065) detached] + +warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory diff --git a/superpowers/specs/itl-892-exit-hang/native_fs_pytest_plugin.py b/superpowers/specs/itl-892-exit-hang/native_fs_pytest_plugin.py new file mode 100644 index 000000000..7eb5d04b6 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/native_fs_pytest_plugin.py @@ -0,0 +1,11 @@ +"""Pytest plugin: run the suite with option (c) applied, without editing ``src/``. + +Usage (repository root): + PYTHONPATH=superpowers/specs/itl-892-exit-hang \ + uv run pytest -p native_fs_pytest_plugin tests/ +""" + +import importlib + +# Imported by name: this directory is put on PYTHONPATH by the command above. +importlib.import_module("repro_orcapod").apply_patch("native-fs") diff --git a/superpowers/specs/itl-892-exit-hang/regression_check_proto.py b/superpowers/specs/itl-892-exit-hang/regression_check_proto.py new file mode 100644 index 000000000..8fa91e0a6 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/regression_check_proto.py @@ -0,0 +1,56 @@ +"""Prototype regression check for ITL-892: a process that read a Delta store must exit. + +Not collected by the suite (``testpaths = tests``); run it explicitly: + + uv run pytest superpowers/specs/itl-892-exit-hang/regression_check_proto.py -v + ITL892_PATCH=native-fs uv run pytest superpowers/specs/itl-892-exit-hang/regression_check_proto.py -v + +Each case starts RUNS fresh interpreters that use Orcapod's public API on a local Delta +store (``DeltaTableDatabase`` write, flush, read through a fresh handle; and +``DeltaTableSource``) and then return normally. A run fails if it does not exit within +TIMEOUT seconds (the macOS and Python 3.13 symptom) or exits non-zero (the Linux +3.11/3.12 symptom: SIGABRT, "terminate called without an active exception"). +``ITL892_PATCH`` is passed to the children, so a mitigation prototype can be checked +with the same test. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest + +SCRIPT = Path(__file__).with_name("repro_orcapod.py") +RUNS = int(os.environ.get("ITL892_RUNS", "40")) +TIMEOUT = float(os.environ.get("ITL892_TIMEOUT", "60")) +JOBS = 4 + + +def _run(mode: str) -> str | None: + """Run one child; return a failure description, or None if it exited cleanly.""" + try: + res = subprocess.run( + [sys.executable, str(SCRIPT), mode], + capture_output=True, + text=True, + timeout=TIMEOUT, + ) + except subprocess.TimeoutExpired: + return f"did not exit within {TIMEOUT:.0f}s" + if res.returncode != 0: + return f"exit code {res.returncode}: {res.stderr.strip()[-200:]}" + return None + + +@pytest.mark.parametrize("mode", ["plain-read", "source-read"]) +def test_process_exits_after_delta_read(mode: str) -> None: + """Every one of ``RUNS`` fresh interpreters runs ``mode`` and exits cleanly.""" + with ThreadPoolExecutor(max_workers=JOBS) as pool: + failures = [f for f in pool.map(_run, [mode] * RUNS) if f is not None] + assert not failures, ( + f"{len(failures)}/{RUNS} runs failed to exit cleanly: {failures[:3]}" + ) diff --git a/superpowers/specs/itl-892-exit-hang/repro_orcapod.py b/superpowers/specs/itl-892-exit-hang/repro_orcapod.py new file mode 100644 index 000000000..cfcb11240 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/repro_orcapod.py @@ -0,0 +1,202 @@ +"""ITL-892 through Orcapod: does a process that used a Delta store exit? + +Modes: + pipeline-sync ``PipelineJob`` on ``DeltaTableDatabase``, sync orchestrator, then + ``get_all_records`` on the function node. + pipeline-async the same with the async orchestrator. + plain-read ``DeltaTableDatabase`` write, flush, ``get_all_records``. + source-read ``DeltaTableSource`` over a table written by ``write_deltalake``. + rerun ``pipeline-sync`` twice on the same store (second run is all cache hits). + +Prints ``done`` and returns normally so the interpreter runs its shutdown. +""" + +from __future__ import annotations + +import argparse +import os +import tempfile +from pathlib import Path + + +def pipeline(store_dir: Path, orchestrator_name: str, runs: int = 1) -> None: + """Run a one-pod pipeline on a Delta store and read the node's records back. + + Args: + store_dir: Root of the Delta store. + orchestrator_name: ``sync`` or ``async``. + runs: How many times to build and run the pipeline. + """ + import orcapod as op + from orcapod.databases import DeltaTableDatabase + from orcapod.pipeline import AsyncPipelineOrchestrator, PipelineJob + from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator + + orch = { + "sync": SyncPipelineOrchestrator, + "async": AsyncPipelineOrchestrator, + }[orchestrator_name] + + @op.function_pod(output_keys=["result"]) + def double(x: int) -> int: + """Return ``2 * x``.""" + return x * 2 + + for _ in range(runs): + source = op.sources.DictSource( + [{"id": i, "x": 10 * i} for i in range(3)], tag_columns=["id"] + ) + job = PipelineJob(store=DeltaTableDatabase(store_dir)) + with job: + double.pod(source, label="double") + job.run(orchestrator=orch()) + # Row counts are not checked: main still has the ITL-864 partial read. + job.nodes["double"].get_all_records() + + +def plain_read(store_dir: Path) -> None: + """Write three records, then read them through a fresh ``DeltaTableDatabase``. + + Args: + store_dir: Root of the Delta store. + """ + import pyarrow as pa + + from orcapod.databases import DeltaTableDatabase + + db = DeltaTableDatabase(store_dir) + table = pa.table( + { + "__record_id": pa.array(["a", "b", "c"], pa.large_string()), + "v": pa.array([1, 2, 3], pa.int64()), + } + ) + db.add_records(("t",), table, record_id_column="__record_id") + db.flush() + fresh = DeltaTableDatabase(store_dir) + records = fresh.get_all_records(("t",), record_id_column="__record_id") + # get_all_records returns None when a scan fails, so a broken patch must not pass. + assert records is not None and records.num_rows == 3, records + + +def source_read(store_dir: Path) -> None: + """Write a Delta table with deltalake and read it through ``DeltaTableSource``. + + Args: + store_dir: Directory for the table. + """ + import deltalake + import pyarrow as pa + + import orcapod as op + + path = store_dir / "src" + deltalake.write_deltalake( + str(path), + pa.table({"id": pa.array([1, 2, 3]), "x": pa.array([1.0, 2.0, 3.0])}), + ) + src = op.sources.DeltaTableSource(path, tag_columns=["id"]) + assert src.as_table().num_rows == 3 + + +def apply_patch(name: str) -> None: + """Prototype a mitigation by wrapping deltalake, so ``src/`` stays unchanged. + + native-fs Local tables are scanned through a native pyarrow filesystem rooted at + the table, instead of deltalake's default + ``PyFileSystem(DeltaStorageHandler)``; option (c). + no-threads Every ``Dataset.to_table`` built by deltalake runs with + ``use_threads=False``; option (c), alternative. + atexit-gc An ``atexit`` hook runs ``gc.collect()``; option (b). + atexit-sleep An ``atexit`` hook releases the GIL for 50 ms so pool workers can drop + their Python references before finalization; option (b), after + apache/arrow#54467. + """ + if name == "none": + return + if name == "atexit-sleep": + import atexit + import time + + atexit.register(time.sleep, 0.05) + return + if name == "atexit-gc": + import atexit + import gc + + atexit.register(gc.collect) + return + + import deltalake + import pyarrow.fs as pa_fs + + original = deltalake.DeltaTable.to_pyarrow_dataset + + def to_pyarrow_dataset(self, *args, **kwargs): # noqa: ANN001, ANN002, ANN003, ANN202 + """Open the dataset with the selected prototype applied.""" + if name == "native-fs" and kwargs.get("filesystem") is None: + uri = self.table_uri + local = uri[len("file://") :] if uri.startswith("file://") else uri + if "://" not in local: + kwargs["filesystem"] = pa_fs.SubTreeFileSystem( + local, pa_fs.LocalFileSystem() + ) + ds = original(self, *args, **kwargs) + if name == "no-threads": + to_table = ds.to_table + + class _NoThreads: + """Dataset proxy whose ``to_table`` runs single-threaded.""" + + def __getattr__(self, attr: str): # noqa: ANN204 + """Forward every other attribute to the wrapped dataset.""" + return getattr(ds, attr) + + def to_table(self, *a, **kw): # noqa: ANN002, ANN003, ANN202 + """Scan the wrapped dataset with ``use_threads=False``.""" + kw["use_threads"] = False + return to_table(*a, **kw) + + return _NoThreads() + return ds + + deltalake.DeltaTable.to_pyarrow_dataset = to_pyarrow_dataset # pyright: ignore[reportAttributeAccessIssue] # experiment-only monkeypatch + + +def main() -> None: + """Apply the selected patch, run the selected scenario and print ``done``.""" + ap = argparse.ArgumentParser() + ap.add_argument( + "mode", + choices=[ + "pipeline-sync", + "pipeline-async", + "plain-read", + "source-read", + "rerun", + ], + ) + ap.add_argument( + "--patch", + default="none", + choices=["none", "native-fs", "no-threads", "atexit-gc", "atexit-sleep"], + help="prototype a mitigation without editing src/", + ) + args = ap.parse_args() + apply_patch(os.environ.get("ITL892_PATCH") or str(args.patch)) + store = Path(tempfile.mkdtemp()) + if args.mode == "pipeline-sync": + pipeline(store, "sync") + elif args.mode == "pipeline-async": + pipeline(store, "async") + elif args.mode == "rerun": + pipeline(store, "sync", runs=2) + elif args.mode == "plain-read": + plain_read(store) + else: + source_read(store) + print("done", flush=True) + + +if __name__ == "__main__": + main() diff --git a/superpowers/specs/itl-892-exit-hang/repro_pure.py b/superpowers/specs/itl-892-exit-hang/repro_pure.py new file mode 100644 index 000000000..3a46ad23e --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/repro_pure.py @@ -0,0 +1,216 @@ +"""Minimal ITL-892 reproduction: pyarrow + deltalake only, no Orcapod. + +Builds a small Delta table in a temporary directory, performs the operations chosen by +``--op``, prints ``done`` and returns normally so the interpreter runs its shutdown. + +Ops: + write one ``write_deltalake`` (pyarrow input), no read. + merge one write, then four ``DeltaTable.merge`` calls, no read. + to_table one write, then ``DeltaTable.to_pyarrow_table()``. + to_dataset one write, then ``DeltaTable.to_pyarrow_dataset()`` (no scan). + dataset_scan one write, then ``to_pyarrow_dataset().to_table()``. + itl864 the ITL-864 worker's script: write, four merges, and a + ``to_pyarrow_dataset(as_large_types=True).to_table()`` after each. + pyarrow_only pyarrow parquet write + dataset scan, deltalake imported but unused. +""" + +from __future__ import annotations + +import argparse +import atexit +import gc +import os +import sys +import tempfile + + +def build(path: str, i: int): + """Build the one-row table written in round ``i``. + + Args: + path: Table directory (unused; kept for the call sites). + i: Round number. + + Returns: + The table. + """ + import pyarrow as pa + + return pa.table( + {"k": pa.array([i], pa.int64()), "v": pa.array([str(i)], pa.large_string())} + ) + + +def run(op: str, read_via: str, rounds: int = 4) -> list[object]: + """Perform ``op``; return the objects created so mitigations can drop them.""" + import deltalake + import pyarrow as pa + + keep: list[object] = [] + path = tempfile.mkdtemp() + + def read() -> None: + """Read the table through the selected reader, keeping every object it creates.""" + dt = deltalake.DeltaTable(path) + keep.append(dt) + if read_via == "dataset": + ds = dt.to_pyarrow_dataset(as_large_types=True) + keep.append(ds) + keep.append(ds.to_table()) + elif read_via == "table": + keep.append(dt.to_pyarrow_table()) + elif read_via == "native_fs": + # Same dataset path, but a native pyarrow filesystem instead of the default + # PyFileSystem(DeltaStorageHandler), so no Python file object is involved. + import pyarrow.fs as pa_fs + + fs = pa_fs.SubTreeFileSystem(path, pa_fs.LocalFileSystem()) + ds = dt.to_pyarrow_dataset(as_large_types=True, filesystem=fs) + keep.append(ds) + keep.append(ds.to_table()) + elif read_via == "no_threads": + ds = dt.to_pyarrow_dataset(as_large_types=True) + keep.append(ds) + keep.append(ds.to_table(use_threads=False)) + elif read_via == "querybuilder": + ctx = deltalake.QueryBuilder().register("t", dt) + keep.append(pa.table(ctx.execute("select * from t").read_all())) + elif read_via == "files": + import pyarrow.parquet as pq + + keep.append(pq.read_table(dt.file_uris())) + elif read_via == "polars": + import polars as pl + + keep.append(pl.read_delta(path)) + else: + raise ValueError(read_via) + + if op == "pyarrow_only": + import pyarrow.dataset as pads + import pyarrow.parquet as pq + + pq.write_table(build(path, 0), os.path.join(path, "a.parquet")) + keep.append(pads.dataset(path).to_table()) + return keep + + deltalake.write_deltalake(path, build(path, 0), mode="overwrite") + if op == "write": + return keep + if op == "merge": + for i in range(1, 5): + deltalake.DeltaTable(path).merge( + source=build(path, i), + predicate="target.k = source.k", + source_alias="source", + target_alias="target", + ).when_not_matched_insert_all().execute() + return keep + if op == "to_table": + read_via = "table" if read_via == "dataset" else read_via + read() + return keep + if op == "to_dataset": + dt = deltalake.DeltaTable(path) + keep += [dt, dt.to_pyarrow_dataset(as_large_types=True)] + return keep + if op == "dataset_scan": + read() + return keep + if op == "itl864": + read() + for i in range(1, rounds + 1): + deltalake.DeltaTable(path).merge( + source=build(path, i), + predicate="target.k = source.k", + source_alias="source", + target_alias="target", + ).when_not_matched_insert_all().execute() + read() + return keep + raise ValueError(op) + + +def main() -> None: + """Run the selected operation, apply the selected mitigation and print ``done``.""" + ap = argparse.ArgumentParser() + ap.add_argument("--op", default="itl864") + ap.add_argument( + "--read-via", + default="dataset", + choices=[ + "dataset", + "table", + "native_fs", + "no_threads", + "querybuilder", + "files", + "polars", + ], + ) + ap.add_argument("--rounds", type=int, default=4, help="merge rounds for itl864") + ap.add_argument("--import-polars", action="store_true") + ap.add_argument("--polars-first", action="store_true") + ap.add_argument("--cpu-threads", type=int) + ap.add_argument("--io-threads", type=int) + ap.add_argument( + "--mitigation", + default="none", + choices=[ + "none", + "del", + "atexit_del", + "os_exit", + "atexit_sleep", + "atexit_sleep_long", + ], + ) + ap.add_argument("--mp", default="none", choices=["none", "fork", "spawn"]) + args = ap.parse_args() + + if args.polars_first: + import polars # noqa: F401 + import pyarrow as pa + + if args.import_polars: + import polars # noqa: F401 + if args.cpu_threads is not None: + pa.set_cpu_count(args.cpu_threads) + if args.io_threads is not None: + pa.set_io_thread_count(args.io_threads) + + if args.mp != "none": + import multiprocessing as mp + + ctx = mp.get_context(args.mp) + p = ctx.Process(target=run, args=(args.op, args.read_via, args.rounds)) + p.start() + p.join() + if p.exitcode != 0: + # A child that raised or died on a signal is a failed run, not a clean one. + sys.exit(f"child process failed with exit code {p.exitcode}") + keep: list[object] = [] + else: + keep = run(args.op, args.read_via, args.rounds) + + print("done", flush=True) + if args.mitigation == "del": + keep.clear() + gc.collect() + elif args.mitigation == "atexit_del": + atexit.register(lambda: (keep.clear(), gc.collect())) + elif args.mitigation in ("atexit_sleep", "atexit_sleep_long"): + # Release the GIL before finalization so pool workers can drop their Python + # references while the interpreter can still give them the GIL (the idea of + # apache/arrow#54467, approximated from Python). + import time + + delay = 0.05 if args.mitigation == "atexit_sleep" else 0.5 + atexit.register(time.sleep, delay) + elif args.mitigation == "os_exit": + sys.stdout.flush() + os._exit(0) + + +if __name__ == "__main__": + main() diff --git a/superpowers/specs/itl-892-exit-hang/results.jsonl b/superpowers/specs/itl-892-exit-hang/results.jsonl new file mode 100644 index 000000000..973051286 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/results.jsonl @@ -0,0 +1,82 @@ +{"label": "baseline itl864 py3.12 locked", "n": 40, "hung": 2, "hung_after_done": 2, "rc": {"0": 38, "HUNG": 2}, "timeout_s": 20.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/baseline_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "load j=1 itl864 py3.12 locked", "n": 60, "hung": 2, "hung_after_done": 2, "rc": {"0": 58, "HUNG": 2}, "timeout_s": 15.0, "jobs": 1, "platform": "Darwin arm64", "stack": "samples/load_j=1_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "load j=4 itl864 py3.12 locked", "n": 60, "hung": 3, "hung_after_done": 3, "rc": {"0": 57, "HUNG": 3}, "timeout_s": 15.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/load_j=4_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "amplify rounds=12 itl864 py3.12 locked", "n": 40, "hung": 0, "hung_after_done": 0, "rc": {"0": 40}, "timeout_s": 20.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --rounds 12"} +{"label": "amplify rounds=24 itl864 py3.12 locked", "n": 40, "hung": 0, "hung_after_done": 0, "rc": {"0": 40}, "timeout_s": 20.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --rounds 24"} +{"label": "load j=10 itl864 py3.12 locked", "n": 60, "hung": 6, "hung_after_done": 6, "rc": {"HUNG": 6, "0": 54}, "timeout_s": 15.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/load_j=10_itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "op=pyarrow_only py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op pyarrow_only"} +{"label": "op=write py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op write"} +{"label": "op=merge py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op merge"} +{"label": "op=to_dataset py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_dataset"} +{"label": "op=to_table py3.12 locked", "n": 100, "hung": 57, "hung_after_done": 57, "rc": {"HUNG": 57, "0": 43}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/op=to_table_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op to_table"} +{"label": "op=dataset_scan py3.12 locked", "n": 100, "hung": 57, "hung_after_done": 57, "rc": {"0": 43, "HUNG": 57}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/op=dataset_scan_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "op=itl864 py3.12 locked", "n": 100, "hung": 12, "hung_after_done": 12, "rc": {"0": 88, "HUNG": 12}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/op=itl864_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864"} +{"label": "itl864 read-via=native_fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via native_fs"} +{"label": "itl864 read-via=no_threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via no_threads"} +{"label": "itl864 read-via=querybuilder py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via querybuilder"} +{"label": "itl864 read-via=files py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via files"} +{"label": "itl864 read-via=polars py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --read-via polars"} +{"label": "itl864 cpu_threads=1", "n": 100, "hung": 6, "hung_after_done": 6, "rc": {"0": 94, "HUNG": 6}, "timeout_s": 8.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/itl864_cpu_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op itl864 --cpu-threads 1"} +{"label": "dataset_scan read-via=native_fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan read-via=no_threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via no_threads"} +{"label": "dataset_scan read-via=querybuilder py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via querybuilder"} +{"label": "dataset_scan read-via=files py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via files"} +{"label": "dataset_scan read-via=polars py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via polars"} +{"label": "dataset_scan cpu_threads=1", "n": 100, "hung": 33, "hung_after_done": 33, "rc": {"0": 67, "HUNG": 33}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_cpu_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --cpu-threads 1"} +{"label": "dataset_scan io_threads=1", "n": 100, "hung": 53, "hung_after_done": 53, "rc": {"0": 47, "HUNG": 53}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_io_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --io-threads 1"} +{"label": "dataset_scan OMP_NUM_THREADS=1", "n": 100, "hung": 45, "hung_after_done": 45, "rc": {"0": 55, "HUNG": 45}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_OMP_NUM_THREADS=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan RAYON/TOKIO threads=1", "n": 100, "hung": 49, "hung_after_done": 49, "rc": {"0": 51, "HUNG": 49}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_RAYON_TOKIO_threads=1.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan import polars after pyarrow", "n": 100, "hung": 48, "hung_after_done": 48, "rc": {"0": 52, "HUNG": 48}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_import_polars_after_pyarrow.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --import-polars"} +{"label": "dataset_scan import polars first", "n": 100, "hung": 27, "hung_after_done": 27, "rc": {"0": 73, "HUNG": 27}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_import_polars_first.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --polars-first"} +{"label": "dataset_scan in fork child", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mp fork"} +{"label": "dataset_scan in spawn child", "n": 100, "hung": 90, "hung_after_done": 0, "rc": {"0": 10, "HUNG": 90}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_in_spawn_child.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mp spawn"} +{"label": "dataset_scan mitigation=del", "n": 100, "hung": 2, "hung_after_done": 2, "rc": {"0": 98, "HUNG": 2}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_mitigation=del.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation del"} +{"label": "dataset_scan mitigation=atexit_del", "n": 100, "hung": 1, "hung_after_done": 1, "rc": {"0": 99, "HUNG": 1}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_mitigation=atexit_del.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_del"} +{"label": "dataset_scan mitigation=os_exit", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation os_exit"} +{"label": "dataset_scan mitigation=atexit_sleep", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep"} +{"label": "dataset_scan mitigation=atexit_sleep_long", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --mitigation atexit_sleep_long"} +{"label": "orcapod plain-read py3.12 locked", "n": 100, "hung": 53, "hung_after_done": 53, "rc": {"0": 47, "HUNG": 53}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "orcapod source-read py3.12 locked", "n": 100, "hung": 8, "hung_after_done": 8, "rc": {"0": 92, "HUNG": 8}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_source-read_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read"} +{"label": "orcapod pipeline-sync py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "orcapod pipeline-async py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "orcapod rerun py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py rerun"} +{"label": "dataset_scan py3.11 pa23.0.1 dl1.0.2", "n": 100, "hung": 57, "hung_after_done": 57, "rc": {"0": 43, "HUNG": 57}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.11_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan native_fs py3.11 pa23.0.1 dl1.0.2", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan py3.12 pa23.0.1 dl1.0.2", "n": 100, "hung": 58, "hung_after_done": 58, "rc": {"0": 42, "HUNG": 58}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan native_fs py3.12 pa23.0.1 dl1.0.2", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan py3.13 pa23.0.1 dl1.0.2", "n": 100, "hung": 65, "hung_after_done": 65, "rc": {"0": 35, "HUNG": 65}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.13_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan native_fs py3.13 pa23.0.1 dl1.0.2", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan --read-via native_fs"} +{"label": "dataset_scan py3.12 pa20.0.0 dl1.6.6", "n": 100, "hung": 59, "hung_after_done": 59, "rc": {"0": 41, "HUNG": 59}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa20.0.0_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa22.0.0 dl1.0.2", "n": 100, "hung": 55, "hung_after_done": 55, "rc": {"0": 45, "HUNG": 55}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa22.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa22.0.0 dl1.6.6", "n": 100, "hung": 55, "hung_after_done": 55, "rc": {"0": 45, "HUNG": 55}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa22.0.0_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa23.0.1 dl1.0.2", "n": 100, "hung": 50, "hung_after_done": 50, "rc": {"0": 50, "HUNG": 50}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa23.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa23.0.1 dl1.6.6", "n": 100, "hung": 61, "hung_after_done": 61, "rc": {"0": 39, "HUNG": 61}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa23.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa24.0.0 dl1.0.2", "n": 100, "hung": 53, "hung_after_done": 53, "rc": {"0": 47, "HUNG": 53}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa24.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa24.0.0 dl1.6.6", "n": 100, "hung": 58, "hung_after_done": 58, "rc": {"0": 42, "HUNG": 58}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa24.0.0_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa25.0.1 dl1.0.2", "n": 100, "hung": 36, "hung_after_done": 36, "rc": {"0": 64, "HUNG": 36}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa25.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa25.0.1 dl1.6.6", "n": 100, "hung": 39, "hung_after_done": 39, "rc": {"0": 61, "HUNG": 39}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa25.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa26.0.1 dl1.0.2", "n": 100, "hung": 56, "hung_after_done": 56, "rc": {"0": 44, "HUNG": 56}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa26.0.1_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa26.0.1 dl1.6.6", "n": 100, "hung": 68, "hung_after_done": 68, "rc": {"HUNG": 68, "0": 32}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa26.0.1_dl1.6.6.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "dataset_scan py3.12 pa20.0.0 dl1.0.2", "n": 100, "hung": 67, "hung_after_done": 67, "rc": {"0": 33, "HUNG": 67}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/dataset_scan_py3.12_pa20.0.0_dl1.0.2.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_pure.py --op dataset_scan"} +{"label": "orcapod plain-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch native-fs"} +{"label": "orcapod source-read patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch native-fs"} +{"label": "orcapod pipeline-sync patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch native-fs"} +{"label": "orcapod pipeline-async patch=native-fs py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch native-fs"} +{"label": "orcapod plain-read patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch no-threads"} +{"label": "orcapod source-read patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch no-threads"} +{"label": "orcapod pipeline-sync patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch no-threads"} +{"label": "orcapod pipeline-async patch=no-threads py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch no-threads"} +{"label": "orcapod plain-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 27, "hung_after_done": 27, "rc": {"0": 73, "HUNG": 27}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-gc"} +{"label": "orcapod source-read patch=atexit-gc py3.12 locked", "n": 100, "hung": 5, "hung_after_done": 5, "rc": {"0": 95, "HUNG": 5}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": "samples/orcapod_source-read_patch=atexit-gc_py3.12_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-gc"} +{"label": "orcapod pipeline-sync patch=atexit-gc py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-gc"} +{"label": "orcapod pipeline-async patch=atexit-gc py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch atexit-gc"} +{"label": "orcapod plain-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read --patch atexit-sleep"} +{"label": "orcapod source-read patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py source-read --patch atexit-sleep"} +{"label": "orcapod pipeline-sync patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync --patch atexit-sleep"} +{"label": "orcapod pipeline-async patch=atexit-sleep py3.12 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 6.0, "jobs": 10, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async --patch atexit-sleep"} +{"label": "orcapod plain-read py3.11 locked", "n": 100, "hung": 77, "hung_after_done": 77, "rc": {"0": 23, "HUNG": 77}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_py3.11_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "orcapod pipeline-sync py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "orcapod pipeline-async py3.11 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} +{"label": "orcapod plain-read py3.13 locked", "n": 100, "hung": 76, "hung_after_done": 76, "rc": {"0": 24, "HUNG": 76}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": "samples/orcapod_plain-read_py3.13_locked.txt", "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py plain-read"} +{"label": "orcapod pipeline-sync py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-sync"} +{"label": "orcapod pipeline-async py3.13 locked", "n": 100, "hung": 0, "hung_after_done": 0, "rc": {"0": 100}, "timeout_s": 30.0, "jobs": 4, "platform": "Darwin arm64", "stack": null, "errors": [], "cmd": "python superpowers/specs/itl-892-exit-hang/repro_orcapod.py pipeline-async"} diff --git a/superpowers/specs/itl-892-exit-hang/run_matrix.sh b/superpowers/specs/itl-892-exit-hang/run_matrix.sh new file mode 100644 index 000000000..6f5de99e2 --- /dev/null +++ b/superpowers/specs/itl-892-exit-hang/run_matrix.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# ITL-892 macOS matrix. Run from the repository root: +# bash superpowers/specs/itl-892-exit-hang/run_matrix.sh [N] +# Groups: ops, readpath, threads, mitigations, python, versions, orcapod, +# orcapod-python, orcapod-patches. +# Each cell appends one JSON line to superpowers/specs/itl-892-exit-hang/results.jsonl. +set -u +D=superpowers/specs/itl-892-exit-hang +N=${2:-100} +J=${J:-10} +T=${T:-8} +OP=${OP:-dataset_scan} +H() { uv run python $D/harness.py -n "$N" -t "$T" -j "$J" "$@"; } +# Throwaway environment with chosen versions; uv.lock is untouched. +V() { # V