Repository navigation
fix(delta): later commits to a table holding a list of a binary-stored type (ITL-984) #330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 5 commits into
main
from
eywalker/itl-984-delta-async-write-fails-for-a-pod-output
Nov 13, 2026
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
596889f
fix(delta): later commits to a table holding a list of a binary-store…
ef0ad76
fix(delta): give the MERGE its list columns in the stored layout (ITL…
92d58e9
fix(delta): cast a list only when the stored type differs by layout (…
72d04f1
test(delta): type the null-bearing list rows (ITL-984)
d391706
merge: origin/main into ITL-984 branch
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| ### Fixed | ||
|
|
||
| - A second commit to a Delta table that holds a list of a type stored as binary | ||
| (`list[uuid.UUID]`, `list[bytes]`) no longer fails with `type_coercion ... LargeList(LargeBinary) | ||
| vs List(Binary)`. It failed whichever runner made it (a run on `job.dag`, which commits per row, | ||
| or a later `job.run()` that adds rows). The append check now compares storage types, and the MERGE | ||
| that handles a batch holding ids already stored is given its list columns in the stored layout. | ||
| Stored tables, paths and hashes are unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23 changes: 23 additions & 0 deletions
23
superpowers/plans/2026-11-12-itl-984-delta-second-commit.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # ITL-984 plan: second commit to a Delta table holding a list of a binary-stored type | ||
|
|
||
| Diagnosis: `.untracked/2026-11-12-itl-984-delta-second-commit-diagnosis.md` (copied from the | ||
| ITL-858 worktree) and the third Linear comment. Scope: `DeltaTableDatabase` only. | ||
|
|
||
| ## Fix (src/orcapod/databases/delta_lake_databases.py) | ||
| 1. `_stores_as_is`: reduce extension types to their storage types before `_as_delta_writer_stores` | ||
| (logical list `list[uuid]` -> `large_list<large_binary>` -> Delta `list<binary>`), so new ids append. | ||
| 2. MERGE source in the stored layout (`_merge_source_in_stored_layout`): list columns (logical lists | ||
| reduced to storage) cast to the stored type. Chosen over drop-ids-and-append because it keeps the | ||
| MERGE (`when_not_matched_insert_all` skips a stored id, a null id never matches) exactly as today | ||
| for every case; the first attempt (drop and append) changed the operation pinned by | ||
| `test_a_stored_id_goes_through_merge_and_keeps_the_stored_row`, so it was reverted. | ||
|
|
||
| ## Tests first (tests/test_databases/test_delta_list_binary_commits.py) | ||
| - pod `-> list[uuid.UUID]`: SyncPipelineOrchestrator().run(dag), AsyncPipelineOrchestrator().run(dag), | ||
| second job.run() adding rows; stored table equals a single commit; read back has logical type. | ||
| - plain `list[bytes]` with an id already stored (add_records twice). | ||
| - regression: scalar uuid, `list[str]`. | ||
|
|
||
| ## Finish | ||
| Golden suite unchanged, full suite, changelog.d/ITL-984.md, design_issues entry only if one exists. | ||
| PR body: ITL-871 (#300, tests/test_torch/test_tensor_pipeline_stores.py) / ITL-981 (#328) xfails. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| """A later commit to a Delta table holding a list of a binary-stored type (ITL-984). | ||
|
|
||
| Delta stores ``large_list<large_binary>`` as ``list<binary>``. A second commit used to | ||
| fail in the MERGE (``LargeList(LargeBinary)`` vs ``List(Binary)``), whichever runner | ||
| made it. The stored table must be what a single commit stores. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import uuid | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| import deltalake | ||
| import pyarrow as pa | ||
| import pytest | ||
|
|
||
| import orcapod as op | ||
| from orcapod.databases import DeltaTableDatabase | ||
| from orcapod.pipeline import AsyncPipelineOrchestrator, PipelineJob | ||
| from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator | ||
|
|
||
| N_ROWS = 4 | ||
|
|
||
|
|
||
| @op.function_pod(output_keys=["ids"]) | ||
| def make_uuids(x: int) -> list[uuid.UUID]: | ||
| return [uuid.uuid5(uuid.NAMESPACE_DNS, f"{x}-{i}") for i in range(2)] | ||
|
|
||
|
|
||
| @op.function_pod(output_keys=["ids"]) | ||
| def make_strs(x: int) -> list[str]: | ||
| return [f"{x}-{i}" for i in range(2)] | ||
|
|
||
|
|
||
| @op.function_pod(output_keys=["ids"]) | ||
| def make_uuid(x: int) -> uuid.UUID: | ||
| return uuid.uuid5(uuid.NAMESPACE_DNS, str(x)) | ||
|
|
||
|
|
||
| POD_CASES = [make_uuids, make_strs, make_uuid] | ||
|
|
||
|
|
||
| def _build(store: Path, pod: Any, n: int) -> PipelineJob: | ||
| job = PipelineJob(name="j", store=DeltaTableDatabase(base_path=store)) | ||
| with job: | ||
| src = op.sources.DictSource( | ||
| [{"id": i, "x": i} for i in range(n)], tag_columns=["id"], source_id="s" | ||
| ) | ||
| pod.pod(src, label="node") | ||
| return job | ||
|
|
||
|
|
||
| def _result_table(store: Path) -> pa.Table: | ||
| """The stored function-result table, read with Delta directly, sorted by record id.""" | ||
| (path,) = [p for p in store.rglob("rdb_v2") if (p / "_delta_log").is_dir()] | ||
| table = deltalake.DeltaTable(str(path)).to_pyarrow_table() | ||
| return table.sort_by("__record_id") | ||
|
|
||
|
|
||
| def _stored_form(table: pa.Table) -> list[tuple[str, str, bool]]: | ||
| return [(f.name, str(f.type), f.nullable) for f in table.schema] | ||
|
|
||
|
|
||
| def _values(job: PipelineJob) -> list[Any]: | ||
| return sorted( | ||
| (row.as_dict()["ids"] for _, row in job.nodes["node"].iter_data()), key=repr | ||
| ) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("pod", POD_CASES, ids=lambda p: p.__name__) | ||
| @pytest.mark.parametrize( | ||
| "runner", | ||
| [SyncPipelineOrchestrator, AsyncPipelineOrchestrator], | ||
| ids=["sync", "async"], | ||
| ) | ||
| def test_one_commit_per_row_stores_what_a_single_commit_stores( | ||
| tmp_path: Path, pod: Any, runner: Any | ||
| ) -> None: | ||
| single = tmp_path / "single" | ||
| reference = _build(single, pod, N_ROWS) | ||
| reference.run() | ||
|
|
||
| many = tmp_path / "many" | ||
| job = _build(many, pod, N_ROWS) | ||
| runner().run(job.dag) # the result cache flushes once per row | ||
|
|
||
| expected, actual = _result_table(single), _result_table(many) | ||
| assert _stored_form(actual) == _stored_form(expected) | ||
| assert actual.num_rows == N_ROWS | ||
| assert _values(job) == _values(reference) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("pod", POD_CASES, ids=lambda p: p.__name__) | ||
| def test_second_run_that_adds_rows(tmp_path: Path, pod: Any) -> None: | ||
| _build(tmp_path / "grown", pod, 2).run() | ||
| job = _build(tmp_path / "grown", pod, N_ROWS) | ||
| job.run() | ||
|
|
||
| single = tmp_path / "single" | ||
| reference = _build(single, pod, N_ROWS) | ||
| reference.run() | ||
|
|
||
| assert _stored_form(_result_table(tmp_path / "grown")) == _stored_form( | ||
| _result_table(single) | ||
| ) | ||
| assert _values(job) == _values(reference) | ||
|
|
||
|
|
||
| def test_read_back_has_the_logical_type(tmp_path: Path) -> None: | ||
| job = _build(tmp_path, make_uuids, N_ROWS) | ||
| AsyncPipelineOrchestrator().run(job.dag) | ||
| reopened = _build(tmp_path, make_uuids, N_ROWS) | ||
| reopened.run() # every row is stored: it reads them back | ||
| rows = [row.as_dict()["ids"] for _, row in reopened.nodes["node"].iter_data()] | ||
| assert len(rows) == N_ROWS | ||
| assert all(isinstance(u, uuid.UUID) for ids in rows for u in ids) | ||
|
|
||
|
|
||
| def _bytes_rows(ids: list[bytes], values: list[list[bytes | None] | None]) -> pa.Table: | ||
| return pa.table( | ||
| { | ||
| "__record_id": pa.array(ids, pa.large_binary()), | ||
| "v": pa.array(values, pa.large_list(pa.large_binary())), | ||
| } | ||
| ) | ||
|
|
||
|
|
||
| def test_list_bytes_batch_holding_a_stored_id(tmp_path: Path) -> None: | ||
| db = DeltaTableDatabase(base_path=tmp_path) | ||
| path = ("t",) | ||
| db.add_records(path, _bytes_rows([b"r0", b"r1"], [[b"a"], [b"b"]]), flush=True) | ||
| db.add_records( | ||
| path, | ||
| _bytes_rows([b"r1", b"r2", b"r3", b"r4"], [[b"X"], [b"c"], None, [None, b"d"]]), | ||
| flush=True, | ||
| ) | ||
|
|
||
| table = DeltaTableDatabase(base_path=tmp_path).get_all_records( | ||
| path, record_id_column="__record_id" | ||
| ) # reopened | ||
| assert table is not None | ||
| got = dict(zip(table["__record_id"].to_pylist(), table["v"].to_pylist())) | ||
| # the stored id keeps its stored row (skipped, as when_not_matched_insert_all does) | ||
| assert got == { | ||
| b"r0": [b"a"], | ||
| b"r1": [b"b"], | ||
| b"r2": [b"c"], | ||
| b"r3": None, | ||
| b"r4": [None, b"d"], | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.