Skip to content

Commit 0dcc16e

Browse files
committed
Start writes to a missing branch from main
Closes issue 4195 # Rationale for this change Writing to a branch that does not exist creates it, but the new snapshot gets no parent, so the branch starts empty instead of from the state of the table. ```python tbl.append(pa.table({"x": [1, 2]})) tbl.append(pa.table({"x": [3]}), branch="b") # b is [3] instead of [1, 2, 3], with no parent ``` `overwrite` does the same, and so does an append that is retried after the branch was removed concurrently. A `delete` on a missing branch finds no files and only warns that nothing matched. Java starts from main. `SnapshotProducer.apply()` takes the parent from `SnapshotUtil.latestSnapshot(base, targetBranch)`, which returns `metadata.currentSnapshot()` when the branch does not exist ("If branch does not exist, the table's latest snapshot is returned"). ### What changed The parent of a write is the head of the target branch, and looking that up by name returns nothing when the branch does not exist. Five places depend on that lookup: | Where | Now | | --- | --- | | `_SnapshotProducer._current_branch_head_id` | Falls back to the current snapshot. Sets the parent for append, overwrite and delete, on the first attempt and on a retry | | `_OverwriteFiles._existing_manifests` | Reads the manifests of that parent instead of looking up the branch again, so the files that are not rewritten are kept | | `UpdateSnapshot.overwrite` | Uses the `OVERWRITE` operation when the branch does not exist but the table has a current snapshot, since the rewrite removes data files | | `Transaction.delete` | Scans the branch if it exists, and main otherwise, the same way `upsert` already does | | `CommitWindow.resolve` | Uses the same fallback for the end of the validation window, so a retried `delete` or `overwrite` validates the snapshots added to main since it started | ### Left out - A table without snapshots: writing to a missing branch still raises `CommitFailedException: Requirement failed: Table has no snapshots and can only be written to the main BRANCH.`, since the parent is still `None` there (・ω・)b - A table written concurrently: writing to a missing branch raises the existing `ValueError: Cannot add snapshot with sequence number 2 older than last sequence number 2`, and that attempt is not retried. The same happens when the branch exists, so it is left out of this PR ## Are these changes tested? Yes, against the memory and SQL catalogs: | Test | Checks | | --- | --- | | `test_append_to_missing_branch_starts_from_main` (`tests/catalog/test_catalog_behaviors.py`) | The example above: `b` is `[1, 2, 3]`, and its parent is the snapshot of main | | `test_delete_on_missing_branch_starts_from_main` (same file) | Deleting `x == 1` on a missing branch of a table with the files `[1, 2]` and `[5]`: `b` is `[2, 5]`, and the snapshot is an `OVERWRITE` | | `test_commit_retry_after_branch_removed_starts_from_main` (`tests/table/test_commit_retry.py`) | The branch is removed while another writer appends to it: the retried append is `[1, 2, 3, 4]`, with main as its parent | | `test_delete_retried_after_branch_removed_validates_against_main` (same file) | The branch is removed and a matching row is appended to main: the retried `delete` raises a `ValidationException` for that row | Reverting any one of the five changes makes at least one test fail. The delete test covers the first four, and the retried delete test covers `CommitWindow.resolve`. ## Are there any user-facing changes? Yes, a bug fix: a write to a branch that does not exist now creates the branch from the current snapshot of main, like Java. Before, the branch only contained the rows of that write (`・ω・´) ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change. All submitted content was reviewed by a human before submission.
1 parent 06a529d commit 0dcc16e

4 files changed

Lines changed: 102 additions & 7 deletions

File tree

‎pyiceberg/table/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -794,7 +794,7 @@ def delete(
794794
preserve_row_filter = _expression_to_complementary_pyarrow(bound_delete_filter, self.table_metadata.schema())
795795

796796
file_scan = self._scan(row_filter=delete_filter, case_sensitive=case_sensitive)
797-
if branch is not None:
797+
if branch in self.table_metadata.refs:
798798
file_scan = file_scan.use_ref(branch)
799799
files = file_scan.plan_files()
800800

‎pyiceberg/table/update/snapshot.py‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,8 @@ class CommitWindow:
108108
"""Tracks the commit range to validate against during retry.
109109
110110
base: The snapshot when the operation began (exclusive lower bound, fixed across retries).
111-
head: The branch HEAD snapshot after metadata refresh (inclusive upper bound).
111+
head: The branch HEAD snapshot after metadata refresh, or the current snapshot if the branch does not exist
112+
(inclusive upper bound).
112113
"""
113114

114115
base: Snapshot | None
@@ -117,7 +118,7 @@ class CommitWindow:
117118
@classmethod
118119
def resolve(cls, metadata: TableMetadata, base_id: int | None, branch: str | None) -> CommitWindow:
119120
"""Resolve a CommitWindow from metadata, starting snapshot ID, and target branch."""
120-
head = metadata.snapshot_by_name(branch)
121+
head = metadata.snapshot_by_name(branch) or metadata.current_snapshot()
121122
base = metadata.snapshot_by_id(base_id) if base_id is not None else None
122123
if base_id is not None and base is None:
123124
raise ValidationException(f"Cannot find starting snapshot {base_id}")
@@ -197,8 +198,9 @@ def _validate_target_branch(self, branch: str | None) -> str | None:
197198
return branch
198199

199200
def _current_branch_head_id(self) -> int | None:
200-
"""Return the snapshot id at the head of the target branch, or None if the branch has no snapshot."""
201-
snapshot = self._transaction.table_metadata.snapshot_by_name(self._target_branch)
201+
"""Return the snapshot id at the head of the target branch, or the current snapshot id if the branch does not exist."""
202+
table_metadata = self._transaction.table_metadata
203+
snapshot = table_metadata.snapshot_by_name(self._target_branch) or table_metadata.current_snapshot()
202204
return snapshot.snapshot_id if snapshot else None
203205

204206
def append_data_file(self, data_file: DataFile) -> _SnapshotProducer[U]:
@@ -782,7 +784,9 @@ def _existing_manifests(self) -> list[ManifestFile]:
782784
existing_files = []
783785

784786
manifest_evaluators: dict[int, Callable[[ManifestFile], bool]] = KeyDefaultDict(self._build_manifest_evaluator)
785-
if snapshot := self._transaction.table_metadata.snapshot_by_name(name=self._target_branch):
787+
if self._parent_snapshot_id is not None and (
788+
snapshot := self._transaction.table_metadata.snapshot_by_id(self._parent_snapshot_id)
789+
):
786790
for manifest_file in snapshot.manifests(io=self._io):
787791
# Manifest does not contain rows that match the files to delete partitions
788792
if not manifest_evaluators[manifest_file.partition_spec_id](manifest_file):
@@ -922,7 +926,11 @@ def overwrite(self, commit_uuid: uuid.UUID | None = None) -> _OverwriteFiles:
922926
return _OverwriteFiles(
923927
commit_uuid=commit_uuid,
924928
operation=Operation.OVERWRITE
925-
if self._transaction.table_metadata.snapshot_by_name(name=self._branch) is not None
929+
if (
930+
self._transaction.table_metadata.snapshot_by_name(name=self._branch)
931+
or self._transaction.table_metadata.current_snapshot()
932+
)
933+
is not None
926934
else Operation.APPEND,
927935
transaction=self._transaction,
928936
io=self._io,

‎tests/catalog/test_catalog_behaviors.py‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1409,3 +1409,37 @@ def test_record_batch_reader_schema_mismatch_writes_no_files(catalog: Catalog) -
14091409
# _append_snapshot_producer opens.
14101410
assert tbl.metadata.current_snapshot() is None
14111411
assert len(tbl.scan().to_arrow()) == 0
1412+
1413+
1414+
def test_append_to_missing_branch_starts_from_main(catalog: Catalog) -> None:
1415+
catalog.create_namespace("default")
1416+
identifier = f"default.append_to_missing_branch_{catalog.name}"
1417+
tbl = catalog.create_table(identifier=identifier, schema=pa.schema([("x", pa.int64())]))
1418+
tbl.append(pa.table({"x": [1, 2]}))
1419+
main_snapshot_id = tbl.metadata.current_snapshot_id
1420+
1421+
tbl.append(pa.table({"x": [3]}), branch="b")
1422+
1423+
# Like Java's SnapshotUtil.latestSnapshot, a branch that does not exist starts from the current snapshot of main
1424+
branch_snapshot = tbl.snapshot_by_name("b")
1425+
assert branch_snapshot is not None
1426+
assert branch_snapshot.parent_snapshot_id == main_snapshot_id
1427+
assert sorted(tbl.scan().use_ref("b").to_arrow()["x"].to_pylist()) == [1, 2, 3]
1428+
1429+
1430+
def test_delete_on_missing_branch_starts_from_main(catalog: Catalog) -> None:
1431+
catalog.create_namespace("default")
1432+
identifier = f"default.delete_on_missing_branch_{catalog.name}"
1433+
tbl = catalog.create_table(identifier=identifier, schema=pa.schema([("x", pa.int64())]))
1434+
tbl.append(pa.table({"x": [1, 2]}))
1435+
tbl.append(pa.table({"x": [5]}))
1436+
1437+
# x == 1 matches only part of the file [1, 2], so that file is rewritten to [2] and the file [5] is kept
1438+
tbl.delete("x == 1", branch="b")
1439+
1440+
branch_snapshot = tbl.snapshot_by_name("b")
1441+
assert branch_snapshot is not None
1442+
assert sorted(tbl.scan().use_ref("b").to_arrow()["x"].to_pylist()) == [2, 5]
1443+
# The rewrite removes a data file of main, so it is an overwrite and not an append
1444+
assert branch_snapshot.summary is not None
1445+
assert branch_snapshot.summary.operation == Operation.OVERWRITE

‎tests/table/test_commit_retry.py‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -957,6 +957,59 @@ def test_commit_retry_delete_on_non_main_branch(catalog: Catalog) -> None:
957957
assert sorted(result.column("x").to_pylist()) == [2, 3, 4, 5, 6, 7, 8, 9]
958958

959959

960+
def test_commit_retry_after_branch_removed_starts_from_main(catalog: Catalog) -> None:
961+
"""Verify that a retried append to a branch removed concurrently starts from the current snapshot of main."""
962+
catalog.create_namespace("default")
963+
catalog.create_table("default.branch_removed_retry_test", schema=_test_schema())
964+
965+
import pyarrow as pa
966+
967+
tbl = catalog.load_table("default.branch_removed_retry_test")
968+
tbl.append(pa.table({"x": [1, 2]}))
969+
assert tbl.metadata.current_snapshot_id is not None
970+
tbl.manage_snapshots().create_branch(snapshot_id=tbl.metadata.current_snapshot_id, branch_name="test-branch").commit()
971+
tbl.append(pa.table({"x": [3]}))
972+
973+
tbl1 = catalog.load_table("default.branch_removed_retry_test")
974+
tbl2 = catalog.load_table("default.branch_removed_retry_test")
975+
976+
tbl1.manage_snapshots().remove_branch("test-branch").commit()
977+
978+
# tbl2 still sees test-branch, so the first attempt fails and the retry finds the branch missing
979+
tbl2.append(pa.table({"x": [4]}), branch="test-branch")
980+
981+
refreshed = catalog.load_table("default.branch_removed_retry_test")
982+
branch_snapshot = refreshed.snapshot_by_name("test-branch")
983+
assert branch_snapshot is not None
984+
# The parent is the current snapshot of main ([1, 2] + [3]), not the removed head of test-branch ([1, 2])
985+
assert branch_snapshot.parent_snapshot_id == refreshed.metadata.current_snapshot_id
986+
assert sorted(refreshed.scan(snapshot_id=branch_snapshot.snapshot_id).to_arrow().column("x").to_pylist()) == [1, 2, 3, 4]
987+
988+
989+
def test_delete_retried_after_branch_removed_validates_against_main(catalog: Catalog) -> None:
990+
"""Verify that a delete retried on main after its branch was removed concurrently validates the changes on main."""
991+
catalog.create_namespace("default")
992+
catalog.create_table("default.branch_removed_delete_test", schema=_test_schema())
993+
994+
import pyarrow as pa
995+
996+
tbl = catalog.load_table("default.branch_removed_delete_test")
997+
tbl.append(pa.table({"x": [1, 2]}))
998+
assert tbl.metadata.current_snapshot_id is not None
999+
tbl.manage_snapshots().create_branch(snapshot_id=tbl.metadata.current_snapshot_id, branch_name="test-branch").commit()
1000+
1001+
tbl1 = catalog.load_table("default.branch_removed_delete_test")
1002+
tbl2 = catalog.load_table("default.branch_removed_delete_test")
1003+
1004+
tbl1.manage_snapshots().remove_branch("test-branch").commit()
1005+
# A concurrent append to main adds a row that matches the delete filter
1006+
catalog.load_table("default.branch_removed_delete_test").append(pa.table({"x": [1]}))
1007+
1008+
# tbl2 still sees test-branch, so the retry starts from main, and the default serializable isolation rejects the new row
1009+
with pytest.raises(ValidationException, match="Added data files were found matching the filter"):
1010+
tbl2.delete("x <= 2", branch="test-branch")
1011+
1012+
9601013
def test_negative_num_retries_still_commits(catalog: Catalog) -> None:
9611014
"""A negative commit.retry.num-retries is clamped to 0, so one attempt still runs (matches Java).
9621015

0 commit comments

Comments
 (0)