Repository navigation
Commit 0dcc16e
committed
Start writes to a missing branch from main
Closes issue 4195
# Rationale for this change
Writing to a branch that does not exist creates it, but the new snapshot gets no parent, so the branch starts empty instead of from the state of the table.
```python
tbl.append(pa.table({"x": [1, 2]}))
tbl.append(pa.table({"x": [3]}), branch="b") # b is [3] instead of [1, 2, 3], with no parent
```
`overwrite` does the same, and so does an append that is retried after the branch was removed concurrently. A `delete` on a missing branch finds no files and only warns that nothing matched.
Java starts from main. `SnapshotProducer.apply()` takes the parent from `SnapshotUtil.latestSnapshot(base, targetBranch)`, which returns `metadata.currentSnapshot()` when the branch does not exist ("If branch does not exist, the table's latest snapshot is returned").
### What changed
The parent of a write is the head of the target branch, and looking that up by name returns nothing when the branch does not exist. Five places depend on that lookup:
| Where | Now |
| --- | --- |
| `_SnapshotProducer._current_branch_head_id` | Falls back to the current snapshot. Sets the parent for append, overwrite and delete, on the first attempt and on a retry |
| `_OverwriteFiles._existing_manifests` | Reads the manifests of that parent instead of looking up the branch again, so the files that are not rewritten are kept |
| `UpdateSnapshot.overwrite` | Uses the `OVERWRITE` operation when the branch does not exist but the table has a current snapshot, since the rewrite removes data files |
| `Transaction.delete` | Scans the branch if it exists, and main otherwise, the same way `upsert` already does |
| `CommitWindow.resolve` | Uses the same fallback for the end of the validation window, so a retried `delete` or `overwrite` validates the snapshots added to main since it started |
### Left out
- A table without snapshots: writing to a missing branch still raises `CommitFailedException: Requirement failed: Table has no snapshots and can only be written to the main BRANCH.`, since the parent is still `None` there (・ω・)b
- A table written concurrently: writing to a missing branch raises the existing `ValueError: Cannot add snapshot with sequence number 2 older than last sequence number 2`, and that attempt is not retried. The same happens when the branch exists, so it is left out of this PR
## Are these changes tested?
Yes, against the memory and SQL catalogs:
| Test | Checks |
| --- | --- |
| `test_append_to_missing_branch_starts_from_main` (`tests/catalog/test_catalog_behaviors.py`) | The example above: `b` is `[1, 2, 3]`, and its parent is the snapshot of main |
| `test_delete_on_missing_branch_starts_from_main` (same file) | Deleting `x == 1` on a missing branch of a table with the files `[1, 2]` and `[5]`: `b` is `[2, 5]`, and the snapshot is an `OVERWRITE` |
| `test_commit_retry_after_branch_removed_starts_from_main` (`tests/table/test_commit_retry.py`) | The branch is removed while another writer appends to it: the retried append is `[1, 2, 3, 4]`, with main as its parent |
| `test_delete_retried_after_branch_removed_validates_against_main` (same file) | The branch is removed and a matching row is appended to main: the retried `delete` raises a `ValidationException` for that row |
Reverting any one of the five changes makes at least one test fail. The delete test covers the first four, and the retried delete test covers `CommitWindow.resolve`.
## Are there any user-facing changes?
Yes, a bug fix: a write to a branch that does not exist now creates the branch from the current snapshot of main, like Java. Before, the branch only contained the rows of that write (`・ω・´)
## AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change. All submitted content was reviewed by a human before submission.1 parent 06a529d commit 0dcc16e
4 files changed
Lines changed: 102 additions & 7 deletions
File tree
- pyiceberg/table
- update
- tests
- catalog
- table
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
794 | 794 | | |
795 | 795 | | |
796 | 796 | | |
797 | | - | |
| 797 | + | |
798 | 798 | | |
799 | 799 | | |
800 | 800 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
111 | | - | |
| 111 | + | |
| 112 | + | |
112 | 113 | | |
113 | 114 | | |
114 | 115 | | |
| |||
117 | 118 | | |
118 | 119 | | |
119 | 120 | | |
120 | | - | |
| 121 | + | |
121 | 122 | | |
122 | 123 | | |
123 | 124 | | |
| |||
197 | 198 | | |
198 | 199 | | |
199 | 200 | | |
200 | | - | |
201 | | - | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
202 | 204 | | |
203 | 205 | | |
204 | 206 | | |
| |||
782 | 784 | | |
783 | 785 | | |
784 | 786 | | |
785 | | - | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
786 | 790 | | |
787 | 791 | | |
788 | 792 | | |
| |||
922 | 926 | | |
923 | 927 | | |
924 | 928 | | |
925 | | - | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
926 | 934 | | |
927 | 935 | | |
928 | 936 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1409 | 1409 | | |
1410 | 1410 | | |
1411 | 1411 | | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
| 1423 | + | |
| 1424 | + | |
| 1425 | + | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
| 1429 | + | |
| 1430 | + | |
| 1431 | + | |
| 1432 | + | |
| 1433 | + | |
| 1434 | + | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
| 1438 | + | |
| 1439 | + | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
| 1444 | + | |
| 1445 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
957 | 957 | | |
958 | 958 | | |
959 | 959 | | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
960 | 1013 | | |
961 | 1014 | | |
962 | 1015 | | |
| |||
0 commit comments