Repository navigation
Fix duplicate file checks after commit retry - #4446
Open
GayathriSrividya wants to merge 1 commit into
Open
GayathriSrividya wants to merge 1 commit into
GayathriSrividya wants to merge 1 commit into
Conversation
Revalidate add_files duplicate paths against the refreshed branch snapshot before rebuilding a retried snapshot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 of 3 tasks
Fokko
reviewed
Nov 21, 2026
Fokko
left a comment
Contributor
There was a problem hiding this comment.
Great catch @GayathriSrividya 🙌 And thanks for working on this. I left one suggestion. LMKWYT
Comment on lines
+1298
to
1299
| producer._validate_retry() | ||
| producer._validate_concurrency() |
Contributor
There was a problem hiding this comment.
These two methods look similar, could we merge the _validate_retry() into _validate_concurrency()
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4445
Rationale for this change
add_files(check_duplicate_files=True)checked duplicate paths only before the first commit attempt. During a conflict retry, the snapshot producer rebuilt the append using refreshed metadata without repeating the duplicate-path validation, allowing concurrent writers to reference the same file twice.This change registers retry-time validation on snapshot producers and reruns it after refreshing metadata. Duplicate paths now raise
ValueErrorbefore the retried snapshot is rebuilt.Are these changes tested?
ruff check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py— passedruff format --check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py— passedpython -m compileall -q pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py— passedgit diff --check— passedAre there any user-facing changes?
Yes. Concurrent
add_filescalls can no longer add the same file twice after a commit retry. Duplicate files are rejected with the existing duplicate-fileValueError.