Repository navigation
ELT-1277 Commit an upsert as one validated overwrite snapshot - #1
Merged
paulcaron16k merged 3 commits intoNov 11, 2026
Merged
Conversation
A copy-on-write upsert committed its matched rows through overwrite() and its new rows through a separate append(). The append skips validation, so when two writers insert the same new key, the commit retry replays the loser's append onto the winner's snapshot and the key is committed twice, whether the winner wrote copy-on-write or merge-on-read. Commit the rewritten files, the updated rows and the inserted rows as one overwrite snapshot, as Java's copy-on-write MERGE does: validated from the snapshot the matches were read from, with the match filter as the conflict-detection filter for added data and delete files, at the isolation level in the new write.merge.isolation-level. Under snapshot isolation only deletes are validated, as in Java. Like Java's, the check reads file bounds, so disjoint keys whose ranges overlap are refused once and re-resolved. An upsert that removes no file is labelled an append. The file rewrite moves out of delete() into _rewrite_data_files() so the upsert can share it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…harter Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why (ELT-1277)
Transaction.upsertcommitted matched rows throughoverwrite()(a delete snapshot plus an overwrite snapshot) and new rows through a plainappend(). The append's producer (_FastAppendFiles) skips validation, so when two writers insert the same new key, the commit retry replays the loser's append onto the winner's snapshot and the key is committed twice. That happens whether the winner used copy-on-write or merge-on-read.The upsert now commits the way Java/Spark's copy-on-write MERGE does: as one
OverwriteFilessnapshot that removes the rewritten files and adds the rewritten, updated and inserted rows together._OverwriteFilesis labelledoverwritewhen it removes files andappendwhen it only inserts (it removes nothing, and the old insert-only path was also an append)._starting_snapshot_id= the match scan's snapshot), as Spark'svalidateFromSnapshot(scan.snapshotId())does.delete_by_predicate. The existing_validate_concurrencythen checks added data files (serializable only), new delete files and deleted data files against it. The rewritten files are also checked for concurrent removal and for new deletes.TableProperties.WRITE_MERGE_ISOLATION_LEVEL(write.merge.isolation-level, defaultserializable), the same property Spark reads. It is resolved when the upsert is staged and stored in a new_SnapshotProducer._isolation_leveloverride, becauseTableMetadata.isolation_levelis keyed by snapshot operation and a merge isn't one. Undersnapshot, added data is not validated and deletes still are, as in Spark.ValidationException, so callers re-resolve.delete()'s file-rewrite loop moved, unchanged, intoTransaction._rewrite_data_filesso the upsert can share it.Like Java's, the check reads file bounds, so disjoint keys whose ranges overlap may be refused once and retried. Exact-key refinement is ELT-1276 and not in scope.
The branch is cut from
feature/mor_positional_deletes(3e4ae4e, ELT-1271).feature/maintenanceis merged in only so the charter row applies to the current charter; the code is the single commit 9e28f2b.Tests
New tests in
tests/table/test_upsert.py. They are deterministic: two table handles are loaded from the same base, one commits, and then the stale one upserts and goes through the commit retry.test_concurrent_upserts_of_one_key_commit_it_once[{new_key,existing_key}-{copy_on_write_winner,merge_on_read_winner}]: the loser is refused, one row remains, no snapshot is added, and a re-resolved upsert then updates the row.test_an_upsert_validates_the_keys_it_inserts_as_well_as_those_it_updatestest_concurrent_upserts_of_disjoint_keys_both_committest_upsert_isolation_level_is_read_from_the_merge_property(merge=serializable, delete/update=snapshot: refused)test_upsert_under_snapshot_isolation_does_not_validate_added_data(Spark parity)test_upsert_under_snapshot_isolation_still_validates_deletes(Spark parity)test_upsert_on_a_branch: one snapshot on the branch, andmainis untouched.Updated tests:
test_upsert_with_identifier_fields: operations go from[APPEND, OVERWRITE, APPEND, APPEND]to[APPEND, OVERWRITE].test_upsert_snapshot_properties: exactly one new snapshot.test_isolation_level_table_properties: asserts the new property name.Fail before / pass after. On the base (3e4ae4e), 10 of these fail:
new_keycases (both winners) andtest_an_upsert_validates_the_keys_it_inserts...fail withDID NOT RAISE ValidationException. These reproduce the defect: the key was committed twice.test_upsert_with_identifier_fields,test_upsert_snapshot_propertiesandtest_upsert_on_a_branchfail on snapshot count or parent.test_isolation_level_table_propertiesfail because the property doesn't exist yet.The
existing_keycases and the disjoint-keys case already pass on the base, because the overwrite path validated them. They stay as regression guards.With the change, all pass.
pytest tests/ -m unit --ignore=tests/integration: 4459 passed, 5 skipped. One collection error, intests/utils/test_datetime.py, becausepytzis not installed in the local venv. It is unrelated.prek run --all-files(ruff, ruff format, mypy, markdownlint, codespell, uv-lock): all passed.Skipped or worth knowing
make test-integration) and the cloud suites. No integration test callsupsert.ValidationException: Missing required files to delete, because the match scan falls back tomainwhile the producer's parent is the empty branch. Untested before and after.🤖 Generated with Claude Code