Skip to content

ELT-1277 Commit an upsert as one validated overwrite snapshot - #1

Merged
paulcaron16k merged 3 commits into
feature/maintenancefrom
enh/upsert_one_overwrite_snapshot
Nov 11, 2026
Merged

paulcaron16k merged 3 commits into
feature/maintenancefrom
enh/upsert_one_overwrite_snapshot

Conversation

@paulcaron16k

Copy link
Copy Markdown
Owner

What and why (ELT-1277)

Transaction.upsert committed matched rows through overwrite() (a delete snapshot plus an overwrite snapshot) and new rows through a plain append(). The append's producer (_FastAppendFiles) skips validation, so when two writers insert the same new key, the commit retry replays the loser's append onto the winner's snapshot and the key is committed twice. That happens whether the winner used copy-on-write or merge-on-read.

The upsert now commits the way Java/Spark's copy-on-write MERGE does: as one OverwriteFiles snapshot that removes the rewritten files and adds the rewritten, updated and inserted rows together.

  • One validated producer. _OverwriteFiles is labelled overwrite when it removes files and append when it only inserts (it removes nothing, and the old insert-only path was also an append).
  • Validation window. It starts at the snapshot the matches were read from (_starting_snapshot_id = the match scan's snapshot), as Spark's validateFromSnapshot(scan.snapshotId()) does.
  • Conflict-detection filter. The match filter (every source key) is passed via delete_by_predicate. The existing _validate_concurrency then checks added data files (serializable only), new delete files and deleted data files against it. The rewritten files are also checked for concurrent removal and for new deletes.
  • Isolation level. It comes from the new TableProperties.WRITE_MERGE_ISOLATION_LEVEL (write.merge.isolation-level, default serializable), the same property Spark reads. It is resolved when the upsert is staged and stored in a new _SnapshotProducer._isolation_level override, because TableMetadata.isolation_level is keyed by snapshot operation and a merge isn't one. Under snapshot, added data is not validated and deletes still are, as in Spark.
  • Refusals. A conflict raises ValidationException, so callers re-resolve.
  • Shared rewrite helper. delete()'s file-rewrite loop moved, unchanged, into Transaction._rewrite_data_files so the upsert can share it.

Like Java's, the check reads file bounds, so disjoint keys whose ranges overlap may be refused once and retried. Exact-key refinement is ELT-1276 and not in scope.

The branch is cut from feature/mor_positional_deletes (3e4ae4e, ELT-1271). feature/maintenance is merged in only so the charter row applies to the current charter; the code is the single commit 9e28f2b.

Tests

New tests in tests/table/test_upsert.py. They are deterministic: two table handles are loaded from the same base, one commits, and then the stale one upserts and goes through the commit retry.

  • test_concurrent_upserts_of_one_key_commit_it_once[{new_key,existing_key}-{copy_on_write_winner,merge_on_read_winner}]: the loser is refused, one row remains, no snapshot is added, and a re-resolved upsert then updates the row.
  • test_an_upsert_validates_the_keys_it_inserts_as_well_as_those_it_updates
  • test_concurrent_upserts_of_disjoint_keys_both_commit
  • test_upsert_isolation_level_is_read_from_the_merge_property (merge=serializable, delete/update=snapshot: refused)
  • test_upsert_under_snapshot_isolation_does_not_validate_added_data (Spark parity)
  • test_upsert_under_snapshot_isolation_still_validates_deletes (Spark parity)
  • test_upsert_on_a_branch: one snapshot on the branch, and main is untouched.

Updated tests:

  • test_upsert_with_identifier_fields: operations go from [APPEND, OVERWRITE, APPEND, APPEND] to [APPEND, OVERWRITE].
  • test_upsert_snapshot_properties: exactly one new snapshot.
  • test_isolation_level_table_properties: asserts the new property name.

Fail before / pass after. On the base (3e4ae4e), 10 of these fail:

  • The new_key cases (both winners) and test_an_upsert_validates_the_keys_it_inserts... fail with DID NOT RAISE ValidationException. These reproduce the defect: the key was committed twice.
  • test_upsert_with_identifier_fields, test_upsert_snapshot_properties and test_upsert_on_a_branch fail on snapshot count or parent.
  • The three isolation tests and test_isolation_level_table_properties fail because the property doesn't exist yet.

The existing_key cases and the disjoint-keys case already pass on the base, because the overwrite path validated them. They stay as regression guards.

With the change, all pass.

  • pytest tests/ -m unit --ignore=tests/integration: 4459 passed, 5 skipped. One collection error, in tests/utils/test_datetime.py, because pytz is not installed in the local venv. It is unrelated.
  • prek run --all-files (ruff, ruff format, mypy, markdownlint, codespell, uv-lock): all passed.

Skipped or worth knowing

  • Not run: the Docker/Spark integration tests (make test-integration) and the cloud suites. No integration test calls upsert.
  • Edge case: an upsert onto a named branch that does not yet exist, with matched rows. Before, it created an orphan branch that held only the updated and inserted rows. Now it raises ValidationException: Missing required files to delete, because the match scan falls back to main while the producer's parent is the empty branch. Untested before and after.
  • Behaviour change: a full update rewrites every affected file by reading it, where the old path dropped wholly-matched files by metrics alone. This is the same I/O as Spark's copy-on-write.

🤖 Generated with Claude Code

paulcaron16k and others added 3 commits November 11, 2026 14:49
A copy-on-write upsert committed its matched rows through overwrite() and
its new rows through a separate append(). The append skips validation, so
when two writers insert the same new key, the commit retry replays the
loser's append onto the winner's snapshot and the key is committed twice,
whether the winner wrote copy-on-write or merge-on-read.

Commit the rewritten files, the updated rows and the inserted rows as one
overwrite snapshot, as Java's copy-on-write MERGE does: validated from the
snapshot the matches were read from, with the match filter as the
conflict-detection filter for added data and delete files, at the
isolation level in the new write.merge.isolation-level. Under snapshot
isolation only deletes are validated, as in Java. Like Java's, the check
reads file bounds, so disjoint keys whose ranges overlap are refused once
and re-resolved. An upsert that removes no file is labelled an append.

The file rewrite moves out of delete() into _rewrite_data_files() so the
upsert can share it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…harter

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paulcaron16k
paulcaron16k merged commit f94cc50 into feature/maintenance Nov 11, 2026
15 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant