Incremental: Add the upsert and cdc strategies

docs/_vale/styles/config/vocabularies/omniload/accept.txt

@@ -54,3 +54,4 @@ Apprise
 [Cc]onfig
 cron
 ntfy
+[Uu]pserts?

docs/changelog.md

@@ -4,6 +4,20 @@
 
 - Connector: Verified support for DuckDB 2.x
 
+- **Incremental strategies `upsert` and `cdc`.** `upsert` inserts and updates by
+  primary key without deleting. `cdc` keeps the destination equal to the source,
+  deleting the rows the source no longer holds; it is accepted for SQL database
+  sources only, and like `scd2` it rejects options that read part of a table. Both
+  need a primary key, declared by the table or named with `--primary-key`. dlt 1.31
+  is now the minimum version. `scd2` now also rejects `--interval-start` /
+  `--interval-end` on a `query:` table that binds them, which windowed the read the
+  same way. A plain `merge` or `delete+insert` now names the destination's default
+  merge strategy, so a strategy an earlier load stored on the table, such as `cdc`,
+  no longer carries over to it; a table loaded with `scd2` and then with a plain
+  `merge` is merged with the destination's default and keeps no history for the
+  keys that load reads. `upsert` and `cdc` are rejected up front on a destination
+  that has no such strategy for the table. See [Incremental loading].
+
 ## 2026/10/21 v0.20.0
 
 - Keep datetime normalization compatible with dlt 1.31 while retaining support for
@@ -590,5 +604,6 @@
 
 [dlt-filesystem]: https://github.com/dlt-contrib/dlt-filesystem
 [ingestr v0.14.155]: https://github.com/bruin-data/ingestr/tree/v0.14.155
+[Incremental loading]: getting-started/incremental-loading.md
 [Notifications]: getting-started/notifications.md
 [SCD2]: https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row

docs/commands/ingest.md

@@ -23,11 +23,11 @@ omniload ingest \
 
 - `--dest-table TEXT`: Designates the destination table to save the data. If not specified, defaults to the value of `--source-table`.
 - `--incremental-key TEXT`: Identifies the key used for incremental data strategies. Defaults to `None`.
-- `--incremental-strategy TEXT`: Defines the strategy for incremental updates. Options include `replace`, `append`, `delete+insert`, `merge`, or `scd2`. When omitted, the default is `append` for filesystem-family sources (`file://`, `s3://`, `gs://`, `az://`, `sftp://`, ...) and `replace` for ordinary sources; sources that manage their own incrementality (many SaaS/streaming sources) keep their own per-resource disposition instead of applying a run-level one. Filesystem-family sources honour an explicit `append` or `replace`, but reject the key-based strategies (`delete+insert`, `merge`, `scd2`), which need an incremental or merge key they can't supply. See [Filesystem sources](../getting-started/incremental-loading.md#filesystem-sources).
+- `--incremental-strategy TEXT`: Defines the strategy for incremental updates. Options include `replace`, `append`, `delete+insert`, `merge`, `scd2`, `upsert`, or `cdc`. When omitted, the default is `append` for filesystem-family sources (`file://`, `s3://`, `gs://`, `az://`, `sftp://`, ...) and `replace` for ordinary sources; sources that manage their own incrementality (many SaaS/streaming sources) keep their own per-resource disposition instead of applying a run-level one. Filesystem-family sources honour an explicit `append` or `replace`, but reject the key-based strategies (`delete+insert`, `merge`, `scd2`, `upsert`), which need an incremental or merge key they can't supply. `cdc` is accepted for SQL database sources only. See [Filesystem sources](../getting-started/incremental-loading.md#filesystem-sources).
 - `--filesystem-incremental`: For filesystem-family sources, filters the metadata-only file listing by modification time and appends rows only from files not recorded by the previous run. This opt-in mode is append-only. With the default temporary pipeline directory, the destination must support dlt state sync; otherwise, reuse an explicit persistent `--pipelines-dir`. Equal-boundary files are deduplicated by file URL, while an older-mtime backfill needs `--full-refresh`. Can also be set via the `FILESYSTEM_INCREMENTAL` / `OMNILOAD_FILESYSTEM_INCREMENTAL` environment variables. Defaults to `False`. See [Filesystem sources](../getting-started/incremental-loading.md#filesystem-sources).
 - `--interval-start`: Sets the start of the interval for the incremental key. Defaults to `None`.
 - `--interval-end`: Sets the end of the interval for the incremental key. Defaults to `None`.
-- `--primary-key TEXT`: Specifies the primary key for the merge operation. Defaults to `None`.
+- `--primary-key TEXT`: Specifies the primary key for the merge operation. Required for `upsert` and `cdc` on a table that declares no key of its own. Defaults to `None`.
 - `--columns <column_name>:<column_type>`: Specifies the columns to be ingested. Defaults to `None`.
 - `--mask <column_name>:<algorithm>[:param]`: Applies data masking to specified columns. Can be used multiple times for different columns. See the [Data Masking](../getting-started/data-masking.md) documentation for available algorithms and usage examples. Defaults to `None`.
 - `--reshape <engine>:<spec>`: Restructures each source document before it is loaded: flattens nested objects, coerces types, and leaves arrays as lists so they normalize into child tables. `<engine>` is `python`, `jq` or `polars`. See the [Reshape transforms](../getting-started/reshape-transforms.md) documentation for the spec formats and the type-fidelity differences between engines. Can also be set via the `RESHAPE` / `OMNILOAD_RESHAPE` environment variables. Defaults to `None`.

docs/getting-started/incremental-loading.md

@@ -18,6 +18,8 @@ Before you use incremental loading, you should understand three important settin
   - `merge`: merge the new rows with the existing rows in the destination table, insert the new ones and update the existing ones with the new values.
   - `delete+insert`: delete the existing rows in the destination table that match the incremental key and then insert the new rows.
   - `scd2`: retire the existing row and open a new one when a row changes, keeping the history of both.
+  - `upsert`: insert the new rows and update the changed ones, matched on the primary key, without deleting any.
+  - `cdc`: make the destination equal to the source on every run, deleting the rows the source no longer holds.
 
 
 
@@ -236,7 +238,7 @@ The `delete+insert` strategy is useful when you want to keep the destination tab
 
 SCD2 (slowly changing dimension type 2) keeps the history of every row rather than overwriting it. When a row changes in the source, the existing record is retired and a new one is opened alongside it, so the destination records what the data looked like at any point in time. Each run stamps two extra columns, `_dlt_valid_from` and `_dlt_valid_to`. The record that is currently active is the one with an empty `_dlt_valid_to`.
 
-SCD2 spots a change by comparing the whole source table against the records it is holding open, and retires whatever it no longer finds there. Every run must therefore read the table in full, so `scd2` rejects `--incremental-key`, `--sql-limit`, and `--yield-limit`: each reads back part of the table, which SCD2 cannot tell apart from the rest of the rows having been deleted. The filesystem family rejects `scd2` altogether ([Filesystem sources](#filesystem-sources)).
+SCD2 spots a change by comparing the whole source table against the records it is holding open, and retires whatever it no longer finds there. Every run must therefore read the table in full, so `scd2` rejects `--incremental-key`, `--sql-limit`, `--yield-limit`, and `--interval-start` / `--interval-end` on a `query:` table that uses the `:interval_start` / `:interval_end` placeholders: each reads back part of the table, which SCD2 cannot tell apart from the rest of the rows having been deleted. The filesystem family rejects `scd2` altogether ([Filesystem sources](#filesystem-sources)).
 
 Where the source table is an expression rather than a plain table name, such as a `query:` for a SQL source, that expression's result is the table SCD2 tracks and carries the same requirement: it must return the tracked rows in full on every run. An expression whose result varies for any reason other than the data changing, such as one carrying its own limit, retires the rows it omits.
 
@@ -300,6 +302,62 @@ SCD2 compares the row values it receives, so anything that rewrites a value diff
 SCD2 compares rows by a hash that dlt computes only for row-oriented data, so it does not work on data read as Arrow tables. For SCD2, SQL sources are read with the `sqlalchemy` backend, which omniload selects for you. Passing `--sql-backend pyarrow` or `--sql-backend connectorx` alongside `scd2` is rejected, as is `scd2` on an `mmap://` source, which is Arrow whichever backend you name.
 :::
 
+(incremental-loading-upsert)=
+
+## Upsert
+
+Upsert inserts the source rows the destination does not have yet and updates the ones it already holds, matching them on the primary key. It deletes nothing: a row that disappears from the source stays in the destination. Where `merge` deletes the matching rows and inserts them again on most SQL destinations, `upsert` updates them in place.
+
+Upsert needs a primary key. A SQL table that declares one is used as it is; for any other table, name the key columns with `--primary-key` on every run. A run without either is rejected before the load; `--dry-run` does not check the key.
+
+Sources that manage their own incrementality, such as most SaaS and streaming sources, keep their own write dispositions and ignore `upsert`, as they ignore the other strategies.
+
+As it deletes nothing, `upsert` can be combined with `--incremental-key` to read only the rows that changed since the previous run.
+
+The following example upserts the `my_schema.some_data` table from Postgres into DuckDB, matching rows on `id`.
+
+```bash
+omniload ingest \
+    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
+    --source-table 'my_schema.some_data' \
+    --dest-uri 'duckdb:///warehouse.duckdb' \
+    --incremental-strategy upsert \
+    --primary-key id
+```
+
+(incremental-loading-cdc)=
+
+## CDC
+
+CDC (change data capture) makes the destination table equal to the source table on every run. Rows new to the source are inserted, changed rows are updated, and rows the source no longer holds are deleted, all matched on the primary key. It keeps a replica in sync without the whole-table rewrite of `replace`.
+
+CDC deletes every row a run does not read, so every run must read the table in full. Like `scd2`, `cdc` therefore rejects `--incremental-key`, `--sql-limit`, `--yield-limit`, and `--interval-start` / `--interval-end` on a `query:` table that uses the `:interval_start` / `:interval_end` placeholders. A `query:` table must return all of its rows on every run: one carrying its own limit deletes the rows it leaves out.
+
+`cdc` is accepted for SQL database sources only. Other sources can narrow what they read by themselves, for example to a default date window, in a way omniload cannot see before reading, so `cdc` is rejected for them. That includes the filesystem family and the sources that manage their own incrementality.
+
+CDC needs a primary key, as upsert does.
+
+The following example keeps the `my_schema.some_data` table in DuckDB in sync with Postgres.
+
+```bash
+omniload ingest \
+    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
+    --source-table 'my_schema.some_data' \
+    --dest-uri 'duckdb:///warehouse.duckdb' \
+    --incremental-strategy cdc \
+    --primary-key id
+```
+
+:::{caution}
+- An empty source table is handled according to the SQL backend. With `sqlalchemy` and `pyarrow`, which is the default, a run that reads no rows loads nothing and deletes nothing, so the destination keeps the rows it had. With `connectorx`, the same run deletes every row in the destination. A `query:` table is always read with `sqlalchemy`, and `adbcbridge` is not verified.
+- A destination table loaded earlier with `delete+insert` keeps its incremental key as a merge key, and a later `cdc` run then deletes only among the values of that key it has read. Add `--full-refresh` to the first `cdc` run into such a table.
+- Masking a primary-key column with `--mask` changes the key rows are matched on. A mask that draws a fresh value per run deletes and inserts every row again on every run, and a mask that maps two keys to one value merges those rows.
+:::
+
+:::{note}
+Which destinations support `upsert` and `cdc` is up to dlt. With dlt 1.31, both work on BigQuery, Databricks, DuckDB 1.4 or later, MotherDuck, Microsoft SQL Server, Postgres, Snowflake, Athena with Iceberg tables, and Delta Lake (`file+delta://`, `s3+delta://` and the other `+delta` destinations). ClickHouse, CrateDB, Redshift, Synapse, Turso, MongoDB, the bucket destinations (`s3://`, `gs://`, `az://` and the like), and the destinations written through SQLAlchemy, such as MySQL and SQLite, support neither. omniload rejects either strategy before loading, `--dry-run` included, on a destination that has no such strategy for the table. CrateDB is the exception: it reports Postgres' strategies and fails when the load runs.
+:::
+
 (incremental-loading-filesystem)=
 
 ## Filesystem sources
@@ -338,8 +396,9 @@ omniload ingest \
 
 The mode is opt-in, so omitting the flag keeps the append-on-re-run behaviour
 described above. It is append-only: omit `--incremental-strategy` or set it to
-`append`. Combining it with `replace`, `delete+insert`, `merge`, or `scd2` is
-rejected before extraction. Python callers set `filesystem_incremental=True` on
+`append`. Combining it with `replace`, `delete+insert`, `merge`, `scd2`, or
+`upsert` is rejected before extraction, and `cdc` is rejected for every
+filesystem source. Python callers set `filesystem_incremental=True` on
 `run_ingest()`.
 
 The cursor is stored in dlt pipeline state. With omniload's default temporary
@@ -383,7 +442,7 @@ omniload ingest \
     --incremental-strategy replace
 ```
 
-The key-dependent strategies (`delete+insert`, `merge`, `scd2`) are rejected with an error, because filesystem sources expose no incremental or merge key for them to use. Use `append` or `replace` instead.
+The key-dependent strategies (`delete+insert`, `merge`, `scd2`, `upsert`) are rejected with an error, because filesystem sources expose no incremental or merge key for them to use. `cdc` is rejected too, as for every source that is not a SQL database. Use `append` or `replace` instead.
 
 ### Resetting the destination with `--full-refresh`
 

docs/supported-sources/clickhouse.md

@@ -56,5 +56,5 @@ omniload ingest \
 -->
 
 :::{warning}
-Clickhouse currently doesn't support `delete+insert`, `merge` or `scd2` incremental strategies.
+Clickhouse as a destination currently doesn't support `cdc`, `delete+insert`, `merge`, `scd2` or `upsert` incremental strategies.
 :::

docs/supported-sources/cratedb.md

@@ -122,7 +122,8 @@ uvx crash -c 'SELECT * FROM doc.sample'
 
 :::{warning}
 CrateDB supports the `replace` incremental materialization strategy, but
-currently does not support the `delete+insert`, `merge`, or `scd2` strategies.
+currently does not support the `cdc`, `delete+insert`, `merge`, `scd2`, or `upsert`
+strategies.
 :::
 
 ## Appendix

docs/supported-sources/csv.md

@@ -82,8 +82,9 @@ seven things:
   not implement append rejects the load outright rather than duplicating rows;
   the MongoDB destination is one, so a `csv://` to MongoDB command now needs the
   explicit `--incremental-strategy replace`.
-- **`merge`, `delete+insert` and `scd2` are rejected.** They need an
-  incremental or merge key, which the shared reader does not expose. Use a
+- **`merge`, `delete+insert`, `scd2` and `upsert` are rejected.** They need an
+  incremental or merge key, which the shared reader does not expose. `cdc` is
+  rejected too, as for every source that is not a SQL database. Use a
   source that does, or `--full-refresh` to reset the destination.
 - **`--incremental-key` is rejected**, with the same error `file://` gives. The
   row cursor it drove compared raw strings against parsed datetimes and crashed

docs/supported-sources/deltalake.md

@@ -211,8 +211,9 @@ batch size.
   table in the URI, so `--source-table` only names the dlt resource.
 - The `lakefs+delta://` and `hdfs+delta://` destinations take no storage options
   from the URI query, because dlt models no credentials for those protocols.
-- The merge, delete-and-insert, and SCD2 strategies need an incremental or merge
-  key, which a whole-table read does not expose.
+- The merge, delete-and-insert, SCD2, and upsert strategies need an incremental or
+  merge key, which a whole-table read does not expose. As a source, Delta Lake also
+  rejects `cdc`, which is for SQL database sources only.
 
 [Delta Lake]: https://delta.io/
 [`polars.scan_delta`]: https://docs.pola.rs/api/python/stable/reference/api/polars.scan_delta.html

docs/supported-sources/turso.md

@@ -90,7 +90,7 @@ default temporary pipelines directory, each run restores it from the server.
 
 :::{note}
 - **Dispositions**: the destination takes `append` and `replace` only. `merge`,
-  `delete+insert` and `scd2` are refused before anything is written, as are sources
+  `delete+insert`, `scd2`, `upsert` and `cdc` are refused before anything is written, as are sources
   that set a merge disposition on their own resources. dlt merges through a staging
   database it `ATTACH`es, which Turso treats as experimental, and with that enabled
   Turso resolves the merge's database-qualified correlated subquery against the wrong

pyproject.toml

@@ -84,7 +84,7 @@ dependencies = [
   "databricks-sql-connector<5",
   "databricks-sqlalchemy<3",
   "dataclasses-json<0.7",
-  "dlt[deltalake]>=1.22,<1.32",
+  "dlt[deltalake]>=1.31,<1.32",
   "dlt-cratedb<0.2",
   "dlt-filesystem",
   "duckdb<2.1",