From 64b55dc2e27f2bd39f49a9e9eccd3635478d2320 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Mon, 9 Nov 2026 15:45:39 +0000 Subject: [PATCH 1/3] docs: record the replace commit route (ADR 0007, ADR 0008) Issue #72 asked for a one-day spike on how a replace, the commit behind merging on @unique_key and @full_refresh, can land on the pinned iceberg-rust, which has no overwrite action and keeps TableCommit::build crate-private. ADR 0007 records the route: build the overwrite snapshot from public iceberg-rust APIs (rewritten manifests with DELETED and EXISTING entries, untouched manifests reused as they are, a manifest of ADDED entries, a manifest list, computed summary fields and the next metadata file through TableMetadata::write_to), check the table UUID and head with TableRequirement::check, then commit with a one-row compare-and-swap on the SQL catalog's iceberg_tables row, mapping zero rows to Error::Conflict with no retry. It records the cargo tree -d result, how far the fork route got (a one-line patch that makes TableCommit::build public commits the same snapshot through Catalog::update_table), why that route is not taken, and the costs of the chosen one: SQLite only, coupling to the catalog's private table layout and record type filter, an own connection pool, working outside the Catalog trait and a redo for the Phase 1 REST catalog. ADR 0008 records the rule for incremental readers of a replaced table: when any input holds a non-append snapshot or an expired from snapshot, the scheduler pins every input of the model with from = None, reports the fallback and replaces the target as a new job; the refusal in appends_between stays as a guard that ends the job without a retry; until the plan-time check in #89 lands, the compile-time rule from #73 and #84 applies instead. The prototype that committed the route stays on a local spike branch and is not part of this change. Closes #72 Signed-off-by: Mayur Kale --- docs/adr/0007-hand-built-replace-commit.md | 143 ++++++++++++++++++ ...-incremental-readers-of-replaced-tables.md | 69 +++++++++ docs/adr/README.md | 2 + 3 files changed, 214 insertions(+) create mode 100644 docs/adr/0007-hand-built-replace-commit.md create mode 100644 docs/adr/0008-incremental-readers-of-replaced-tables.md diff --git a/docs/adr/0007-hand-built-replace-commit.md b/docs/adr/0007-hand-built-replace-commit.md new file mode 100644 index 0000000..aae6141 --- /dev/null +++ b/docs/adr/0007-hand-built-replace-commit.md @@ -0,0 +1,143 @@ +# 0007. Build the replace commit ourselves on the SQL catalog + +- **Status:** Accepted +- **Date:** 2026-11-09 + +## Context + +Merging on `@unique_key` (#35) and rebuilding with `@full_refresh` (#90) +both replace data files in a table. The pinned iceberg-rust (rev `5b82ab9`) +cannot commit such a replace. Its `Transaction` offers `fast_append` and +actions for the format version, properties, schema, location, statistics, +sort order and snapshot expiry, but no overwrite or delete action, and a +`TableCommit` can only be built inside the crate, so a snapshot built +outside it cannot go through `Catalog::update_table` either. Its +copy-on-write rewrite module plans and rewrites data files but does not +commit. Upstream main had not changed this as of 2026-11-09; the pull +request adding an overwrite action (apache/iceberg-rust#3300) was open. + +[ADR 0006](0006-ordered-job-commits.md) requires one atomic commit per +output table, carrying the job id, that fails rather than rebases when the +head has moved. Any replace route must keep that. + +Three routes were open, to be tried in this order: build the snapshot from +public spec writers and commit it with a compare-and-swap of our own; fork +iceberg-rust through a `[patch]`; or give up replacing files in 0.1.0 and +make Clean tables append-only with read-time deduplication. The spike for +this decision (#72) had one day to commit a replace on the first route that +could. + +Before any prototype code, `cargo tree -d` was run on the workspace. It +lists neither `iceberg` nor `sqlx`, so the workspace builds one copy of +each and the route can use iceberg's types and the catalog's database +driver without a second copy. The duplicates it lists are lower-level +crates such as `zstd`, `hashbrown`, `rand`, `itertools` and `syn`, none of +which the route uses directly. + +The first route committed a replace within the day. A throwaway prototype +in `crates/storage` showed that two replaces prepared on one head give one +commit and one conflict, with the loser's files referenced by no snapshot, +and that exactly one wins when the two commit concurrently; that a replace +racing an append that lands first is a conflict; that the table keeps its +UUID, snapshot history and metadata log; that a reader holding the old +metadata still reads the old files; that iceberg-rust's own `fast_append` +commits on top of the hand-made snapshot and `appends_between` then reads +the appended rows incrementally; that the refusal in `appends_between` +fires on the real overwrite; and that a full refresh with no rows still +commits a snapshot that removes every live file. + +The cheapest fork was probed alongside, to learn whether our snapshot +commits through `Catalog::update_table`, the path a REST catalog needs: a +copy of the pin with one line changed, making the `TableCommit` builder +public, applied through `[patch]`. The workspace still built one copy of +`iceberg`, and the patched crate built in under a minute. The same +snapshot committed through `update_table` with the UUID and head +requirements. A second commit prepared from the same head failed with the +catalog's `CatalogCommitConflicts`, which iceberg-rust marks as retryable, +and left the table as it was. Without the requirements, the catalog's own +swap did not stop a commit built on an old head: a stale snapshot carrying +a stale sequence number was refused by iceberg-rust's metadata builder, +and one given a fresh sequence number landed over the other writer's +commit, a lost update. On that path the requirements guard the head, as +they do for `fast_append`, which adds the same two. The fork that #72 +would have accepted, one that processes delete entries itself, was not +attempted: upstream is adding that, and the first route had already +committed. + +## Decision + +We build the replace commit ourselves, in `crates/storage`, on the SQL +catalog, and keep ADR 0006's rule: one commit per table that fails rather +than rebases. + +We build the snapshot from public iceberg-rust APIs. We rewrite each +manifest of the base snapshot that lists a removed file, marking that file +`DELETED` and carrying its other live files as `EXISTING` entries that keep +their snapshot id and sequence numbers; `DELETED` entries of earlier +snapshots are not copied, since such an entry names the snapshot that +deleted the file. We reuse every other manifest as it is, as `fast_append` +does, including one left with only `DELETED` entries. The new files go in +one manifest of `ADDED` entries. We write the manifest list and an +`overwrite` snapshot whose `added-*`, `deleted-*` and `total-*` +summary fields we compute ourselves, apply the snapshot, the `main` +reference and the table properties through `TableMetadataBuilder`, and +write the next `metadata.json` with `TableMetadata::write_to` at the path +`MetadataLocation::with_next_version` gives. `infostash.job-id` and the +checkpoint properties go in that same commit, placed as `append_if_head` +places them. + +We commit by checking the table UUID and the `main` head with +`TableRequirement::check` against freshly loaded metadata, then running a +copy of the SQL catalog's own commit statement on a `sqlx` SQLite pool of +our own, opened on the same `catalog.db`: one `UPDATE` of `iceberg_tables` +that sets `metadata_location` and `previous_metadata_location` where the +row still points at the metadata the replace was built on. Zero rows +updated is `Error::Conflict` when a re-read shows the head moved, with no +retry and no rebase; an unchanged row means the statement no longer fits +the catalog and is a storage error. The swap is what makes the commit +safe, also against a concurrent writer; the requirement check runs first +to give the clearer error and to catch a changed UUID. + +We do not take the fork. The probed variant only opens a hook, costs a +fork to host and carry across every pin bump, and saves only the catalog +half of the commit, since the snapshot is built the same way on both +routes. Append-only was not needed, so RFC 0001's merge semantics and the +README's description of Clean tables stand. + +## Consequences + +- Clean tables can be deduplicated and models fully refreshed in 0.1.0. + #88 builds `Warehouse::replace_if_head` to this record; #35, #89 and #90 + follow it. +- The route replaces whole data files in unpartitioned tables and writes + no delete files. Partition-scoped replaces (#36), compaction and the + deletion vectors `docs/architecture.md` names for updates are later work + on the same commit. +- This works on SQLite only, because `crates/storage` enables only the + SQLite driver of `sqlx` and the statement is copied from the SQL catalog. + We bypass the `Catalog` trait, depend on the catalog's private + `iceberg_tables` layout and `catalog_name` key, keep a connection pool + next to the catalog's, and carry a copy of its private + `record_type_filter` in the form a fresh `catalog.db` gets. A layout + change upstream makes the statement fail or match no row, and nothing is + corrupted; a new column the catalog keeps in step with the location would + be left stale, so every pin bump (ADR 0002) must compare the catalog's + statement with ours. +- Phase 1's REST catalog ([ROADMAP](../../ROADMAP.md)) cannot run our + statement. The commit must then move to an upstream overwrite action, or + to `Catalog::update_table` once a `TableCommit` can be built outside + iceberg-rust. On the SQL catalog the probe committed our snapshot through + that path with the same two requirements; a REST catalog's retryable + conflict must then be mapped to `Error::Conflict` with no retry. The + snapshot-building half carries over; the commit half is a redo. +- A replace that loses leaves its data files on disk, as a refused append + does, and also its manifests, manifest list and metadata file, which it + writes before the swap. Removing unreferenced files (#40) must cover all + of them. +- #88 must test each head guard on its own: the swap with the requirement + check removed, and the UUID check with the table dropped and recreated + between prepare and commit. +- Once #88 imports `sqlx`, the exemption that keeps it off the + unused-dependency check in `crates/storage/Cargo.toml` goes. +- Incremental readers of a replaced table need a rule of their own: + [ADR 0008](0008-incremental-readers-of-replaced-tables.md). diff --git a/docs/adr/0008-incremental-readers-of-replaced-tables.md b/docs/adr/0008-incremental-readers-of-replaced-tables.md new file mode 100644 index 0000000..17669d7 --- /dev/null +++ b/docs/adr/0008-incremental-readers-of-replaced-tables.md @@ -0,0 +1,69 @@ +# 0008. Incremental readers of a replaced table fall back to a full refresh + +- **Status:** Accepted +- **Date:** 2026-11-09 + +## Context + +An incremental model reads only the snapshots added to its inputs since its +last run. That works while every new snapshot is an append: the rows those +snapshots added are exactly the rows to process. A replace commit +([ADR 0007](0007-hand-built-replace-commit.md)) breaks this. An `overwrite` +snapshot removes files as well as adding them, so the rows it added do not +describe the change. Today `appends_between` in `crates/storage` refuses to +read a range that holds a non-append snapshot. Retrying cannot get past +that refusal, since running the job again meets the same snapshot, yet it +surfaces as a plain storage error, which nothing tells apart from the +transient failures a runner retries. + +A job carries the snapshot range pinned for each input, and +[ADR 0006](0006-ordered-job-commits.md) makes a rerun read the same pinned +snapshots as the first attempt, so the choice of how to read a replaced +input belongs before the job exists, not inside it. RFC 0001 lets a model +be rebuilt in full only when its author declares `@full_refresh`; this +rule adds a second cause, decided at plan time. + +## Decision + +Before creating a job, the scheduler checks the operation of every +snapshot in the range it would pin for each input. If any input holds a +snapshot that is not an append, or the input's `from` snapshot has been +expired, the scheduler pins every input of that model with `from = None`, +so the model is rebuilt from all of its inputs in full, reports the +fallback and its cause in the run output, and has the job replace its +target instead of appending to it. This is a new job with newly pinned +inputs; nothing is patched into an existing job. + +The refusal in `appends_between` stays as a guard. Reaching it ends the +job as a failure that is not retried, with a message saying the plan-time +check missed that input. Until the plan-time check (#89) lands, the +compile-time rule that #73 adds to RFC 0001 and #84 enforces applies +instead: no incremental model reads a table that declares `@unique_key` +or `@full_refresh`, because those are the tables that get replaced. The +rule matters from the first replace commit (#88, used by #35 and #90), +and #89 removes it. + +## Consequences + +- A model downstream of a deduplicated Clean table or of a fully refreshed + table is rebuilt from scratch whenever that table was replaced since the + model's last run. That is correct and costs a full read of every input, + so for these models a run no longer processes only the rows that + changed, as the README and the 0.1.0 roadmap promise; those statements + need that qualification (#94). Reading a replace as a set of changes is + a later decision. +- The fallback cascades. The rebuilt target is itself an `overwrite` + snapshot, so every model downstream of it falls back on its next run, + and everything downstream of a `@full_refresh` model is rebuilt after + every run in which that model committed. How often a merge triggers this + is #35's call: a merge that removes no file can commit an append. +- Compaction (Phase 1) also writes snapshots that are not appends while + changing no rows. Before it lands, the check needs a way to tell such a + snapshot from a replace, or every compaction rebuilds everything + downstream. +- #73 records in RFC 0001 that a model can be rebuilt in full without + declaring `@full_refresh`, and why. +- The run output names each model that fell back and the input that + caused it, so a slow run has a visible reason. +- The check reads snapshot summaries only, not data, so it costs little + at plan time. diff --git a/docs/adr/README.md b/docs/adr/README.md index 8503e45..7462486 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -53,3 +53,5 @@ What gets easier, what gets harder, and what we must now do. | [0004](0004-stage-then-atomic-commit.md) | Tasks stage files; only the job commits | Superseded by 0006 | | [0005](0005-apache-2-with-dco.md) | Apache 2.0 with a DCO, not a CLA | Accepted | | [0006](0006-ordered-job-commits.md) | Jobs commit each table once, in order, under a job id | Accepted | +| [0007](0007-hand-built-replace-commit.md) | Build the replace commit ourselves on the SQL catalog | Accepted | +| [0008](0008-incremental-readers-of-replaced-tables.md) | Incremental readers of a replaced table fall back to a full refresh | Accepted | From a008579099ac58a9d7fec72fcbce64e300bd71f0 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Mon, 9 Nov 2026 15:49:04 +0000 Subject: [PATCH 2/3] docs: say why ADR 0008 pins every input of a model A model rebuilt from one input in full and another input's increment would lose the second input's earlier rows, so the fallback pins every input with from = None, not only the replaced one. ADR 0008 now says so. Signed-off-by: Mayur Kale --- .../adr/0008-incremental-readers-of-replaced-tables.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/0008-incremental-readers-of-replaced-tables.md b/docs/adr/0008-incremental-readers-of-replaced-tables.md index 17669d7..039b23b 100644 --- a/docs/adr/0008-incremental-readers-of-replaced-tables.md +++ b/docs/adr/0008-incremental-readers-of-replaced-tables.md @@ -29,10 +29,12 @@ Before creating a job, the scheduler checks the operation of every snapshot in the range it would pin for each input. If any input holds a snapshot that is not an append, or the input's `from` snapshot has been expired, the scheduler pins every input of that model with `from = None`, -so the model is rebuilt from all of its inputs in full, reports the -fallback and its cause in the run output, and has the job replace its -target instead of appending to it. This is a new job with newly pinned -inputs; nothing is patched into an existing job. +not only the one that was replaced: a model rebuilt from one input in full +and another input's increment would lose the second input's earlier rows. +The model is rebuilt from all of its inputs, the scheduler reports the +fallback and its cause in the run output, and the job replaces its target +instead of appending to it. This is a new job with newly pinned inputs; +nothing is patched into an existing job. The refusal in `appends_between` stays as a guard. Reaching it ends the job as a failure that is not retried, with a message saying the plan-time From 57661683bcfd6f412c1871c3f1e8d536ad970447 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Mon, 9 Nov 2026 16:10:41 +0000 Subject: [PATCH 3/3] docs: leave the record type filter out of the replace commit statement The SQL catalog reads an iceberg_tables in its older layout, which has no iceberg_type column, and appends to tables in it. A statement that names that column fails there before any row changes. The filter adds nothing under the compare-and-swap: the table's primary key names one row, and that row must still hold the metadata location loaded through the catalog, which a view row never does. ADR 0007 now records the statement as the catalog's commit statement less its record type filter, drops the copied filter from the costs, and asks #88 to test a replace on a catalog without the column. The prototype's tests pass with the filter removed, including a replace on a catalog whose iceberg_tables has no iceberg_type column, after which the catalog's own append still lands. Signed-off-by: Mayur Kale --- docs/adr/0007-hand-built-replace-commit.md | 47 +++++++++++++--------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/docs/adr/0007-hand-built-replace-commit.md b/docs/adr/0007-hand-built-replace-commit.md index aae6141..e683c08 100644 --- a/docs/adr/0007-hand-built-replace-commit.md +++ b/docs/adr/0007-hand-built-replace-commit.md @@ -87,16 +87,22 @@ checkpoint properties go in that same commit, placed as `append_if_head` places them. We commit by checking the table UUID and the `main` head with -`TableRequirement::check` against freshly loaded metadata, then running a -copy of the SQL catalog's own commit statement on a `sqlx` SQLite pool of -our own, opened on the same `catalog.db`: one `UPDATE` of `iceberg_tables` -that sets `metadata_location` and `previous_metadata_location` where the -row still points at the metadata the replace was built on. Zero rows -updated is `Error::Conflict` when a re-read shows the head moved, with no -retry and no rebase; an unchanged row means the statement no longer fits -the catalog and is a storage error. The swap is what makes the commit -safe, also against a concurrent writer; the requirement check runs first -to give the clearer error and to catch a changed UUID. +`TableRequirement::check` against freshly loaded metadata, then running +the SQL catalog's own commit statement, less its record type filter, on a +`sqlx` SQLite pool of our own, opened on the same `catalog.db`: one +`UPDATE` of `iceberg_tables` that sets `metadata_location` and +`previous_metadata_location` where the row with the table's key still +points at the metadata the replace was built on. The filter adds nothing +here: the key is the table's primary key, so at most one row matches, and +that row must still hold the metadata location we loaded through the +catalog, which a view row never does. Without it the statement fits both +layouts of `iceberg_tables` the catalog reads and appends to, with and +without the `iceberg_type` column. Zero rows updated is `Error::Conflict` +when a re-read shows the head moved, with no retry and no rebase; an +unchanged row means the statement no longer fits the catalog and is a +storage error. The swap is what makes the commit safe, also against a +concurrent writer; the requirement check runs first to give the clearer +error and to catch a changed UUID. We do not take the fork. The probed variant only opens a hook, costs a fork to host and carry across every pin bump, and saves only the catalog @@ -114,15 +120,13 @@ README's description of Clean tables stand. deletion vectors `docs/architecture.md` names for updates are later work on the same commit. - This works on SQLite only, because `crates/storage` enables only the - SQLite driver of `sqlx` and the statement is copied from the SQL catalog. - We bypass the `Catalog` trait, depend on the catalog's private - `iceberg_tables` layout and `catalog_name` key, keep a connection pool - next to the catalog's, and carry a copy of its private - `record_type_filter` in the form a fresh `catalog.db` gets. A layout - change upstream makes the statement fail or match no row, and nothing is - corrupted; a new column the catalog keeps in step with the location would - be left stale, so every pin bump (ADR 0002) must compare the catalog's - statement with ours. + SQLite driver of `sqlx` and the statement is the SQL catalog's. We bypass + the `Catalog` trait, depend on the catalog's private `iceberg_tables` + layout and `catalog_name` key, and keep a connection pool next to the + catalog's. A layout change upstream makes the statement fail or match no + row, and nothing is corrupted; a new column the catalog keeps in step + with the location would be left stale, so every pin bump (ADR 0002) must + compare the catalog's commit statement with ours. - Phase 1's REST catalog ([ROADMAP](../../ROADMAP.md)) cannot run our statement. The commit must then move to an upstream overwrite action, or to `Catalog::update_table` once a `TableCommit` can be built outside @@ -136,7 +140,10 @@ README's description of Clean tables stand. of them. - #88 must test each head guard on its own: the swap with the requirement check removed, and the UUID check with the table dropped and recreated - between prepare and commit. + between prepare and commit. It must also test a replace on a + `catalog.db` whose `iceberg_tables` has no `iceberg_type` column, the + older layout the catalog reads and appends to but cannot create tables + on: create the table on a fresh catalog, drop the column, then reopen. - Once #88 imports `sqlx`, the exemption that keeps it off the unused-dependency check in `crates/storage/Cargo.toml` goes. - Incremental readers of a replaced table need a rule of their own: