diff --git a/docs/adr/0007-hand-built-replace-commit.md b/docs/adr/0007-hand-built-replace-commit.md new file mode 100644 index 0000000..e683c08 --- /dev/null +++ b/docs/adr/0007-hand-built-replace-commit.md @@ -0,0 +1,150 @@ +# 0007. Build the replace commit ourselves on the SQL catalog + +- **Status:** Accepted +- **Date:** 2026-11-09 + +## Context + +Merging on `@unique_key` (#35) and rebuilding with `@full_refresh` (#90) +both replace data files in a table. The pinned iceberg-rust (rev `5b82ab9`) +cannot commit such a replace. Its `Transaction` offers `fast_append` and +actions for the format version, properties, schema, location, statistics, +sort order and snapshot expiry, but no overwrite or delete action, and a +`TableCommit` can only be built inside the crate, so a snapshot built +outside it cannot go through `Catalog::update_table` either. Its +copy-on-write rewrite module plans and rewrites data files but does not +commit. Upstream main had not changed this as of 2026-11-09; the pull +request adding an overwrite action (apache/iceberg-rust#3300) was open. + +[ADR 0006](0006-ordered-job-commits.md) requires one atomic commit per +output table, carrying the job id, that fails rather than rebases when the +head has moved. Any replace route must keep that. + +Three routes were open, to be tried in this order: build the snapshot from +public spec writers and commit it with a compare-and-swap of our own; fork +iceberg-rust through a `[patch]`; or give up replacing files in 0.1.0 and +make Clean tables append-only with read-time deduplication. The spike for +this decision (#72) had one day to commit a replace on the first route that +could. + +Before any prototype code, `cargo tree -d` was run on the workspace. It +lists neither `iceberg` nor `sqlx`, so the workspace builds one copy of +each and the route can use iceberg's types and the catalog's database +driver without a second copy. The duplicates it lists are lower-level +crates such as `zstd`, `hashbrown`, `rand`, `itertools` and `syn`, none of +which the route uses directly. + +The first route committed a replace within the day. A throwaway prototype +in `crates/storage` showed that two replaces prepared on one head give one +commit and one conflict, with the loser's files referenced by no snapshot, +and that exactly one wins when the two commit concurrently; that a replace +racing an append that lands first is a conflict; that the table keeps its +UUID, snapshot history and metadata log; that a reader holding the old +metadata still reads the old files; that iceberg-rust's own `fast_append` +commits on top of the hand-made snapshot and `appends_between` then reads +the appended rows incrementally; that the refusal in `appends_between` +fires on the real overwrite; and that a full refresh with no rows still +commits a snapshot that removes every live file. + +The cheapest fork was probed alongside, to learn whether our snapshot +commits through `Catalog::update_table`, the path a REST catalog needs: a +copy of the pin with one line changed, making the `TableCommit` builder +public, applied through `[patch]`. The workspace still built one copy of +`iceberg`, and the patched crate built in under a minute. The same +snapshot committed through `update_table` with the UUID and head +requirements. A second commit prepared from the same head failed with the +catalog's `CatalogCommitConflicts`, which iceberg-rust marks as retryable, +and left the table as it was. Without the requirements, the catalog's own +swap did not stop a commit built on an old head: a stale snapshot carrying +a stale sequence number was refused by iceberg-rust's metadata builder, +and one given a fresh sequence number landed over the other writer's +commit, a lost update. On that path the requirements guard the head, as +they do for `fast_append`, which adds the same two. The fork that #72 +would have accepted, one that processes delete entries itself, was not +attempted: upstream is adding that, and the first route had already +committed. + +## Decision + +We build the replace commit ourselves, in `crates/storage`, on the SQL +catalog, and keep ADR 0006's rule: one commit per table that fails rather +than rebases. + +We build the snapshot from public iceberg-rust APIs. We rewrite each +manifest of the base snapshot that lists a removed file, marking that file +`DELETED` and carrying its other live files as `EXISTING` entries that keep +their snapshot id and sequence numbers; `DELETED` entries of earlier +snapshots are not copied, since such an entry names the snapshot that +deleted the file. We reuse every other manifest as it is, as `fast_append` +does, including one left with only `DELETED` entries. The new files go in +one manifest of `ADDED` entries. We write the manifest list and an +`overwrite` snapshot whose `added-*`, `deleted-*` and `total-*` +summary fields we compute ourselves, apply the snapshot, the `main` +reference and the table properties through `TableMetadataBuilder`, and +write the next `metadata.json` with `TableMetadata::write_to` at the path +`MetadataLocation::with_next_version` gives. `infostash.job-id` and the +checkpoint properties go in that same commit, placed as `append_if_head` +places them. + +We commit by checking the table UUID and the `main` head with +`TableRequirement::check` against freshly loaded metadata, then running +the SQL catalog's own commit statement, less its record type filter, on a +`sqlx` SQLite pool of our own, opened on the same `catalog.db`: one +`UPDATE` of `iceberg_tables` that sets `metadata_location` and +`previous_metadata_location` where the row with the table's key still +points at the metadata the replace was built on. The filter adds nothing +here: the key is the table's primary key, so at most one row matches, and +that row must still hold the metadata location we loaded through the +catalog, which a view row never does. Without it the statement fits both +layouts of `iceberg_tables` the catalog reads and appends to, with and +without the `iceberg_type` column. Zero rows updated is `Error::Conflict` +when a re-read shows the head moved, with no retry and no rebase; an +unchanged row means the statement no longer fits the catalog and is a +storage error. The swap is what makes the commit safe, also against a +concurrent writer; the requirement check runs first to give the clearer +error and to catch a changed UUID. + +We do not take the fork. The probed variant only opens a hook, costs a +fork to host and carry across every pin bump, and saves only the catalog +half of the commit, since the snapshot is built the same way on both +routes. Append-only was not needed, so RFC 0001's merge semantics and the +README's description of Clean tables stand. + +## Consequences + +- Clean tables can be deduplicated and models fully refreshed in 0.1.0. + #88 builds `Warehouse::replace_if_head` to this record; #35, #89 and #90 + follow it. +- The route replaces whole data files in unpartitioned tables and writes + no delete files. Partition-scoped replaces (#36), compaction and the + deletion vectors `docs/architecture.md` names for updates are later work + on the same commit. +- This works on SQLite only, because `crates/storage` enables only the + SQLite driver of `sqlx` and the statement is the SQL catalog's. We bypass + the `Catalog` trait, depend on the catalog's private `iceberg_tables` + layout and `catalog_name` key, and keep a connection pool next to the + catalog's. A layout change upstream makes the statement fail or match no + row, and nothing is corrupted; a new column the catalog keeps in step + with the location would be left stale, so every pin bump (ADR 0002) must + compare the catalog's commit statement with ours. +- Phase 1's REST catalog ([ROADMAP](../../ROADMAP.md)) cannot run our + statement. The commit must then move to an upstream overwrite action, or + to `Catalog::update_table` once a `TableCommit` can be built outside + iceberg-rust. On the SQL catalog the probe committed our snapshot through + that path with the same two requirements; a REST catalog's retryable + conflict must then be mapped to `Error::Conflict` with no retry. The + snapshot-building half carries over; the commit half is a redo. +- A replace that loses leaves its data files on disk, as a refused append + does, and also its manifests, manifest list and metadata file, which it + writes before the swap. Removing unreferenced files (#40) must cover all + of them. +- #88 must test each head guard on its own: the swap with the requirement + check removed, and the UUID check with the table dropped and recreated + between prepare and commit. It must also test a replace on a + `catalog.db` whose `iceberg_tables` has no `iceberg_type` column, the + older layout the catalog reads and appends to but cannot create tables + on: create the table on a fresh catalog, drop the column, then reopen. +- Once #88 imports `sqlx`, the exemption that keeps it off the + unused-dependency check in `crates/storage/Cargo.toml` goes. +- Incremental readers of a replaced table need a rule of their own: + [ADR 0008](0008-incremental-readers-of-replaced-tables.md). diff --git a/docs/adr/0008-incremental-readers-of-replaced-tables.md b/docs/adr/0008-incremental-readers-of-replaced-tables.md new file mode 100644 index 0000000..039b23b --- /dev/null +++ b/docs/adr/0008-incremental-readers-of-replaced-tables.md @@ -0,0 +1,71 @@ +# 0008. Incremental readers of a replaced table fall back to a full refresh + +- **Status:** Accepted +- **Date:** 2026-11-09 + +## Context + +An incremental model reads only the snapshots added to its inputs since its +last run. That works while every new snapshot is an append: the rows those +snapshots added are exactly the rows to process. A replace commit +([ADR 0007](0007-hand-built-replace-commit.md)) breaks this. An `overwrite` +snapshot removes files as well as adding them, so the rows it added do not +describe the change. Today `appends_between` in `crates/storage` refuses to +read a range that holds a non-append snapshot. Retrying cannot get past +that refusal, since running the job again meets the same snapshot, yet it +surfaces as a plain storage error, which nothing tells apart from the +transient failures a runner retries. + +A job carries the snapshot range pinned for each input, and +[ADR 0006](0006-ordered-job-commits.md) makes a rerun read the same pinned +snapshots as the first attempt, so the choice of how to read a replaced +input belongs before the job exists, not inside it. RFC 0001 lets a model +be rebuilt in full only when its author declares `@full_refresh`; this +rule adds a second cause, decided at plan time. + +## Decision + +Before creating a job, the scheduler checks the operation of every +snapshot in the range it would pin for each input. If any input holds a +snapshot that is not an append, or the input's `from` snapshot has been +expired, the scheduler pins every input of that model with `from = None`, +not only the one that was replaced: a model rebuilt from one input in full +and another input's increment would lose the second input's earlier rows. +The model is rebuilt from all of its inputs, the scheduler reports the +fallback and its cause in the run output, and the job replaces its target +instead of appending to it. This is a new job with newly pinned inputs; +nothing is patched into an existing job. + +The refusal in `appends_between` stays as a guard. Reaching it ends the +job as a failure that is not retried, with a message saying the plan-time +check missed that input. Until the plan-time check (#89) lands, the +compile-time rule that #73 adds to RFC 0001 and #84 enforces applies +instead: no incremental model reads a table that declares `@unique_key` +or `@full_refresh`, because those are the tables that get replaced. The +rule matters from the first replace commit (#88, used by #35 and #90), +and #89 removes it. + +## Consequences + +- A model downstream of a deduplicated Clean table or of a fully refreshed + table is rebuilt from scratch whenever that table was replaced since the + model's last run. That is correct and costs a full read of every input, + so for these models a run no longer processes only the rows that + changed, as the README and the 0.1.0 roadmap promise; those statements + need that qualification (#94). Reading a replace as a set of changes is + a later decision. +- The fallback cascades. The rebuilt target is itself an `overwrite` + snapshot, so every model downstream of it falls back on its next run, + and everything downstream of a `@full_refresh` model is rebuilt after + every run in which that model committed. How often a merge triggers this + is #35's call: a merge that removes no file can commit an append. +- Compaction (Phase 1) also writes snapshots that are not appends while + changing no rows. Before it lands, the check needs a way to tell such a + snapshot from a replace, or every compaction rebuilds everything + downstream. +- #73 records in RFC 0001 that a model can be rebuilt in full without + declaring `@full_refresh`, and why. +- The run output names each model that fell back and the input that + caused it, so a slow run has a visible reason. +- The check reads snapshot summaries only, not data, so it costs little + at plan time. diff --git a/docs/adr/README.md b/docs/adr/README.md index 8503e45..7462486 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -53,3 +53,5 @@ What gets easier, what gets harder, and what we must now do. | [0004](0004-stage-then-atomic-commit.md) | Tasks stage files; only the job commits | Superseded by 0006 | | [0005](0005-apache-2-with-dco.md) | Apache 2.0 with a DCO, not a CLA | Accepted | | [0006](0006-ordered-job-commits.md) | Jobs commit each table once, in order, under a job id | Accepted | +| [0007](0007-hand-built-replace-commit.md) | Build the replace commit ourselves on the SQL catalog | Accepted | +| [0008](0008-incremental-readers-of-replaced-tables.md) | Incremental readers of a replaced table fall back to a full refresh | Accepted |