From 64b55dc2e27f2bd39f49a9e9eccd3635478d2320 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Mon, 9 Nov 2026 15:45:39 +0000 Subject: [PATCH 1/5] docs: record the replace commit route (ADR 0007, ADR 0008) Issue #72 asked for a one-day spike on how a replace, the commit behind merging on @unique_key and @full_refresh, can land on the pinned iceberg-rust, which has no overwrite action and keeps TableCommit::build crate-private. ADR 0007 records the route: build the overwrite snapshot from public iceberg-rust APIs (rewritten manifests with DELETED and EXISTING entries, untouched manifests reused as they are, a manifest of ADDED entries, a manifest list, computed summary fields and the next metadata file through TableMetadata::write_to), check the table UUID and head with TableRequirement::check, then commit with a one-row compare-and-swap on the SQL catalog's iceberg_tables row, mapping zero rows to Error::Conflict with no retry. It records the cargo tree -d result, how far the fork route got (a one-line patch that makes TableCommit::build public commits the same snapshot through Catalog::update_table), why that route is not taken, and the costs of the chosen one: SQLite only, coupling to the catalog's private table layout and record type filter, an own connection pool, working outside the Catalog trait and a redo for the Phase 1 REST catalog. ADR 0008 records the rule for incremental readers of a replaced table: when any input holds a non-append snapshot or an expired from snapshot, the scheduler pins every input of the model with from = None, reports the fallback and replaces the target as a new job; the refusal in appends_between stays as a guard that ends the job without a retry; until the plan-time check in #89 lands, the compile-time rule from #73 and #84 applies instead. The prototype that committed the route stays on a local spike branch and is not part of this change. Closes #72 Signed-off-by: Mayur Kale --- docs/adr/0007-hand-built-replace-commit.md | 143 ++++++++++++++++++ ...-incremental-readers-of-replaced-tables.md | 69 +++++++++ docs/adr/README.md | 2 + 3 files changed, 214 insertions(+) create mode 100644 docs/adr/0007-hand-built-replace-commit.md create mode 100644 docs/adr/0008-incremental-readers-of-replaced-tables.md diff --git a/docs/adr/0007-hand-built-replace-commit.md b/docs/adr/0007-hand-built-replace-commit.md new file mode 100644 index 0000000..aae6141 --- /dev/null +++ b/docs/adr/0007-hand-built-replace-commit.md @@ -0,0 +1,143 @@ +# 0007. Build the replace commit ourselves on the SQL catalog + +- **Status:** Accepted +- **Date:** 2026-11-09 + +## Context + +Merging on `@unique_key` (#35) and rebuilding with `@full_refresh` (#90) +both replace data files in a table. The pinned iceberg-rust (rev `5b82ab9`) +cannot commit such a replace. Its `Transaction` offers `fast_append` and +actions for the format version, properties, schema, location, statistics, +sort order and snapshot expiry, but no overwrite or delete action, and a +`TableCommit` can only be built inside the crate, so a snapshot built +outside it cannot go through `Catalog::update_table` either. Its +copy-on-write rewrite module plans and rewrites data files but does not +commit. Upstream main had not changed this as of 2026-11-09; the pull +request adding an overwrite action (apache/iceberg-rust#3300) was open. + +[ADR 0006](0006-ordered-job-commits.md) requires one atomic commit per +output table, carrying the job id, that fails rather than rebases when the +head has moved. Any replace route must keep that. + +Three routes were open, to be tried in this order: build the snapshot from +public spec writers and commit it with a compare-and-swap of our own; fork +iceberg-rust through a `[patch]`; or give up replacing files in 0.1.0 and +make Clean tables append-only with read-time deduplication. The spike for +this decision (#72) had one day to commit a replace on the first route that +could. + +Before any prototype code, `cargo tree -d` was run on the workspace. It +lists neither `iceberg` nor `sqlx`, so the workspace builds one copy of +each and the route can use iceberg's types and the catalog's database +driver without a second copy. The duplicates it lists are lower-level +crates such as `zstd`, `hashbrown`, `rand`, `itertools` and `syn`, none of +which the route uses directly. + +The first route committed a replace within the day. A throwaway prototype +in `crates/storage` showed that two replaces prepared on one head give one +commit and one conflict, with the loser's files referenced by no snapshot, +and that exactly one wins when the two commit concurrently; that a replace +racing an append that lands first is a conflict; that the table keeps its +UUID, snapshot history and metadata log; that a reader holding the old +metadata still reads the old files; that iceberg-rust's own `fast_append` +commits on top of the hand-made snapshot and `appends_between` then reads +the appended rows incrementally; that the refusal in `appends_between` +fires on the real overwrite; and that a full refresh with no rows still +commits a snapshot that removes every live file. + +The cheapest fork was probed alongside, to learn whether our snapshot +commits through `Catalog::update_table`, the path a REST catalog needs: a +copy of the pin with one line changed, making the `TableCommit` builder +public, applied through `[patch]`. The workspace still built one copy of +`iceberg`, and the patched crate built in under a minute. The same +snapshot committed through `update_table` with the UUID and head +requirements. A second commit prepared from the same head failed with the +catalog's `CatalogCommitConflicts`, which iceberg-rust marks as retryable, +and left the table as it was. Without the requirements, the catalog's own +swap did not stop a commit built on an old head: a stale snapshot carrying +a stale sequence number was refused by iceberg-rust's metadata builder, +and one given a fresh sequence number landed over the other writer's +commit, a lost update. On that path the requirements guard the head, as +they do for `fast_append`, which adds the same two. The fork that #72 +would have accepted, one that processes delete entries itself, was not +attempted: upstream is adding that, and the first route had already +committed. + +## Decision + +We build the replace commit ourselves, in `crates/storage`, on the SQL +catalog, and keep ADR 0006's rule: one commit per table that fails rather +than rebases. + +We build the snapshot from public iceberg-rust APIs. We rewrite each +manifest of the base snapshot that lists a removed file, marking that file +`DELETED` and carrying its other live files as `EXISTING` entries that keep +their snapshot id and sequence numbers; `DELETED` entries of earlier +snapshots are not copied, since such an entry names the snapshot that +deleted the file. We reuse every other manifest as it is, as `fast_append` +does, including one left with only `DELETED` entries. The new files go in +one manifest of `ADDED` entries. We write the manifest list and an +`overwrite` snapshot whose `added-*`, `deleted-*` and `total-*` +summary fields we compute ourselves, apply the snapshot, the `main` +reference and the table properties through `TableMetadataBuilder`, and +write the next `metadata.json` with `TableMetadata::write_to` at the path +`MetadataLocation::with_next_version` gives. `infostash.job-id` and the +checkpoint properties go in that same commit, placed as `append_if_head` +places them. + +We commit by checking the table UUID and the `main` head with +`TableRequirement::check` against freshly loaded metadata, then running a +copy of the SQL catalog's own commit statement on a `sqlx` SQLite pool of +our own, opened on the same `catalog.db`: one `UPDATE` of `iceberg_tables` +that sets `metadata_location` and `previous_metadata_location` where the +row still points at the metadata the replace was built on. Zero rows +updated is `Error::Conflict` when a re-read shows the head moved, with no +retry and no rebase; an unchanged row means the statement no longer fits +the catalog and is a storage error. The swap is what makes the commit +safe, also against a concurrent writer; the requirement check runs first +to give the clearer error and to catch a changed UUID. + +We do not take the fork. The probed variant only opens a hook, costs a +fork to host and carry across every pin bump, and saves only the catalog +half of the commit, since the snapshot is built the same way on both +routes. Append-only was not needed, so RFC 0001's merge semantics and the +README's description of Clean tables stand. + +## Consequences + +- Clean tables can be deduplicated and models fully refreshed in 0.1.0. + #88 builds `Warehouse::replace_if_head` to this record; #35, #89 and #90 + follow it. +- The route replaces whole data files in unpartitioned tables and writes + no delete files. Partition-scoped replaces (#36), compaction and the + deletion vectors `docs/architecture.md` names for updates are later work + on the same commit. +- This works on SQLite only, because `crates/storage` enables only the + SQLite driver of `sqlx` and the statement is copied from the SQL catalog. + We bypass the `Catalog` trait, depend on the catalog's private + `iceberg_tables` layout and `catalog_name` key, keep a connection pool + next to the catalog's, and carry a copy of its private + `record_type_filter` in the form a fresh `catalog.db` gets. A layout + change upstream makes the statement fail or match no row, and nothing is + corrupted; a new column the catalog keeps in step with the location would + be left stale, so every pin bump (ADR 0002) must compare the catalog's + statement with ours. +- Phase 1's REST catalog ([ROADMAP](../../ROADMAP.md)) cannot run our + statement. The commit must then move to an upstream overwrite action, or + to `Catalog::update_table` once a `TableCommit` can be built outside + iceberg-rust. On the SQL catalog the probe committed our snapshot through + that path with the same two requirements; a REST catalog's retryable + conflict must then be mapped to `Error::Conflict` with no retry. The + snapshot-building half carries over; the commit half is a redo. +- A replace that loses leaves its data files on disk, as a refused append + does, and also its manifests, manifest list and metadata file, which it + writes before the swap. Removing unreferenced files (#40) must cover all + of them. +- #88 must test each head guard on its own: the swap with the requirement + check removed, and the UUID check with the table dropped and recreated + between prepare and commit. +- Once #88 imports `sqlx`, the exemption that keeps it off the + unused-dependency check in `crates/storage/Cargo.toml` goes. +- Incremental readers of a replaced table need a rule of their own: + [ADR 0008](0008-incremental-readers-of-replaced-tables.md). diff --git a/docs/adr/0008-incremental-readers-of-replaced-tables.md b/docs/adr/0008-incremental-readers-of-replaced-tables.md new file mode 100644 index 0000000..17669d7 --- /dev/null +++ b/docs/adr/0008-incremental-readers-of-replaced-tables.md @@ -0,0 +1,69 @@ +# 0008. Incremental readers of a replaced table fall back to a full refresh + +- **Status:** Accepted +- **Date:** 2026-11-09 + +## Context + +An incremental model reads only the snapshots added to its inputs since its +last run. That works while every new snapshot is an append: the rows those +snapshots added are exactly the rows to process. A replace commit +([ADR 0007](0007-hand-built-replace-commit.md)) breaks this. An `overwrite` +snapshot removes files as well as adding them, so the rows it added do not +describe the change. Today `appends_between` in `crates/storage` refuses to +read a range that holds a non-append snapshot. Retrying cannot get past +that refusal, since running the job again meets the same snapshot, yet it +surfaces as a plain storage error, which nothing tells apart from the +transient failures a runner retries. + +A job carries the snapshot range pinned for each input, and +[ADR 0006](0006-ordered-job-commits.md) makes a rerun read the same pinned +snapshots as the first attempt, so the choice of how to read a replaced +input belongs before the job exists, not inside it. RFC 0001 lets a model +be rebuilt in full only when its author declares `@full_refresh`; this +rule adds a second cause, decided at plan time. + +## Decision + +Before creating a job, the scheduler checks the operation of every +snapshot in the range it would pin for each input. If any input holds a +snapshot that is not an append, or the input's `from` snapshot has been +expired, the scheduler pins every input of that model with `from = None`, +so the model is rebuilt from all of its inputs in full, reports the +fallback and its cause in the run output, and has the job replace its +target instead of appending to it. This is a new job with newly pinned +inputs; nothing is patched into an existing job. + +The refusal in `appends_between` stays as a guard. Reaching it ends the +job as a failure that is not retried, with a message saying the plan-time +check missed that input. Until the plan-time check (#89) lands, the +compile-time rule that #73 adds to RFC 0001 and #84 enforces applies +instead: no incremental model reads a table that declares `@unique_key` +or `@full_refresh`, because those are the tables that get replaced. The +rule matters from the first replace commit (#88, used by #35 and #90), +and #89 removes it. + +## Consequences + +- A model downstream of a deduplicated Clean table or of a fully refreshed + table is rebuilt from scratch whenever that table was replaced since the + model's last run. That is correct and costs a full read of every input, + so for these models a run no longer processes only the rows that + changed, as the README and the 0.1.0 roadmap promise; those statements + need that qualification (#94). Reading a replace as a set of changes is + a later decision. +- The fallback cascades. The rebuilt target is itself an `overwrite` + snapshot, so every model downstream of it falls back on its next run, + and everything downstream of a `@full_refresh` model is rebuilt after + every run in which that model committed. How often a merge triggers this + is #35's call: a merge that removes no file can commit an append. +- Compaction (Phase 1) also writes snapshots that are not appends while + changing no rows. Before it lands, the check needs a way to tell such a + snapshot from a replace, or every compaction rebuilds everything + downstream. +- #73 records in RFC 0001 that a model can be rebuilt in full without + declaring `@full_refresh`, and why. +- The run output names each model that fell back and the input that + caused it, so a slow run has a visible reason. +- The check reads snapshot summaries only, not data, so it costs little + at plan time. diff --git a/docs/adr/README.md b/docs/adr/README.md index 8503e45..7462486 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -53,3 +53,5 @@ What gets easier, what gets harder, and what we must now do. | [0004](0004-stage-then-atomic-commit.md) | Tasks stage files; only the job commits | Superseded by 0006 | | [0005](0005-apache-2-with-dco.md) | Apache 2.0 with a DCO, not a CLA | Accepted | | [0006](0006-ordered-job-commits.md) | Jobs commit each table once, in order, under a job id | Accepted | +| [0007](0007-hand-built-replace-commit.md) | Build the replace commit ourselves on the SQL catalog | Accepted | +| [0008](0008-incremental-readers-of-replaced-tables.md) | Incremental readers of a replaced table fall back to a full refresh | Accepted | From a008579099ac58a9d7fec72fcbce64e300bd71f0 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Mon, 9 Nov 2026 15:49:04 +0000 Subject: [PATCH 2/5] docs: say why ADR 0008 pins every input of a model A model rebuilt from one input in full and another input's increment would lose the second input's earlier rows, so the fallback pins every input with from = None, not only the replaced one. ADR 0008 now says so. Signed-off-by: Mayur Kale --- .../adr/0008-incremental-readers-of-replaced-tables.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/0008-incremental-readers-of-replaced-tables.md b/docs/adr/0008-incremental-readers-of-replaced-tables.md index 17669d7..039b23b 100644 --- a/docs/adr/0008-incremental-readers-of-replaced-tables.md +++ b/docs/adr/0008-incremental-readers-of-replaced-tables.md @@ -29,10 +29,12 @@ Before creating a job, the scheduler checks the operation of every snapshot in the range it would pin for each input. If any input holds a snapshot that is not an append, or the input's `from` snapshot has been expired, the scheduler pins every input of that model with `from = None`, -so the model is rebuilt from all of its inputs in full, reports the -fallback and its cause in the run output, and has the job replace its -target instead of appending to it. This is a new job with newly pinned -inputs; nothing is patched into an existing job. +not only the one that was replaced: a model rebuilt from one input in full +and another input's increment would lose the second input's earlier rows. +The model is rebuilt from all of its inputs, the scheduler reports the +fallback and its cause in the run output, and the job replaces its target +instead of appending to it. This is a new job with newly pinned inputs; +nothing is patched into an existing job. The refusal in `appends_between` stays as a guard. Reaching it ends the job as a failure that is not retried, with a message saying the plan-time From 57661683bcfd6f412c1871c3f1e8d536ad970447 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Mon, 9 Nov 2026 16:10:41 +0000 Subject: [PATCH 3/5] docs: leave the record type filter out of the replace commit statement The SQL catalog reads an iceberg_tables in its older layout, which has no iceberg_type column, and appends to tables in it. A statement that names that column fails there before any row changes. The filter adds nothing under the compare-and-swap: the table's primary key names one row, and that row must still hold the metadata location loaded through the catalog, which a view row never does. ADR 0007 now records the statement as the catalog's commit statement less its record type filter, drops the copied filter from the costs, and asks #88 to test a replace on a catalog without the column. The prototype's tests pass with the filter removed, including a replace on a catalog whose iceberg_tables has no iceberg_type column, after which the catalog's own append still lands. Signed-off-by: Mayur Kale --- docs/adr/0007-hand-built-replace-commit.md | 47 +++++++++++++--------- 1 file changed, 27 insertions(+), 20 deletions(-) diff --git a/docs/adr/0007-hand-built-replace-commit.md b/docs/adr/0007-hand-built-replace-commit.md index aae6141..e683c08 100644 --- a/docs/adr/0007-hand-built-replace-commit.md +++ b/docs/adr/0007-hand-built-replace-commit.md @@ -87,16 +87,22 @@ checkpoint properties go in that same commit, placed as `append_if_head` places them. We commit by checking the table UUID and the `main` head with -`TableRequirement::check` against freshly loaded metadata, then running a -copy of the SQL catalog's own commit statement on a `sqlx` SQLite pool of -our own, opened on the same `catalog.db`: one `UPDATE` of `iceberg_tables` -that sets `metadata_location` and `previous_metadata_location` where the -row still points at the metadata the replace was built on. Zero rows -updated is `Error::Conflict` when a re-read shows the head moved, with no -retry and no rebase; an unchanged row means the statement no longer fits -the catalog and is a storage error. The swap is what makes the commit -safe, also against a concurrent writer; the requirement check runs first -to give the clearer error and to catch a changed UUID. +`TableRequirement::check` against freshly loaded metadata, then running +the SQL catalog's own commit statement, less its record type filter, on a +`sqlx` SQLite pool of our own, opened on the same `catalog.db`: one +`UPDATE` of `iceberg_tables` that sets `metadata_location` and +`previous_metadata_location` where the row with the table's key still +points at the metadata the replace was built on. The filter adds nothing +here: the key is the table's primary key, so at most one row matches, and +that row must still hold the metadata location we loaded through the +catalog, which a view row never does. Without it the statement fits both +layouts of `iceberg_tables` the catalog reads and appends to, with and +without the `iceberg_type` column. Zero rows updated is `Error::Conflict` +when a re-read shows the head moved, with no retry and no rebase; an +unchanged row means the statement no longer fits the catalog and is a +storage error. The swap is what makes the commit safe, also against a +concurrent writer; the requirement check runs first to give the clearer +error and to catch a changed UUID. We do not take the fork. The probed variant only opens a hook, costs a fork to host and carry across every pin bump, and saves only the catalog @@ -114,15 +120,13 @@ README's description of Clean tables stand. deletion vectors `docs/architecture.md` names for updates are later work on the same commit. - This works on SQLite only, because `crates/storage` enables only the - SQLite driver of `sqlx` and the statement is copied from the SQL catalog. - We bypass the `Catalog` trait, depend on the catalog's private - `iceberg_tables` layout and `catalog_name` key, keep a connection pool - next to the catalog's, and carry a copy of its private - `record_type_filter` in the form a fresh `catalog.db` gets. A layout - change upstream makes the statement fail or match no row, and nothing is - corrupted; a new column the catalog keeps in step with the location would - be left stale, so every pin bump (ADR 0002) must compare the catalog's - statement with ours. + SQLite driver of `sqlx` and the statement is the SQL catalog's. We bypass + the `Catalog` trait, depend on the catalog's private `iceberg_tables` + layout and `catalog_name` key, and keep a connection pool next to the + catalog's. A layout change upstream makes the statement fail or match no + row, and nothing is corrupted; a new column the catalog keeps in step + with the location would be left stale, so every pin bump (ADR 0002) must + compare the catalog's commit statement with ours. - Phase 1's REST catalog ([ROADMAP](../../ROADMAP.md)) cannot run our statement. The commit must then move to an upstream overwrite action, or to `Catalog::update_table` once a `TableCommit` can be built outside @@ -136,7 +140,10 @@ README's description of Clean tables stand. of them. - #88 must test each head guard on its own: the swap with the requirement check removed, and the UUID check with the table dropped and recreated - between prepare and commit. + between prepare and commit. It must also test a replace on a + `catalog.db` whose `iceberg_tables` has no `iceberg_type` column, the + older layout the catalog reads and appends to but cannot create tables + on: create the table on a fresh catalog, drop the column, then reopen. - Once #88 imports `sqlx`, the exemption that keeps it off the unused-dependency check in `crates/storage/Cargo.toml` goes. - Incremental readers of a replaced table need a rule of their own: From ce64bebc39a8dc98221bf1f4931b9f1f8b8b3ee5 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Tue, 10 Nov 2026 10:56:07 +0000 Subject: [PATCH 4/5] docs: add ADR 0009 amending 0006 on position, reruns and expiry ADR 0006 mirrors a connector's position in every snapshot summary, so the metadata.json of a CSV connector table grows with the square of the number of runs and the metadata directory with the cube. ADR 0009 makes the infostash.position table property the one authoritative position, keeps only the job id in the summary, and settles three rules the implementation tickets wait on: a run refuses to start with a Diverged error when the table property's job id and the head summary's job id disagree; a job that failed after any commit is rerun under the same id before any new job for that target, with the abandon exception from RFC 0002; and snapshot expiry never goes past the oldest consumer checkpoint or a snapshot committed by a job whose record still exists. ADR 0006 changes only on its status line. The README index lists 0009, shows 0006 as amended, and the Status format line gains the amended form. Closes #75 Signed-off-by: Mayur Kale --- docs/adr/0006-ordered-job-commits.md | 2 +- docs/adr/0009-positions-reruns-and-expiry.md | 125 +++++++++++++++++++ docs/adr/README.md | 10 +- 3 files changed, 133 insertions(+), 4 deletions(-) create mode 100644 docs/adr/0009-positions-reruns-and-expiry.md diff --git a/docs/adr/0006-ordered-job-commits.md b/docs/adr/0006-ordered-job-commits.md index 221bd3e..1a3f5a6 100644 --- a/docs/adr/0006-ordered-job-commits.md +++ b/docs/adr/0006-ordered-job-commits.md @@ -1,6 +1,6 @@ # 0006. Jobs commit each table once, in order, under a job id -- **Status:** Accepted +- **Status:** Accepted, amended by [0009](0009-positions-reruns-and-expiry.md) - **Date:** 2026-10-30 - **Supersedes:** [0004](0004-stage-then-atomic-commit.md) diff --git a/docs/adr/0009-positions-reruns-and-expiry.md b/docs/adr/0009-positions-reruns-and-expiry.md new file mode 100644 index 0000000..2119369 --- /dev/null +++ b/docs/adr/0009-positions-reruns-and-expiry.md @@ -0,0 +1,125 @@ +# 0009. Table positions, same-id reruns and expiry at checkpoints + +- **Status:** Accepted +- **Date:** 2026-11-10 +- **Amends:** [0006](0006-ordered-job-commits.md) + +## Context + +ADR 0006 stores a connector's read position, and a model's last processed +snapshot per input, as a table property mirrored in the snapshot summary. +`Warehouse::append_if_head` writes one properties map into both places, so +every snapshot of a connector table keeps its own copy of the position. For +the CSV connector the position is the list of every file loaded so far, and +Iceberg keeps every snapshot's summary in every later `metadata.json`. The +file written by run n therefore holds n copies of a list that is n entries +long, and the metadata directory keeps every file ever written. With paths +of 65 characters and one new file per run: + +| Hourly one-file runs | Latest `metadata.json` | Metadata on disk | +|---|---|---| +| 100 | 0.3 MB | 11 MB | +| 1000 | 35 MB | 12 GB | + +Only the table property is read: `connectors::prepare` takes the position +from it, and no production code reads the summary copy. Four alternatives +were ruled out. Compressing `metadata.json` does not help a reader that +still inflates the whole file. A per-connector delta position cannot be +defined, because the position is opaque to everything except the connector. +Reading the head snapshot's summary as the position keeps the growth. Tags, +or `gc.enabled = false`, are not a retention rule: a tag keeps one snapshot +and not the ancestors a delta read walks back through, and with +`gc.enabled = false` nothing can ever be expired. + +Two rules that the connector, runner and maintenance work need are also +missing. ADR 0006 makes a rerun skip the tables that already carry its job +id, but does not say that a job which failed after one of its commits is +rerun at all, or before what. No ADR covers snapshot expiry, although any +engine that expires under a table's own settings removes every snapshot +older than five days except the head and those a branch or tag points at, +including the one a consumer's checkpoint names, and `appends_between` +then fails with the same `NotFound` as a snapshot that was never an +ancestor. RFC 0002 (proposed in #102) gives each model job a record under +`.infostash/jobs/` that lives until the job completes or is abandoned, +classes each error as retried or not, and asks for one exception to the +rerun rule and one addition to the retention rule; both are below. + +## Decision + +- **The `infostash.position` table property is the one authoritative + position.** Every commit, by a connector or a model and to a target or a + quarantine table, sets `infostash.job-id` as a table property and in the + snapshot summary. A connector commit also sets `infostash.position` as + a table property only, with no digest of it in the summary: the job id + is enough to tie a snapshot to the commit that moved the position. A model's + per-input checkpoints stay mirrored in the summary as ADR 0006 says. They + grow with the number of inputs, not with the number of runs, and the copy + in the summary records which input snapshots each target snapshot was + built from. +- **A run refuses to start when the table disagrees with its head.** If the + `infostash.job-id` table property and the `infostash.job-id` in the head + snapshot's summary differ, or one is present and the other is not, the + table was changed outside Infostash, as after a rollback by another + engine, and the position can no longer be trusted. `connectors::prepare` + (#81) and the runner's precheck over every table the job writes (#13) + fail with a new `Error::Diverged`, naming the table and both ids, before + anything is written. `Diverged` maps to `Retry::Never` (RFC 0002). A + person sets `infostash.job-id`, and for a connector table + `infostash.position`, to match the head, then runs again; the command + for that is not yet ticketed. When `Diverged` fires after a partial + commit the abandon exception below applies, so the run after the fix is + a new job. A table with no snapshot and no property is not a + disagreement. +- **A job that failed after any of its commits is rerun under the same job + id**, skipping the tables that already carry that id, before any new job + for that target is created. The skip rule of ADR 0006 is unchanged. One + exception, from RFC 0002: the job is abandoned, and the next run prepares + new work under a new id, when its rerun finds a changed head, schema or + partition spec on a table it has not committed to, when a run fails with + a `Retry::Never` error after a partial commit, or when the project + version it was prepared from is no longer the current one. +- **Snapshot expiry never goes past the oldest consumer checkpoint.** The + cutoff for a table is the timestamp of the oldest snapshot that any + consumer's checkpoint still names, read from the checkpoint table + properties consumers already store; there is no separate watermark + store. A table with no consumer is expired under its own settings. The + consumers of a table are the models that read it in the project, so a + reader outside the project gets no guarantee. The pinned iceberg-rust + keeps every snapshot whose timestamp is at or after the cutoff and every + later ancestor on the branch, as the last test of #91 confirms, so a + checkpoint snapshot and everything after it survive. Expiry also keeps + every snapshot committed by a model job whose record still exists, so + that a rerun can find its job id on each table. A connector job keeps + no record; its rerun runs before any new job for the table, so the head + it needs is never expired. The expiry properties that #91 writes at + `create_table` are a floor under any engine that expires on its own, not + the guarantee. The rule is decided now and implemented with the Phase 1 + snapshot expiry. +- **A reader whose `from` snapshot was expired gets its own error.** #89 + adds a variant of `infostash_model::Error` that names truncated history, + distinct from the `NotFound` of a snapshot that was never an ancestor, and + it is never retried. Once a full refresh exists (it needs the replace + commit of ADR 0007), the plan-time fallback of ADR 0008 takes over. + +## Consequences + +- #80 splits `append_if_head` into a summary map and a table-properties + map and takes the position out of the summary; #81 implements the connector + skip and the `Diverged` check in `prepare`, with the rollback test + below; #12 and #13 implement the rerun rule, the model skip and the + runner precheck; #91 writes the expiry floor; #40 removes the orphan + files of abandoned jobs and the old `metadata.json` files that nothing + deletes today. +- Three tests check the rules: the size of each `metadata.json` grows + linearly over 200 one-file commits; a run after the head is rolled back + by hand through the catalog refuses with `Diverged` and writes nothing; + and a checkpoint snapshot and every later one survive expiry at the + checkpoint's timestamp, while expiry past the checkpoint gives the + truncated-history error and the planned fallback. +- Rows that an abandoned job committed to side tables stay there and are + hidden by the reader rule of ADR 0006. The step that removes them is + not yet ticketed. +- A consumer that never runs holds expiry back forever on every table it + reads. Removing a model from the project removes its checkpoints with it. +- Branch writes will need a position key per branch, since one + `infostash.position` property names one line of history. diff --git a/docs/adr/README.md b/docs/adr/README.md index 7462486..c34dabc 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -18,7 +18,9 @@ a dependency, a storage layout, an invariant. ADRs are never edited after they are accepted, except for typos, links and the status line. To change a decision, write a new ADR and change the old -one's status to **Superseded by NNNN**. +one's status to **Superseded by NNNN**. To add to a decision that still +stands, write a new ADR that names what it amends and change the old +one's status to **Accepted, amended by NNNN**. ## Format @@ -27,7 +29,8 @@ Copy this into `docs/adr/NNNN-short-name.md` with the next free number: ```markdown # NNNN. Title in the active voice -- **Status:** Proposed | Accepted | Superseded by NNNN +- **Status:** Proposed | Accepted | Accepted, amended by NNNN | + Superseded by NNNN - **Date:** YYYY-MM-DD ## Context @@ -52,6 +55,7 @@ What gets easier, what gets harder, and what we must now do. | [0003](0003-layers-as-namespaces.md) | Medallion layers are Iceberg namespaces | Accepted | | [0004](0004-stage-then-atomic-commit.md) | Tasks stage files; only the job commits | Superseded by 0006 | | [0005](0005-apache-2-with-dco.md) | Apache 2.0 with a DCO, not a CLA | Accepted | -| [0006](0006-ordered-job-commits.md) | Jobs commit each table once, in order, under a job id | Accepted | +| [0006](0006-ordered-job-commits.md) | Jobs commit each table once, in order, under a job id | Accepted, amended by 0009 | | [0007](0007-hand-built-replace-commit.md) | Build the replace commit ourselves on the SQL catalog | Accepted | | [0008](0008-incremental-readers-of-replaced-tables.md) | Incremental readers of a replaced table fall back to a full refresh | Accepted | +| [0009](0009-positions-reruns-and-expiry.md) | Table positions, same-id reruns and expiry at checkpoints | Accepted | From d152db2bec9c6787860f4dba70eab10bb3969e23 Mon Sep 17 00:00:00 2001 From: Mayur Kale Date: Tue, 10 Nov 2026 14:39:09 +0000 Subject: [PATCH 5/5] docs: match ADR 0009 to the job record lifecycle RFC 0002 now keeps a job record with its Committed outcome, so say what the record holds instead of when it goes, and scope the retention rule to records without a Committed outcome. Refs #75 Signed-off-by: Mayur Kale --- docs/adr/0009-positions-reruns-and-expiry.md | 50 ++++++++++---------- 1 file changed, 24 insertions(+), 26 deletions(-) diff --git a/docs/adr/0009-positions-reruns-and-expiry.md b/docs/adr/0009-positions-reruns-and-expiry.md index 2119369..23685e2 100644 --- a/docs/adr/0009-positions-reruns-and-expiry.md +++ b/docs/adr/0009-positions-reruns-and-expiry.md @@ -32,17 +32,16 @@ and not the ancestors a delta read walks back through, and with `gc.enabled = false` nothing can ever be expired. Two rules that the connector, runner and maintenance work need are also -missing. ADR 0006 makes a rerun skip the tables that already carry its job -id, but does not say that a job which failed after one of its commits is -rerun at all, or before what. No ADR covers snapshot expiry, although any -engine that expires under a table's own settings removes every snapshot -older than five days except the head and those a branch or tag points at, -including the one a consumer's checkpoint names, and `appends_between` -then fails with the same `NotFound` as a snapshot that was never an -ancestor. RFC 0002 (proposed in #102) gives each model job a record under -`.infostash/jobs/` that lives until the job completes or is abandoned, -classes each error as retried or not, and asks for one exception to the -rerun rule and one addition to the retention rule; both are below. +missing. ADR 0006 makes a rerun skip the tables that already carry its job id, +but does not say that a job which failed after one of its commits is rerun at +all, or before what. No ADR covers snapshot expiry, although any engine that +expires under a table's own settings removes every snapshot older than five +days except the head and those a branch or tag points at, including the one a +consumer's checkpoint names, and `appends_between` then fails with the same +`NotFound` as a snapshot that was never an ancestor. RFC 0002 (proposed in +#102) gives each model job a record under `.infostash/jobs/` that says what it +committed, classes each error as retried or not, and asks for one exception to +the rerun rule and one addition to the retention rule; both are below. ## Decision @@ -80,21 +79,20 @@ rerun rule and one addition to the retention rule; both are below. version it was prepared from is no longer the current one. - **Snapshot expiry never goes past the oldest consumer checkpoint.** The cutoff for a table is the timestamp of the oldest snapshot that any - consumer's checkpoint still names, read from the checkpoint table - properties consumers already store; there is no separate watermark - store. A table with no consumer is expired under its own settings. The - consumers of a table are the models that read it in the project, so a - reader outside the project gets no guarantee. The pinned iceberg-rust - keeps every snapshot whose timestamp is at or after the cutoff and every - later ancestor on the branch, as the last test of #91 confirms, so a - checkpoint snapshot and everything after it survive. Expiry also keeps - every snapshot committed by a model job whose record still exists, so - that a rerun can find its job id on each table. A connector job keeps - no record; its rerun runs before any new job for the table, so the head - it needs is never expired. The expiry properties that #91 writes at - `create_table` are a floor under any engine that expires on its own, not - the guarantee. The rule is decided now and implemented with the Phase 1 - snapshot expiry. + consumer's checkpoint still names, read from the checkpoint table properties + consumers already store; there is no separate watermark store. A table with + no consumer is expired under its own settings. The consumers of a table are + the models that read it in the project, so a reader outside the project gets + no guarantee. The pinned iceberg-rust keeps every snapshot whose timestamp + is at or after the cutoff and every later ancestor on the branch, as the + last test of #91 confirms, so a checkpoint snapshot and everything after it + survive. Expiry also keeps every snapshot committed by a model job whose + record still exists without a `Committed` outcome, so that a rerun can find + its job id on each table. A connector job keeps no record; its rerun runs + before any new job for the table, so the head it needs is never expired. The + expiry properties that #91 writes at `create_table` are a floor under any + engine that expires on its own, not the guarantee. The rule is decided now + and implemented with the Phase 1 snapshot expiry. - **A reader whose `from` snapshot was expired gets its own error.** #89 adds a variant of `infostash_model::Error` that names truncated history, distinct from the `NotFound` of a snapshot that was never an ancestor, and