Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/adr/0006-ordered-job-commits.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 0006. Jobs commit each table once, in order, under a job id

- **Status:** Accepted
- **Status:** Accepted, amended by [0009](0009-positions-reruns-and-expiry.md)
- **Date:** 2026-10-30
- **Supersedes:** [0004](0004-stage-then-atomic-commit.md)

Expand Down
123 changes: 123 additions & 0 deletions docs/adr/0009-positions-reruns-and-expiry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# 0009. Table positions, same-id reruns and expiry at checkpoints

- **Status:** Accepted
- **Date:** 2026-11-10
- **Amends:** [0006](0006-ordered-job-commits.md)

## Context

ADR 0006 stores a connector's read position, and a model's last processed
snapshot per input, as a table property mirrored in the snapshot summary.
`Warehouse::append_if_head` writes one properties map into both places, so
every snapshot of a connector table keeps its own copy of the position. For
the CSV connector the position is the list of every file loaded so far, and
Iceberg keeps every snapshot's summary in every later `metadata.json`. The
file written by run n therefore holds n copies of a list that is n entries
long, and the metadata directory keeps every file ever written. With paths
of 65 characters and one new file per run:

| Hourly one-file runs | Latest `metadata.json` | Metadata on disk |
|---|---|---|
| 100 | 0.3 MB | 11 MB |
| 1000 | 35 MB | 12 GB |

Only the table property is read: `connectors::prepare` takes the position
from it, and no production code reads the summary copy. Four alternatives
were ruled out. Compressing `metadata.json` does not help a reader that
still inflates the whole file. A per-connector delta position cannot be
defined, because the position is opaque to everything except the connector.
Reading the head snapshot's summary as the position keeps the growth. Tags,
or `gc.enabled = false`, are not a retention rule: a tag keeps one snapshot
and not the ancestors a delta read walks back through, and with
`gc.enabled = false` nothing can ever be expired.

Two rules that the connector, runner and maintenance work need are also
missing. ADR 0006 makes a rerun skip the tables that already carry its job id,
but does not say that a job which failed after one of its commits is rerun at
all, or before what. No ADR covers snapshot expiry, although any engine that
expires under a table's own settings removes every snapshot older than five
days except the head and those a branch or tag points at, including the one a
consumer's checkpoint names, and `appends_between` then fails with the same
`NotFound` as a snapshot that was never an ancestor. RFC 0002 (proposed in
#102) gives each model job a record under `.infostash/jobs/` that says what it
committed, classes each error as retried or not, and asks for one exception to
the rerun rule and one addition to the retention rule; both are below.

## Decision

- **The `infostash.position` table property is the one authoritative
position.** Every commit, by a connector or a model and to a target or a
quarantine table, sets `infostash.job-id` as a table property and in the
snapshot summary. A connector commit also sets `infostash.position` as
a table property only, with no digest of it in the summary: the job id
is enough to tie a snapshot to the commit that moved the position. A model's
per-input checkpoints stay mirrored in the summary as ADR 0006 says. They
grow with the number of inputs, not with the number of runs, and the copy
in the summary records which input snapshots each target snapshot was
built from.
- **A run refuses to start when the table disagrees with its head.** If the
`infostash.job-id` table property and the `infostash.job-id` in the head
snapshot's summary differ, or one is present and the other is not, the
table was changed outside Infostash, as after a rollback by another
engine, and the position can no longer be trusted. `connectors::prepare`
(#81) and the runner's precheck over every table the job writes (#13)
fail with a new `Error::Diverged`, naming the table and both ids, before
anything is written. `Diverged` maps to `Retry::Never` (RFC 0002). A
person sets `infostash.job-id`, and for a connector table
`infostash.position`, to match the head, then runs again; the command
for that is not yet ticketed. When `Diverged` fires after a partial
commit the abandon exception below applies, so the run after the fix is
a new job. A table with no snapshot and no property is not a
disagreement.
- **A job that failed after any of its commits is rerun under the same job
id**, skipping the tables that already carry that id, before any new job
for that target is created. The skip rule of ADR 0006 is unchanged. One
exception, from RFC 0002: the job is abandoned, and the next run prepares
new work under a new id, when its rerun finds a changed head, schema or
partition spec on a table it has not committed to, when a run fails with
a `Retry::Never` error after a partial commit, or when the project
version it was prepared from is no longer the current one.
- **Snapshot expiry never goes past the oldest consumer checkpoint.** The
cutoff for a table is the timestamp of the oldest snapshot that any
consumer's checkpoint still names, read from the checkpoint table properties
consumers already store; there is no separate watermark store. A table with
no consumer is expired under its own settings. The consumers of a table are
the models that read it in the project, so a reader outside the project gets
no guarantee. The pinned iceberg-rust keeps every snapshot whose timestamp
is at or after the cutoff and every later ancestor on the branch, as the
last test of #91 confirms, so a checkpoint snapshot and everything after it
survive. Expiry also keeps every snapshot committed by a model job whose
record still exists without a `Committed` outcome, so that a rerun can find
its job id on each table. A connector job keeps no record; its rerun runs
before any new job for the table, so the head it needs is never expired. The
expiry properties that #91 writes at `create_table` are a floor under any
engine that expires on its own, not the guarantee. The rule is decided now
and implemented with the Phase 1 snapshot expiry.
- **A reader whose `from` snapshot was expired gets its own error.** #89
adds a variant of `infostash_model::Error` that names truncated history,
distinct from the `NotFound` of a snapshot that was never an ancestor, and
it is never retried. Once a full refresh exists (it needs the replace
commit of ADR 0007), the plan-time fallback of ADR 0008 takes over.

## Consequences

- #80 splits `append_if_head` into a summary map and a table-properties
map and takes the position out of the summary; #81 implements the connector
skip and the `Diverged` check in `prepare`, with the rollback test
below; #12 and #13 implement the rerun rule, the model skip and the
runner precheck; #91 writes the expiry floor; #40 removes the orphan
files of abandoned jobs and the old `metadata.json` files that nothing
deletes today.
- Three tests check the rules: the size of each `metadata.json` grows
linearly over 200 one-file commits; a run after the head is rolled back
by hand through the catalog refuses with `Diverged` and writes nothing;
and a checkpoint snapshot and every later one survive expiry at the
checkpoint's timestamp, while expiry past the checkpoint gives the
truncated-history error and the planned fallback.
- Rows that an abandoned job committed to side tables stay there and are
hidden by the reader rule of ADR 0006. The step that removes them is
not yet ticketed.
- A consumer that never runs holds expiry back forever on every table it
reads. Removing a model from the project removes its checkpoints with it.
- Branch writes will need a position key per branch, since one
`infostash.position` property names one line of history.
10 changes: 7 additions & 3 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@ a dependency, a storage layout, an invariant.

ADRs are never edited after they are accepted, except for typos, links and
the status line. To change a decision, write a new ADR and change the old
one's status to **Superseded by NNNN**.
one's status to **Superseded by NNNN**. To add to a decision that still
stands, write a new ADR that names what it amends and change the old
one's status to **Accepted, amended by NNNN**.

## Format

Expand All @@ -27,7 +29,8 @@ Copy this into `docs/adr/NNNN-short-name.md` with the next free number:
```markdown
# NNNN. Title in the active voice

- **Status:** Proposed | Accepted | Superseded by NNNN
- **Status:** Proposed | Accepted | Accepted, amended by NNNN |
Superseded by NNNN
- **Date:** YYYY-MM-DD

## Context
Expand All @@ -52,6 +55,7 @@ What gets easier, what gets harder, and what we must now do.
| [0003](0003-layers-as-namespaces.md) | Medallion layers are Iceberg namespaces | Accepted |
| [0004](0004-stage-then-atomic-commit.md) | Tasks stage files; only the job commits | Superseded by 0006 |
| [0005](0005-apache-2-with-dco.md) | Apache 2.0 with a DCO, not a CLA | Accepted |
| [0006](0006-ordered-job-commits.md) | Jobs commit each table once, in order, under a job id | Accepted |
| [0006](0006-ordered-job-commits.md) | Jobs commit each table once, in order, under a job id | Accepted, amended by 0009 |
| [0007](0007-hand-built-replace-commit.md) | Build the replace commit ourselves on the SQL catalog | Accepted |
| [0008](0008-incremental-readers-of-replaced-tables.md) | Incremental readers of a replaced table fall back to a full refresh | Accepted |
| [0009](0009-positions-reruns-and-expiry.md) | Table positions, same-id reruns and expiry at checkpoints | Accepted |
Loading