Repository navigation
Description
Write a new ADR that amends ADR 0006 (one commit per table carrying the job id; a connector's position commits with its data) on four points, so that the connector, runner and maintenance work built on it starts from settled rules. At a0b5e6c, append_if_head writes one properties map into both the snapshot summary (crates/storage/src/lib.rs:209) and the table properties (:212-218), as docs/adr/0006-ordered-job-commits.md:49-51 requires, so every commit writes a new metadata.json in which each snapshot's summary keeps its own copy of the CSV connector's loaded-file list (crates/connectors/src/csv.rs:33-36). Each metadata.json therefore grows with the square of the number of runs (with 65-character paths, an estimated 35 MB per file and 12 GB on disk after 1000 hourly one-file runs), yet prepare reads only the table property (crates/connectors/src/lib.rs:72) and no production code reads the summary copy. ADR 0006:47-48 also does not say that a job which failed after one of its commits must be rerun under the same id, and no ADR covers snapshot expiry, although a checkpoint lost to expiry fails with the same NotFound variant as a snapshot that was never an ancestor (crates/storage/src/lib.rs:564-566, :581-583). Accepted ADRs are not edited (docs/adr/README.md:19-21), so the change is a new record that 0006's status line points to.
Where: docs/adr/ (a new record at the next free number), docs/adr/README.md (index), docs/adr/0006-ordered-job-commits.md (status line only)
Done when
- A new ADR in
docs/adr/at the next free number, in the format ofdocs/adr/README.md:27-44, amends ADR 0006, which changes only on its status line (docs/adr/README.md:19-21) so that it points to the new record, and the index atdocs/adr/README.md:46-55lists the new record and shows 0006's new status, as the row for 0004 does (docs/adr/README.md:53). - For connectors, the ADR replaces the summary mirror of ADR 0006:49-51: the table properties keep the read position as
infostash.positiontogether withinfostash.job-id, the snapshot summary keepsinfostash.job-idand at most a digest of the position, and the ADR says whether a model's per-input checkpoints, which grow with the number of inputs rather than runs, stay mirrored. - The ADR makes the
infostash.positiontable property the one authoritative position and says that a run refuses to start, with a named error and no write, when theinfostash.job-idtable property and the job id in the head snapshot's summary disagree, as after a rollback by another engine. - The ADR adds that a job which failed after any of its commits (for example after its quarantine commit and before its target commit) is rerun under the same job id, skipping the tables that already carry that id, before any new job for that target is created, while the skip rule of ADR 0006:47-48 itself stays unchanged and is implemented in connectors: skip tables that already carry the job id and use distinct test ids #81 and In-process runner: stage, run tasks, commit once #13.
- The ADR adds the retention rule that no ADR covers today: snapshot expiry never goes past the oldest consumer checkpoint, because its cutoff is that checkpoint's timestamp, read from the checkpoints consumers already store as table properties rather than from a new watermark store, and the expiry properties that storage: set a snapshot expiry floor at create_table #91 sets at
create_tableare a floor, not the guarantee; the rule is decided now and implemented with the Phase 1 snapshot expiry (ROADMAP.md:28). - The ADR says that a reader whose
fromsnapshot was expired gets an error distinct from theNotFoundof a non-ancestor, so that once a full refresh exists (it needs the replace commit that storage: spike the replace commit route (decision D1) #72 decides), the plan-time fallback (engine: fall back to a full refresh at plan time and fix files_added_by #89) takes over instead of a retried failure. - The Context section records the estimated metadata growth and the alternatives ruled out: compressing
metadata.json(the reader still inflates the whole file), a per-connector delta position (the position is opaque,crates/connectors/src/lib.rs:27-31), reading the head's summary as the position (it keeps the growth), and tags orgc.enabled = falsefor retention. - The Consequences section names the follow-up work (connectors: split the snapshot summary from the position #80, connectors: skip tables that already carry the job id and use distinct test ids #81, storage: set a snapshot expiry floor at create_table #91, In-process runner: stage, run tasks, commit once #13, and storage: remove orphan data files #40, where deleting the old
metadata.jsonfiles that nothing removes today also belongs), the tests that check the rules (per-file metadata size growing linearly over 200 one-file commits; a run that refuses with the named error and writes nothing after the head is rolled back by hand through the catalog; a checkpoint snapshot and every later one surviving expiry at the checkpoint's timestamp, and expiry past the checkpoint giving the distinct error and the planned fallback), and the per-branch position key that branch writes will need.
Points: 2 (D4)
Depends on: nothing
Blocks: #13 (where per-input checkpoints live, and reruns after a partial commit), #40 (the retention rule decides which snapshots stay live, and so which files are orphans), #80, #81 (the rerun-after-partial-commit rule only), #91
Source: design review of 2026-11-01, section 8.5, D4 (findings C05, C10, C11 and C23, and design point S6-10).
Activity
codecapo commentedon Nov 10, 2026
RFC 0002 (#102) asks this ADR to carry two more rules alongside the ones above:
Conflict), when a run fails withRetry::Neverafter a partial commit, or when the recorded project version is no longer the current one. Rerunning any of those could only fail again. Rows the job committed to side tables stay and are hidden by the ADR 0006 reader rule.Warehouse::committed_bycan still find the id when the job is rerun. The RFC's open question 5 depends on this.codecapo commentedon Nov 10, 2026
ADR 0009 is in PR #103, stacked on #100. Beyond the Done-when list it carries the abandon exception and the rule that expiry keeps every snapshot of a model job whose record still exists, as the comment above asked. Two choices to note: the summary keeps
infostash.job-idand no digest of the position, since the job id alone ties a snapshot to the commit that moved it; and the rollback disagreement is a newDivergederror mapped toRetry::Never, owned by #81 forprepareand #13 for the runner precheck, becauseConflictis retried at the job level and would meet the same disagreement again.docs: match ADR 0009 to the job record lifecycle
docs: add ADR 0009 amending 0006 on position, reruns and expiry (#103)
docs: add RFC 0002 for the job and runner contracts (#102)