Repository navigation
Description
ADR 0006 says that a rerun of a job commits only to tables that do not yet carry its job id (docs/adr/0006-ordered-job-commits.md:47-48). Nothing in production reads infostash.job-id at a0b5e6c. JOB_ID (crates/connectors/src/lib.rs:25) is written at :127 and read only by a test (crates/connectors/tests/csv_ingest.rs:184), and prepare reads only the head and the position from the table state (crates/connectors/src/lib.rs:70-73).
When a connector commit lands but its caller crashes or times out before it sees the result, the rerun prepares from the moved position. If no file arrived in the meantime, the rerun returns Ok(None), which a caller cannot tell apart from a run that found nothing new. The job would then report JobOutcome::NoChange (crates/model/src/lib.rs:138-139), which carries no JobMetrics, so the rows it did commit would be missing from metering and lineage (:121). If a file did arrive, the rerun makes a second commit under the same job id, against the rule of one commit per output table carrying the job id (docs/adr/0006-ordered-job-commits.md:39-40). No row loads twice on this path, because the position moves in the same commit as the rows. A rerun that prepared before the first commit landed gets Error::Conflict instead, which does not tell it that the winning commit was its own.
This ticket makes the connector recognize its own job id before it reads and after a conflict, adds a storage method that also finds the id on older snapshots, and gives the tests distinct ids. Setup::run and three hand-written prepares all use JobId(7) (crates/connectors/tests/csv_ingest.rs:54, :155, :195, :198), so once the rule holds their later runs would load nothing. The skip itself is already decided by ADR 0006, so it needs no RFC of its own. The same skip for model jobs stays with #13, which can reuse the storage method, and the quarantine-then-target rerun stays with #12. Run ids, and how JobOutcome (crates/model/src/lib.rs:132-139) reports an already-committed table, belong to the execution-contracts RFC in #74.
Crates: crates/connectors, crates/storage (the new Warehouse::committed_by)
Done when
-
preparecompares theinfostash.job-idtable property, read in the sameWarehouse::stateload as the position (crates/connectors/src/lib.rs:70-73), with its own job id. When they match, it reads nothing from the connector, and thecommitthat follows writes nothing and reports the run as already committed. - A new
Warehouse::committed_by(table, job)incrates/storagereturns the newest snapshot whose summary (Summary::additional_properties) carries the job id, or none. It walks back from the head through parent snapshots likeappends_between(crates/storage/src/lib.rs:561-579), but with no stop atfromand no error on a snapshot that is not an append (:570-577). Where a parent is no longer in the metadata after snapshot expiry, the walk ends there instead of failing withNotFound(:564-566).prepareuses it when the table property names another job, because every commit that sets properties overwrites that property (crates/storage/src/lib.rs:212-218). - When
append_if_headfails withError::Conflict(crates/storage/src/lib.rs:198-202,:539-546),Prepared::commitrepeats the check thatpreparemakes. If its own job id is now on a snapshot, it reports the run as already committed. Otherwise it still returnsError::Conflictand commits nothing. - An already-committed run gets its own result from
Prepared::commit, which names the snapshot that carries the job id. That result is distinct from a newCommitand fromOk(None), which keeps meaning that there was nothing to commit (crates/connectors/src/lib.rs:101-102). - The tests use a new job id for each new run and the same id only for a rerun.
Setup::run(crates/connectors/tests/csv_ingest.rs:51-58) takes the id as an argument instead of fixingJobId(7)(:54). Ina_crash_before_the_commit_loads_each_row_once_on_rerun(:146-165), the crashed prepare (:155) gets a new id and the rerun after it (:162) reuses that id.the_position_and_job_id_are_table_propertiesasserts the id it used instead of"7"(:184). - Test: commit under
JobId(9), add a file and rerun underJobId(9). The rerun makes no snapshot, leaves the new file for a run under a new id, and its result names the snapshot of the first run. In a second case, another job commits between the two runs, so the table property names that job. The rerun underJobId(9)still makes no snapshot and names the same snapshot, which onlyWarehouse::committed_bycan find. - Test:
two_runs_from_the_same_position_conflict(crates/connectors/tests/csv_ingest.rs:187-207) is replaced by two races. In a race under one id, the loser reports already committed and names the winner's snapshot. In a race under two ids, the loser still getsError::Conflictand the table keeps one new snapshot. - The doc comments on the module,
prepareandPrepared::commit(crates/connectors/src/lib.rs:1-7,:60-62,:101-106) say that a job id names one unit of work and that a rerun under the same id commits nothing to a table that already carries it. Once docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 is accepted, they also state its rule that a job which failed after any of its commits is rerun under the same id before a new job for that table is created.
Points: 3 (C11 3)
Depends on: #75 (the rerun-after-partial-commit rule only)
Blocks: nothing
Source: design review of 2026-11-01, finding C11 and design point S6-10.
Activity
codecapo commentedon Nov 10, 2026
RFC 0002 (#102) answers the two points this ticket defers to it. The run id is
RunId(u64), assigned by the control plane for every run and never reused; it goes into staged file names,TaskContextand logs, not into snapshot properties. An already-committed table is reported throughCommitRecord { table, snapshot, already_committed: true }insideJobOutcome::Committed, and a rerun that finds every table already committed returnsCommittedwith zero metrics rather thanNoChange, so the metrics gap described above closes. For the connector, the RFC leavesPrepared::commitreturningResult<Option<Commit>>until this ticket, which should make itResult<Option<CommitRecord>>:Nonekeeps meaning nothing to load, and the already-committed case is reported through the same flag.connectors::preparetakes theRunIdnext to theJobId, both assigned by the caller.codecapo commentedon Nov 10, 2026
ADR 0009 (PR #103) settles the rerun rule this ticket implements: a job that failed after any of its commits is rerun under the same job id, skipping the tables that already carry it, before any new job for the target is created. The abandon exception from RFC 0002 (a changed head, schema or partition spec on a table not yet committed to, an error that is never retried after a partial commit, or a changed project version) is carried into the ADR. It also adds a
Divergederror, never retried, for a table whoseinfostash.job-idproperty and head summary disagree; the runner precheck should fail with it before anything is written.codecapo commentedon Nov 10, 2026
To be precise about ownership: the ADR assigns the
Divergedcheck inprepare, the newErrorvariant and the test that a run after a hand rollback through the catalog refuses and writes nothing to this ticket. The runner precheck over every table a job writes goes to #13.docs: add ADR 0009 amending 0006 on position, reruns and expiry (#103)
docs: add RFC 0002 for the job and runner contracts (#102)