Skip to content

connectors: skip tables that already carry the job id and use distinct test ids #81

Open

Description

@codecapo

ADR 0006 says that a rerun of a job commits only to tables that do not yet carry its job id (docs/adr/0006-ordered-job-commits.md:47-48). Nothing in production reads infostash.job-id at a0b5e6c. JOB_ID (crates/connectors/src/lib.rs:25) is written at :127 and read only by a test (crates/connectors/tests/csv_ingest.rs:184), and prepare reads only the head and the position from the table state (crates/connectors/src/lib.rs:70-73).

When a connector commit lands but its caller crashes or times out before it sees the result, the rerun prepares from the moved position. If no file arrived in the meantime, the rerun returns Ok(None), which a caller cannot tell apart from a run that found nothing new. The job would then report JobOutcome::NoChange (crates/model/src/lib.rs:138-139), which carries no JobMetrics, so the rows it did commit would be missing from metering and lineage (:121). If a file did arrive, the rerun makes a second commit under the same job id, against the rule of one commit per output table carrying the job id (docs/adr/0006-ordered-job-commits.md:39-40). No row loads twice on this path, because the position moves in the same commit as the rows. A rerun that prepared before the first commit landed gets Error::Conflict instead, which does not tell it that the winning commit was its own.

This ticket makes the connector recognize its own job id before it reads and after a conflict, adds a storage method that also finds the id on older snapshots, and gives the tests distinct ids. Setup::run and three hand-written prepares all use JobId(7) (crates/connectors/tests/csv_ingest.rs:54, :155, :195, :198), so once the rule holds their later runs would load nothing. The skip itself is already decided by ADR 0006, so it needs no RFC of its own. The same skip for model jobs stays with #13, which can reuse the storage method, and the quarantine-then-target rerun stays with #12. Run ids, and how JobOutcome (crates/model/src/lib.rs:132-139) reports an already-committed table, belong to the execution-contracts RFC in #74.

Crates: crates/connectors, crates/storage (the new Warehouse::committed_by)

Done when

  • prepare compares the infostash.job-id table property, read in the same Warehouse::state load as the position (crates/connectors/src/lib.rs:70-73), with its own job id. When they match, it reads nothing from the connector, and the commit that follows writes nothing and reports the run as already committed.
  • A new Warehouse::committed_by(table, job) in crates/storage returns the newest snapshot whose summary (Summary::additional_properties) carries the job id, or none. It walks back from the head through parent snapshots like appends_between (crates/storage/src/lib.rs:561-579), but with no stop at from and no error on a snapshot that is not an append (:570-577). Where a parent is no longer in the metadata after snapshot expiry, the walk ends there instead of failing with NotFound (:564-566). prepare uses it when the table property names another job, because every commit that sets properties overwrites that property (crates/storage/src/lib.rs:212-218).
  • When append_if_head fails with Error::Conflict (crates/storage/src/lib.rs:198-202, :539-546), Prepared::commit repeats the check that prepare makes. If its own job id is now on a snapshot, it reports the run as already committed. Otherwise it still returns Error::Conflict and commits nothing.
  • An already-committed run gets its own result from Prepared::commit, which names the snapshot that carries the job id. That result is distinct from a new Commit and from Ok(None), which keeps meaning that there was nothing to commit (crates/connectors/src/lib.rs:101-102).
  • The tests use a new job id for each new run and the same id only for a rerun. Setup::run (crates/connectors/tests/csv_ingest.rs:51-58) takes the id as an argument instead of fixing JobId(7) (:54). In a_crash_before_the_commit_loads_each_row_once_on_rerun (:146-165), the crashed prepare (:155) gets a new id and the rerun after it (:162) reuses that id. the_position_and_job_id_are_table_properties asserts the id it used instead of "7" (:184).
  • Test: commit under JobId(9), add a file and rerun under JobId(9). The rerun makes no snapshot, leaves the new file for a run under a new id, and its result names the snapshot of the first run. In a second case, another job commits between the two runs, so the table property names that job. The rerun under JobId(9) still makes no snapshot and names the same snapshot, which only Warehouse::committed_by can find.
  • Test: two_runs_from_the_same_position_conflict (crates/connectors/tests/csv_ingest.rs:187-207) is replaced by two races. In a race under one id, the loser reports already committed and names the winner's snapshot. In a race under two ids, the loser still gets Error::Conflict and the table keeps one new snapshot.
  • The doc comments on the module, prepare and Prepared::commit (crates/connectors/src/lib.rs:1-7, :60-62, :101-106) say that a job id names one unit of work and that a rerun under the same id commits nothing to a table that already carries it. Once docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 is accepted, they also state its rule that a job which failed after any of its commits is rerun under the same id before a new job for that table is created.

Points: 3 (C11 3)

Depends on: #75 (the rerun-after-partial-commit rule only)
Blocks: nothing

Source: design review of 2026-11-01, finding C11 and design point S6-10.

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    RFC 0002 (#102) answers the two points this ticket defers to it. The run id is RunId(u64), assigned by the control plane for every run and never reused; it goes into staged file names, TaskContext and logs, not into snapshot properties. An already-committed table is reported through CommitRecord { table, snapshot, already_committed: true } inside JobOutcome::Committed, and a rerun that finds every table already committed returns Committed with zero metrics rather than NoChange, so the metrics gap described above closes. For the connector, the RFC leaves Prepared::commit returning Result<Option<Commit>> until this ticket, which should make it Result<Option<CommitRecord>>: None keeps meaning nothing to load, and the already-committed case is reported through the same flag. connectors::prepare takes the RunId next to the JobId, both assigned by the caller.

  3. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    ADR 0009 (PR #103) settles the rerun rule this ticket implements: a job that failed after any of its commits is rerun under the same job id, skipping the tables that already carry it, before any new job for the target is created. The abandon exception from RFC 0002 (a changed head, schema or partition spec on a table not yet committed to, an error that is never retried after a partial commit, or a changed project version) is carried into the ADR. It also adds a Diverged error, never retried, for a table whose infostash.job-id property and head summary disagree; the runner precheck should fail with it before anything is written.

  4. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    To be precise about ownership: the ADR assigns the Diverged check in prepare, the new Error variant and the test that a run after a hand rollback through the catalog refuses and writes nothing to this ticket. The runner precheck over every table a job writes goes to #13.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions