Skip to content

In-process runner: stage, run tasks, commit once #13

Open

Description

@codecapo

Run a Job end to end in this process: plan stages with crates/dist, run tasks, then make one Iceberg commit.

Crates: crates/jobs, crates/dist

Done when

  • An in-process Runner executes a job's stages and tasks
  • Tasks write only to staging; the job makes one atomic commit (ADR 0004)
  • Each model reads only upstream snapshots since its last run (uses Incremental scan: read rows added between two snapshots #11)
  • The last processed snapshot per input is stored so the next run knows where to start
  • Returns JobOutcome::NoChange when no input has new snapshots
  • A failed task retries; a test shows a retried task does not duplicate rows
  • Staged files are named by job, task and attempt, and each attempt writes a versioned staging manifest of its DataFiles
  • Each snapshot carries infostash.job-id; rerunning a finished job commits nothing
  • A stale run fails instead of committing (same head guard as CSV connector with exactly-once loading into Raw #9)

Depends on: #10, #11
Blocks: #14

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Oct 30, 2026

    @codecapo
    OwnerAuthor

    Spike findings (commit semantics and engine)

    • Staging works. Each task attempt writes Parquet with a job-<id>-task-<n>-attempt-<m> name in the table's data location and a staging manifest (its DataFiles via the public serialize_data_file_to_json). The coordinator records one winning manifest per task and commits only those in one fast_append. Failed and duplicate attempts are never committed.
    • Files are committed where they were written, so "staging" means an attempt-specific name, not a separate area.
    • iceberg-datafusion's INSERT commits once per statement (IcebergCommitExec). To commit once per job, the runner runs IcebergWriteExec and commits the data files itself.
    • Input positions: keep the last processed snapshot per input in a table property of the output, written in the same transaction, with the same stale-run guard as CSV connector with exactly-once loading into Raw #9. iceberg-rust silently rebases stale commits.
    • Use the job id marker from Row checks and quarantine tables #12 for idempotent reruns; replays under new file names aren't caught by the duplicate-file check.
    • NoChange must skip the commit; an empty append is an error.
    • Pin each input to the snapshot seen at job start. The upstream table provider reloads to the latest snapshot on every scan, so this needs our own provider (new issue, see Tracking: Phase 0 (0.1.0) #19).

    Points: 13 -> 8.

  3. codecapo commented on Nov 6, 2026

    @codecapo
    OwnerAuthor

    The design review of 2026-11-01 touches this issue. The runner keeps the input range step, the per-input checkpoint and the model commit path, takes on the job's pinned start time, the job-id skip and the rerun rules for model jobs, and now waits on two decisions, #74 and #75, and on the delta provider of #85.

    Suggested additions to the Done-when:

    • A model job pins each input's range in Job.inputs (crates/model/src/lib.rs:107-108 at a0b5e6c) when it is created, from the stored checkpoint to that input's head at that moment, and every attempt and every rerun of the job reads exactly those ranges through the delta provider of engine: add a StreamingTable delta provider over scan_range #85, as ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:71-72).
    • When the runner creates a model job, it also fixes one start time for that job, keeps the time with the job id and supplies the same time to the engine (Run a model with DataFusion and append to its table #10) on every attempt and on every rerun, so that now() in a @check (Row checks and quarantine tables #12) splits rows the same way each time, under the rule docs: revise RFC 0001 with the incremental rules (decision D2) #73 adds to RFC 0001.
    • Each input's checkpoint, which is the end of its pinned range and not the input's head at commit time, is an infostash.* table property of the model's table, set in the same commit as the rows (crates/storage/src/lib.rs:212-218), and goes into the snapshot summary as well (:209) only if docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 keeps model checkpoints mirrored, using the two maps of connectors: split the snapshot summary from the position #80.
    • The existing line "Tasks write only to staging; the job makes one atomic commit (ADR 0004)" now follows ADR 0006, which supersedes 0004 (docs/adr/0004-stage-then-atomic-commit.md:3): the job makes one commit per output table, the commit path that ADR 0006 leaves to this issue (docs/adr/0006-ordered-job-commits.md:39-40, :56-58), and once dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 is accepted JobOutcome::Committed lists those commits in place of its single snapshot (crates/model/src/lib.rs:134-137). Test: a job that writes two tables reports two commits.
    • Tasks stage files through the public stage and the runner commits them through commit_staged, as dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 specifies, both built from the private halves of append_if_head (the head check at crates/storage/src/lib.rs:198-202, write_files at :337 and commit_if_head at :525; the test at :852-855 already calls the last two on their own); the runner supplies the run id, the task (its stage and partition) and the attempt for the staged-file prefix of connectors: stage under a job prefix and run checks before the writer #82, and model commits never go through IcebergCommitExec, the INSERT INTO commit of the iceberg-datafusion crate pinned at Cargo.toml:31.
    • Once dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 is accepted, run_in_process retries only errors that Error::retry classifies as Retry::Task, where today it retries every error, Conflict included (crates/dist/src/lib.rs:102-109), so a lost commit race goes back to the job instead of rerunning the task, and storage errors keep iceberg's retryable flag, which storage() drops (crates/storage/src/lib.rs:322-324); attempt counting stays with dist: document and test RetryPolicy attempt counting #29. Tests: the Fake test stage (crates/dist/src/lib.rs:123) is not retried when it fails with Conflict, and is retried when it fails with an I/O Execution error.
    • The existing line "Each snapshot carries infostash.job-id; rerunning a finished job commits nothing" becomes a check per output table: before staging, the runner finds the output tables that already carry the job id, from the infostash.job-id table property and, when that names another job, from Warehouse::committed_by of connectors: skip tables that already carry the job id and use distinct test ids #81; it commits only to the other tables, checks again after a Conflict, and reports the skipped tables as already committed, so a rerun of a finished job no longer looks like NoChange, which stays the outcome when no input has new snapshots. Today the id is written (crates/connectors/src/lib.rs:127) but read only by a test (crates/connectors/tests/csv_ingest.rs:184), and the quarantine-then-target rerun of Row checks and quarantine tables #12 builds on this skip. Test: run a model job, append to an input, rerun under the same id: no new snapshot on its table, and the outcome names the first one.
    • If a commit of staged files fails on iceberg-rust's duplicate-file check while the table already carries the job id, the runner reports the table as already committed, not as the Error::Storage that commit_if_head returns today for any commit error that did not trip the head guard (crates/storage/src/lib.rs:547, :322-324), and turns the check off only for commits the job-id skip has already cleared. Test: commit under one job id, commit the same staged paths again, and get already committed.
    • Once docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 is accepted, a job that failed after any of its commits, for example after its quarantine commit, is rerun under the same id before the runner creates any new job for that target; who recovers that id after a crash is left to dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74.
    • Once docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 is accepted, a model run refuses to start, with a named error and no write, when its table's infostash.job-id property and the job id in the head snapshot's summary disagree, as after a rollback by another engine, although the TableState doc assumes that a value stored in the properties and the snapshot it belongs to always agree (crates/storage/src/lib.rs:51-52). Test: commit, roll the head back by hand through the catalog, and the rerun refuses without writing.
    • Test: a job that stops after its tasks staged files and before any commit, then a rerun under the same job id, leaves one new snapshot per output table, and every file the first run left behind carries that run's id and its attempt in its name, so storage: remove orphan data files #40 can attribute it.

    Work that moved to new tickets:

    Suggested re-score: 13 instead of the 8 in #19, an estimate. Section 8.5 of the design review of 2026-11-01 scores three milestone M2 items that stay in this issue, the per-input checkpoint property at 3, the scheduler range step and NoChange at 5 and the model commit path through append_if_head at 5, which make 13; the StreamingTable delta provider it scores at 8 is now #85. It names no separate points for the pinned start time, the retry change in run_in_process or the duplicate-file, start-up and rerun checks above, which may take this issue past 13, and #19 splits anything above 13 before work starts. This is a suggestion only; #19 decides.

    Source: design review of 2026-11-01, findings C10, C11, C12, C13, C15, C20, C23 and C24, contracts S5-01 to S5-06 and S5-12, design point S6-10, crash matrix rows 2, 4 and 10, and section 8.5 (D3, D4 and M2).

  4. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    RFC 0002 (#102) settles the contracts this ticket builds on. One Done-when item changes once it is accepted: "Tasks write only to staging; the job makes one atomic commit (ADR 0004)" becomes one Warehouse::commit_staged per output table, side tables first and the target last, each carrying infostash.job-id (ADR 0006). The per-attempt staging manifest stays: Warehouse::stage writes it as {job}-{run}-{stage}-{partition}-{attempt}.staged.json next to the data files and returns the same record as StagedFiles; commit_staged deletes it after the commit.

    Also from the RFC: Runner::run is async fn run(&self, job: &Job, run: RunId) -> JobOutcome; the runner skips tables that already carry the job id (reusing Warehouse::committed_by from #81) and checks Job::expected (head, schema id, partition spec id per output table) on the rest before any task runs; the Done-when item "returns JobOutcome::NoChange" moves to a new jobs::prepare function, since the runner only sees jobs that have work; a retryable storage error from commit_staged is retried in place up to max_task_attempts; run_in_process returns a TaskRun per task; and the test fixture's Job struct literal is replaced by Job::new, since Job becomes #[non_exhaustive].

  5. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    ADR 0009 (PR #103) confirms the skip rule of ADR 0006 unchanged and names this ticket, with #81, as where the rerun-under-the-same-id rule is implemented. See the ADR for the abandon exception and the Diverged precheck.

  6. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    Two further points from the ADR for this ticket: the runner precheck over every table the job writes fails with Diverged, never retried, when a table's infostash.job-id property and its head summary disagree; and every model commit sets infostash.job-id as a table property as well as in the summary, so the precheck has both to compare.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions