You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
In-process runner: stage, run tasks, commit once #13
Staging works. Each task attempt writes Parquet with a job-<id>-task-<n>-attempt-<m> name in the table's data location and a staging manifest (its DataFiles via the public serialize_data_file_to_json). The coordinator records one winning manifest per task and commits only those in one fast_append. Failed and duplicate attempts are never committed.
Files are committed where they were written, so "staging" means an attempt-specific name, not a separate area.
iceberg-datafusion's INSERT commits once per statement (IcebergCommitExec). To commit once per job, the runner runs IcebergWriteExec and commits the data files itself.
Input positions: keep the last processed snapshot per input in a table property of the output, written in the same transaction, with the same stale-run guard as CSV connector with exactly-once loading into Raw #9. iceberg-rust silently rebases stale commits.
Use the job id marker from Row checks and quarantine tables #12 for idempotent reruns; replays under new file names aren't caught by the duplicate-file check.
NoChange must skip the commit; an empty append is an error.
Pin each input to the snapshot seen at job start. The upstream table provider reloads to the latest snapshot on every scan, so this needs our own provider (new issue, see Tracking: Phase 0 (0.1.0) #19).
The design review of 2026-11-01 touches this issue. The runner keeps the input range step, the per-input checkpoint and the model commit path, takes on the job's pinned start time, the job-id skip and the rerun rules for model jobs, and now waits on two decisions, #74 and #75, and on the delta provider of #85.
Suggested additions to the Done-when:
A model job pins each input's range in Job.inputs (crates/model/src/lib.rs:107-108 at a0b5e6c) when it is created, from the stored checkpoint to that input's head at that moment, and every attempt and every rerun of the job reads exactly those ranges through the delta provider of engine: add a StreamingTable delta provider over scan_range #85, as ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:71-72).
The existing line "Tasks write only to staging; the job makes one atomic commit (ADR 0004)" now follows ADR 0006, which supersedes 0004 (docs/adr/0004-stage-then-atomic-commit.md:3): the job makes one commit per output table, the commit path that ADR 0006 leaves to this issue (docs/adr/0006-ordered-job-commits.md:39-40, :56-58), and once dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 is accepted JobOutcome::Committed lists those commits in place of its single snapshot (crates/model/src/lib.rs:134-137). Test: a job that writes two tables reports two commits.
Tasks stage files through the public stage and the runner commits them through commit_staged, as dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 specifies, both built from the private halves of append_if_head (the head check at crates/storage/src/lib.rs:198-202, write_files at :337 and commit_if_head at :525; the test at :852-855 already calls the last two on their own); the runner supplies the run id, the task (its stage and partition) and the attempt for the staged-file prefix of connectors: stage under a job prefix and run checks before the writer #82, and model commits never go through IcebergCommitExec, the INSERT INTO commit of the iceberg-datafusion crate pinned at Cargo.toml:31.
Once dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 is accepted, run_in_process retries only errors that Error::retry classifies as Retry::Task, where today it retries every error, Conflict included (crates/dist/src/lib.rs:102-109), so a lost commit race goes back to the job instead of rerunning the task, and storage errors keep iceberg's retryable flag, which storage() drops (crates/storage/src/lib.rs:322-324); attempt counting stays with dist: document and test RetryPolicy attempt counting #29. Tests: the Fake test stage (crates/dist/src/lib.rs:123) is not retried when it fails with Conflict, and is retried when it fails with an I/O Execution error.
The existing line "Each snapshot carries infostash.job-id; rerunning a finished job commits nothing" becomes a check per output table: before staging, the runner finds the output tables that already carry the job id, from the infostash.job-id table property and, when that names another job, from Warehouse::committed_by of connectors: skip tables that already carry the job id and use distinct test ids #81; it commits only to the other tables, checks again after a Conflict, and reports the skipped tables as already committed, so a rerun of a finished job no longer looks like NoChange, which stays the outcome when no input has new snapshots. Today the id is written (crates/connectors/src/lib.rs:127) but read only by a test (crates/connectors/tests/csv_ingest.rs:184), and the quarantine-then-target rerun of Row checks and quarantine tables #12 builds on this skip. Test: run a model job, append to an input, rerun under the same id: no new snapshot on its table, and the outcome names the first one.
If a commit of staged files fails on iceberg-rust's duplicate-file check while the table already carries the job id, the runner reports the table as already committed, not as the Error::Storage that commit_if_head returns today for any commit error that did not trip the head guard (crates/storage/src/lib.rs:547, :322-324), and turns the check off only for commits the job-id skip has already cleared. Test: commit under one job id, commit the same staged paths again, and get already committed.
Once docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 is accepted, a model run refuses to start, with a named error and no write, when its table's infostash.job-id property and the job id in the head snapshot's summary disagree, as after a rollback by another engine, although the TableState doc assumes that a value stored in the properties and the snapshot it belongs to always agree (crates/storage/src/lib.rs:51-52). Test: commit, roll the head back by hand through the catalog, and the rerun refuses without writing.
Test: a job that stops after its tasks staged files and before any commit, then a rerun under the same job id, leaves one new snapshot per output table, and every file the first run left behind carries that run's id and its attempt in its name, so storage: remove orphan data files #40 can attribute it.
Work that moved to new tickets:
dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74: the RFC that defines the per-table outcome, the Retry classification, the run id, the task context and the public stage and commit_staged, which this runner implements. It also settles whether Phase 0 writes the per-attempt staging manifest of the existing Done-when, which the design review of 2026-11-01 proposes to defer to the Phase 2 stage and commit split (S5-03), since the staged-file prefix and reachability from the head already identify staged files.
Suggested re-score: 13 instead of the 8 in #19, an estimate. Section 8.5 of the design review of 2026-11-01 scores three milestone M2 items that stay in this issue, the per-input checkpoint property at 3, the scheduler range step and NoChange at 5 and the model commit path through append_if_head at 5, which make 13; the StreamingTable delta provider it scores at 8 is now #85. It names no separate points for the pinned start time, the retry change in run_in_process or the duplicate-file, start-up and rerun checks above, which may take this issue past 13, and #19 splits anything above 13 before work starts. This is a suggestion only; #19 decides.
Source: design review of 2026-11-01, findings C10, C11, C12, C13, C15, C20, C23 and C24, contracts S5-01 to S5-06 and S5-12, design point S6-10, crash matrix rows 2, 4 and 10, and section 8.5 (D3, D4 and M2).
RFC 0002 (#102) settles the contracts this ticket builds on. One Done-when item changes once it is accepted: "Tasks write only to staging; the job makes one atomic commit (ADR 0004)" becomes one Warehouse::commit_staged per output table, side tables first and the target last, each carrying infostash.job-id (ADR 0006). The per-attempt staging manifest stays: Warehouse::stage writes it as {job}-{run}-{stage}-{partition}-{attempt}.staged.json next to the data files and returns the same record as StagedFiles; commit_staged deletes it after the commit.
Also from the RFC: Runner::run is async fn run(&self, job: &Job, run: RunId) -> JobOutcome; the runner skips tables that already carry the job id (reusing Warehouse::committed_by from #81) and checks Job::expected (head, schema id, partition spec id per output table) on the rest before any task runs; the Done-when item "returns JobOutcome::NoChange" moves to a new jobs::prepare function, since the runner only sees jobs that have work; a retryable storage error from commit_staged is retried in place up to max_task_attempts; run_in_process returns a TaskRun per task; and the test fixture's Job struct literal is replaced by Job::new, since Job becomes #[non_exhaustive].
ADR 0009 (PR #103) confirms the skip rule of ADR 0006 unchanged and names this ticket, with #81, as where the rerun-under-the-same-id rule is implemented. See the ADR for the abandon exception and the Diverged precheck.
Two further points from the ADR for this ticket: the runner precheck over every table the job writes fails with Diverged, never retried, when a table's infostash.job-id property and its head summary disagree; and every model commit sets infostash.job-id as a table property as well as in the summary, so the precheck has both to compare.
Activity
codecapo commentedon Oct 30, 2026
Spike findings (commit semantics and engine)
job-<id>-task-<n>-attempt-<m>name in the table's data location and a staging manifest (itsDataFiles via the publicserialize_data_file_to_json). The coordinator records one winning manifest per task and commits only those in onefast_append. Failed and duplicate attempts are never committed.INSERTcommits once per statement (IcebergCommitExec). To commit once per job, the runner runs IcebergWriteExec and commits the data files itself.NoChangemust skip the commit; an empty append is an error.Points: 13 -> 8.
feat(storage): stream rows added between two snapshots (#42)
docs: supersede ADR 0004 with ordered per-table job commits (#41)
feat(connectors): load CSV files into Raw exactly once (#43)
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. The runner keeps the input range step, the per-input checkpoint and the model commit path, takes on the job's pinned start time, the job-id skip and the rerun rules for model jobs, and now waits on two decisions, #74 and #75, and on the delta provider of #85.
Suggested additions to the Done-when:
Job.inputs(crates/model/src/lib.rs:107-108at a0b5e6c) when it is created, from the stored checkpoint to that input's head at that moment, and every attempt and every rerun of the job reads exactly those ranges through the delta provider of engine: add a StreamingTable delta provider over scan_range #85, as ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:71-72).now()in a@check(Row checks and quarantine tables #12) splits rows the same way each time, under the rule docs: revise RFC 0001 with the incremental rules (decision D2) #73 adds to RFC 0001.infostash.*table property of the model's table, set in the same commit as the rows (crates/storage/src/lib.rs:212-218), and goes into the snapshot summary as well (:209) only if docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 keeps model checkpoints mirrored, using the two maps of connectors: split the snapshot summary from the position #80.docs/adr/0004-stage-then-atomic-commit.md:3): the job makes one commit per output table, the commit path that ADR 0006 leaves to this issue (docs/adr/0006-ordered-job-commits.md:39-40,:56-58), and once dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 is acceptedJobOutcome::Committedlists those commits in place of its single snapshot (crates/model/src/lib.rs:134-137). Test: a job that writes two tables reports two commits.stageand the runner commits them throughcommit_staged, as dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 specifies, both built from the private halves ofappend_if_head(the head check atcrates/storage/src/lib.rs:198-202,write_filesat:337andcommit_if_headat:525; the test at:852-855already calls the last two on their own); the runner supplies the run id, the task (its stage and partition) and the attempt for the staged-file prefix of connectors: stage under a job prefix and run checks before the writer #82, and model commits never go throughIcebergCommitExec, theINSERT INTOcommit of theiceberg-datafusioncrate pinned atCargo.toml:31.run_in_processretries only errors thatError::retryclassifies asRetry::Task, where today it retries every error,Conflictincluded (crates/dist/src/lib.rs:102-109), so a lost commit race goes back to the job instead of rerunning the task, and storage errors keep iceberg's retryable flag, whichstorage()drops (crates/storage/src/lib.rs:322-324); attempt counting stays with dist: document and test RetryPolicy attempt counting #29. Tests: theFaketest stage (crates/dist/src/lib.rs:123) is not retried when it fails withConflict, and is retried when it fails with an I/OExecutionerror.infostash.job-id; rerunning a finished job commits nothing" becomes a check per output table: before staging, the runner finds the output tables that already carry the job id, from theinfostash.job-idtable property and, when that names another job, fromWarehouse::committed_byof connectors: skip tables that already carry the job id and use distinct test ids #81; it commits only to the other tables, checks again after aConflict, and reports the skipped tables as already committed, so a rerun of a finished job no longer looks likeNoChange, which stays the outcome when no input has new snapshots. Today the id is written (crates/connectors/src/lib.rs:127) but read only by a test (crates/connectors/tests/csv_ingest.rs:184), and the quarantine-then-target rerun of Row checks and quarantine tables #12 builds on this skip. Test: run a model job, append to an input, rerun under the same id: no new snapshot on its table, and the outcome names the first one.Error::Storagethatcommit_if_headreturns today for any commit error that did not trip the head guard (crates/storage/src/lib.rs:547,:322-324), and turns the check off only for commits the job-id skip has already cleared. Test: commit under one job id, commit the same staged paths again, and get already committed.infostash.job-idproperty and the job id in the head snapshot's summary disagree, as after a rollback by another engine, although theTableStatedoc assumes that a value stored in the properties and the snapshot it belongs to always agree (crates/storage/src/lib.rs:51-52). Test: commit, roll the head back by hand through the catalog, and the rerun refuses without writing.Work that moved to new tickets:
Retryclassification, the run id, the task context and the publicstageandcommit_staged, which this runner implements. It also settles whether Phase 0 writes the per-attempt staging manifest of the existing Done-when, which the design review of 2026-11-01 proposes to defer to the Phase 2 stage and commit split (S5-03), since the staged-file prefix and reachability from the head already identify staged files.append_if_headinto one map for the snapshot summary and one for the table properties.Warehouse::committed_bylookup that the runner reuses for model jobs.StreamingTabledelta provider overscan_rangethrough which each model input is read.HeadGuardcommit races toConflict, pins the commit retry count and corrects the guard's doc, which the stale-run item relies on.Suggested re-score: 13 instead of the 8 in #19, an estimate. Section 8.5 of the design review of 2026-11-01 scores three milestone M2 items that stay in this issue, the per-input checkpoint property at 3, the scheduler range step and
NoChangeat 5 and the model commit path throughappend_if_headat 5, which make 13; theStreamingTabledelta provider it scores at 8 is now #85. It names no separate points for the pinned start time, the retry change inrun_in_processor the duplicate-file, start-up and rerun checks above, which may take this issue past 13, and #19 splits anything above 13 before work starts. This is a suggestion only; #19 decides.Source: design review of 2026-11-01, findings C10, C11, C12, C13, C15, C20, C23 and C24, contracts S5-01 to S5-06 and S5-12, design point S6-10, crash matrix rows 2, 4 and 10, and section 8.5 (D3, D4 and M2).
codecapo commentedon Nov 10, 2026
RFC 0002 (#102) settles the contracts this ticket builds on. One Done-when item changes once it is accepted: "Tasks write only to staging; the job makes one atomic commit (ADR 0004)" becomes one
Warehouse::commit_stagedper output table, side tables first and the target last, each carryinginfostash.job-id(ADR 0006). The per-attempt staging manifest stays:Warehouse::stagewrites it as{job}-{run}-{stage}-{partition}-{attempt}.staged.jsonnext to the data files and returns the same record asStagedFiles;commit_stageddeletes it after the commit.Also from the RFC:
Runner::runisasync fn run(&self, job: &Job, run: RunId) -> JobOutcome; the runner skips tables that already carry the job id (reusingWarehouse::committed_byfrom #81) and checksJob::expected(head, schema id, partition spec id per output table) on the rest before any task runs; the Done-when item "returnsJobOutcome::NoChange" moves to a newjobs::preparefunction, since the runner only sees jobs that have work; a retryable storage error fromcommit_stagedis retried in place up tomax_task_attempts;run_in_processreturns aTaskRunper task; and the test fixture'sJobstruct literal is replaced byJob::new, sinceJobbecomes#[non_exhaustive].codecapo commentedon Nov 10, 2026
ADR 0009 (PR #103) confirms the skip rule of ADR 0006 unchanged and names this ticket, with #81, as where the rerun-under-the-same-id rule is implemented. See the ADR for the abandon exception and the
Divergedprecheck.codecapo commentedon Nov 10, 2026
Two further points from the ADR for this ticket: the runner precheck over every table the job writes fails with
Diverged, never retried, when a table'sinfostash.job-idproperty and its head summary disagree; and every model commit setsinfostash.job-idas a table property as well as in the summary, so the precheck has both to compare.docs: add ADR 0009 amending 0006 on position, reruns and expiry (#103)
docs: add RFC 0002 for the job and runner contracts (#102)