Repository navigation
Description
Write one RFC that settles the execution and retry contracts of Job, JobOutcome, Error, the task interface and the runner before #12 and #13 build on them. At a0b5e6c these types cannot express what ADR 0006 decides: one commit per output table, each carrying the job id, in a fixed order, from files that each task attempt stages under its own name. JobOutcome::Committed holds one snapshot (crates/model/src/lib.rs:134-137), although a job with a quarantine table commits two tables in order (docs/adr/0006-ordered-job-commits.md:39-46). No code builds Failed { error, retryable } (crates/model/src/lib.rs:140-143), even though the Runner doc says a retryable failure lets the whole job be requeued (crates/jobs/src/lib.rs:16-18). run_in_process retries any error at once, a Conflict included (crates/dist/src/lib.rs:102-107), but a lost commit race has to re-prepare the whole job. run_task receives only a TaskId (crates/dist/src/lib.rs:46), so no task knows its attempt and staged files cannot carry it as docs/adr/0006-ordered-job-commits.md:33-36 requires. Nothing keeps iceberg's own retryable flag either: iceberg errors become Error::Storage(String) with only the message (the storage helper at crates/storage/src/lib.rs:322-324), and the one iceberg failure mapped to another variant is a commit stopped by the head guard, which becomes Conflict (crates/storage/src/lib.rs:539-546). So no caller can tell a transient storage failure from a permanent one. These are public API changes in model, jobs, dist and storage, not only in the Runner as docs/architecture.md:55-56 says, so they need an RFC (CONTRIBUTING.md:80-82, GOVERNANCE.md:28-29). The code lands in #12, #13 and #82, the QueryContext parameter in #95 and the doc fixes in #94, while #29 keeps the RetryPolicy attempt counting.
Where: docs/rfcs/ (a new RFC at the next free number, copied from docs/rfcs/0000-template.md), covering the public API of crates/model, crates/jobs, crates/dist and crates/storage
Done when
- The RFC replaces the single snapshot in
JobOutcome::Committedwith a list of per-table commit records (table, snapshot, and whether a rerun found the table already committed under the job id) under a name other than iceberg'sTableCommit, and replacesFailed { error, retryable }with a kind, a message and aRetryvalue. - The RFC defines
Error::retry(&self) -> Retryfor every error:Neverfor parse, plan, not-found, data and schema errors,Taskfor I/O errors and timeouts, andJobforConflict, which re-prepares the whole job. It also keeps iceberg's retryable flag when storage errors are mapped, hasrun_in_processretry onlyRetry::Task, and says whether a retried task waits before its next attempt. - The RFC adds one
Datavariant toErrorfor bad input values, chosen from the root cause of a DataFusion error (a cast, parse, arithmetic overflow or divide-by-zero error), classifiedNeverand naming the model that failed. - The RFC defines what a
JobIdidentifies: one unit of work, fixed by the project version, the target, the pinned inputs or read position and the expected table heads, and stored before the job's first commit. It adds a separate run id that is never reused, sinceJobcarries neither a run id nor an attempt today (crates/model/src/lib.rs:98-111). connectors: skip tables that already carry the job id and use distinct test ids #81 adds the storage lookup by job id and the connector skip, which need no RFC. - The RFC specifies public
stageandcommit_stagedfunctions incrates/storage, built from today's private halves (the precheck atcrates/storage/src/lib.rs:198-201,write_filesat:203and:337,commit_if_headat:219and:525). Staged files are named{job}-{run}-{stage}-{partition}-{attempt}plus a random suffix, so that no staged path repeats across attempts or reruns (the job, task and attempt alone repeat on a rerun, because attempts start again at 1).stagereturns a versioned staged-files type that records the attempt that wrote the files and the table, schema and partition spec they were written under. - The RFC decides the public
run_tasksignature now, including whether it is async and whetherRunner::run(crates/jobs/src/lib.rs:19) follows.run_tasktakes aTaskContextthat carries the job, run, task and attempt and reserves slots for exchange inputs (with anInputLosterror), cancellation and limits, and it returns aTaskOutputwith the task's stats and staged files. The stats gain the quarantined-row and byte counts thatJobMetricsreports (crates/model/src/lib.rs:121-130) butTaskStatslacks (crates/dist/src/lib.rs:49-54), andrun_in_processreturns each task's output with its attempt count instead of one summedTaskStats(crates/dist/src/lib.rs:91). - The RFC fixes the shape of the
QueryContextparameter that cli: give infoctl query a QueryContext, a single-pass load and a memory limit #95 adds to theengine::queryentry point of PR feat(cli): infoctl query runs read-only SQL against the warehouse #46: onelocalprincipal and an optionalon_behalf_ofslot, in a#[non_exhaustive]struct so that later fields need no signature change. Theinfoctl queryRFC in cli: give infoctl query a QueryContext, a single-pass load and a memory limit #95 links here for it. - The RFC marks
Job,JobOutcome,ErrorandTaskStats#[non_exhaustive]and says how aJobis built outsidecrates/model, since the attribute forbids the struct literal that the fixture atcrates/jobs/src/lib.rs:49-57uses. It also states that model runs never commit through datafusion-iceberg'sIcebergCommitExec, a plainfast_appendwith no head check and no snapshot properties. - The RFC names the tests the implementation must meet: a
Fakestage failing withConflictis not retried byrun_in_process, one failing with an I/OExecutionerror is, and a two-table job reports two commits. - The RFC answers or defers these open questions: who recovers a
JobIdafter a crash when there is no queue (the CLI from the table property, or a job file in the project); oneExecutionStageper model or per model and input partition (Phase 0 can run each model as one stage with one task throughrun_in_processand an in-processRunner); whether Phase 0 writes the per-attempt staging manifests that ADR 0006 and In-process runner: stage, run tasks, commit once #13 ask for or leaves them to the stage and commit split of Phase 2, and then where they live and who deletes them; whether the runner or the compiler guarantees that a rerun produces the same output; what records a job as done once the snapshot summaries carrying its job id expire; whetherrun_taskmay returnConflictor only the commit step may; and which principal the jobs in eachPoolrun as. - The RFC is merged with status Accepted after at least one week open for comment and approval by two maintainers (
GOVERNANCE.md:28-29).
Points: 3 (D3)
Depends on: nothing
Blocks: #12 (the per-table outcome and the retry classification), #13, #82 (the attempt in the staged-file name only)
Source: design review of 2026-11-01, section 8.5, D3 (findings C11, C12, C13, C14 and C31, and sections 5.3 to 5.6).
Activity
codecapo commentedon Nov 10, 2026
The RFC is up for review in #102 as
docs/rfcs/0002-job-and-runner-contracts.md. Every Done-when item except the last is covered there; the last (Accepted after a week open and two approvals) closes this issue when the status line changes.Decisions worth reading first, because they deviate from or add to the text above:
Conflict, and maps every I/O error to kindUnexpectedwithout it, so keeping only the flag would make storage I/ORetry::Never.UnexpectedisTask;DataInvalid,PreconditionFailed,FeatureUnsupportedand the already-exists kinds areNever; an unlisted kind keeps the library's flag. Storage's own errors split the same way: local I/O isTask, validation isNever.TaskStatsstays incrates/distand gainsrows_quarantined,bytes_inandbytes_out.StagedFilesis plain data incrates/model(version, table, attempt key, schema id, partition spec id, and eachDataFilein iceberg-rust's JSON form). dist carries it inTaskOutputwithout depending on storage, andWarehouse::stagewrites it beside the data files as the per-attempt staging manifest of ADR 0006;commit_stageddeletes it once the commit lands.run_taskandRunner::runare async throughasync-trait.Runner::runtakes theRunId.JobIdis assigned (randomu64, never reused).Jobcarries the expected head, schema id and partition spec id of every output table. A model job is recorded at.infostash/jobs/<job-id>.jsonafter preparation and before the first commit; the record is deleted after the last commit, or after a failed run that committed to no table, so only jobs with partial commits survive.infoctl runreruns any such record for its target before preparing a new job. A connector load commits once and keeps no record. A job is abandoned for a new id when its rerun finds a changed table state (Conflict), when a run fails withRetry::Neverafter a partial commit, or when the recorded project version is no longer current; this is an exception to the same-id rule that docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 should state.NoChangemeans no input past its checkpoint, decided by a newjobs::preparefunction thatinfoctl runcalls before the runner. A rerun that finds every table already carrying its id returnsCommittedwith every record flaggedalready_committedand zero metrics.RetryPolicyis#[non_exhaustive]so a wait can be added. A retryable storage error fromcommit_stagedis retried in place by the runner, since no task exists at that point.Executionis classifiedTask;InputLost(reserved) isJob; aRetry::Stageclass is left open.docs: answer the review on RFC 0002