Skip to content

dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74

Closed

Description

@codecapo

Write one RFC that settles the execution and retry contracts of Job, JobOutcome, Error, the task interface and the runner before #12 and #13 build on them. At a0b5e6c these types cannot express what ADR 0006 decides: one commit per output table, each carrying the job id, in a fixed order, from files that each task attempt stages under its own name. JobOutcome::Committed holds one snapshot (crates/model/src/lib.rs:134-137), although a job with a quarantine table commits two tables in order (docs/adr/0006-ordered-job-commits.md:39-46). No code builds Failed { error, retryable } (crates/model/src/lib.rs:140-143), even though the Runner doc says a retryable failure lets the whole job be requeued (crates/jobs/src/lib.rs:16-18). run_in_process retries any error at once, a Conflict included (crates/dist/src/lib.rs:102-107), but a lost commit race has to re-prepare the whole job. run_task receives only a TaskId (crates/dist/src/lib.rs:46), so no task knows its attempt and staged files cannot carry it as docs/adr/0006-ordered-job-commits.md:33-36 requires. Nothing keeps iceberg's own retryable flag either: iceberg errors become Error::Storage(String) with only the message (the storage helper at crates/storage/src/lib.rs:322-324), and the one iceberg failure mapped to another variant is a commit stopped by the head guard, which becomes Conflict (crates/storage/src/lib.rs:539-546). So no caller can tell a transient storage failure from a permanent one. These are public API changes in model, jobs, dist and storage, not only in the Runner as docs/architecture.md:55-56 says, so they need an RFC (CONTRIBUTING.md:80-82, GOVERNANCE.md:28-29). The code lands in #12, #13 and #82, the QueryContext parameter in #95 and the doc fixes in #94, while #29 keeps the RetryPolicy attempt counting.

Where: docs/rfcs/ (a new RFC at the next free number, copied from docs/rfcs/0000-template.md), covering the public API of crates/model, crates/jobs, crates/dist and crates/storage

Done when

  • The RFC replaces the single snapshot in JobOutcome::Committed with a list of per-table commit records (table, snapshot, and whether a rerun found the table already committed under the job id) under a name other than iceberg's TableCommit, and replaces Failed { error, retryable } with a kind, a message and a Retry value.
  • The RFC defines Error::retry(&self) -> Retry for every error: Never for parse, plan, not-found, data and schema errors, Task for I/O errors and timeouts, and Job for Conflict, which re-prepares the whole job. It also keeps iceberg's retryable flag when storage errors are mapped, has run_in_process retry only Retry::Task, and says whether a retried task waits before its next attempt.
  • The RFC adds one Data variant to Error for bad input values, chosen from the root cause of a DataFusion error (a cast, parse, arithmetic overflow or divide-by-zero error), classified Never and naming the model that failed.
  • The RFC defines what a JobId identifies: one unit of work, fixed by the project version, the target, the pinned inputs or read position and the expected table heads, and stored before the job's first commit. It adds a separate run id that is never reused, since Job carries neither a run id nor an attempt today (crates/model/src/lib.rs:98-111). connectors: skip tables that already carry the job id and use distinct test ids #81 adds the storage lookup by job id and the connector skip, which need no RFC.
  • The RFC specifies public stage and commit_staged functions in crates/storage, built from today's private halves (the precheck at crates/storage/src/lib.rs:198-201, write_files at :203 and :337, commit_if_head at :219 and :525). Staged files are named {job}-{run}-{stage}-{partition}-{attempt} plus a random suffix, so that no staged path repeats across attempts or reruns (the job, task and attempt alone repeat on a rerun, because attempts start again at 1). stage returns a versioned staged-files type that records the attempt that wrote the files and the table, schema and partition spec they were written under.
  • The RFC decides the public run_task signature now, including whether it is async and whether Runner::run (crates/jobs/src/lib.rs:19) follows. run_task takes a TaskContext that carries the job, run, task and attempt and reserves slots for exchange inputs (with an InputLost error), cancellation and limits, and it returns a TaskOutput with the task's stats and staged files. The stats gain the quarantined-row and byte counts that JobMetrics reports (crates/model/src/lib.rs:121-130) but TaskStats lacks (crates/dist/src/lib.rs:49-54), and run_in_process returns each task's output with its attempt count instead of one summed TaskStats (crates/dist/src/lib.rs:91).
  • The RFC fixes the shape of the QueryContext parameter that cli: give infoctl query a QueryContext, a single-pass load and a memory limit #95 adds to the engine::query entry point of PR feat(cli): infoctl query runs read-only SQL against the warehouse #46: one local principal and an optional on_behalf_of slot, in a #[non_exhaustive] struct so that later fields need no signature change. The infoctl query RFC in cli: give infoctl query a QueryContext, a single-pass load and a memory limit #95 links here for it.
  • The RFC marks Job, JobOutcome, Error and TaskStats #[non_exhaustive] and says how a Job is built outside crates/model, since the attribute forbids the struct literal that the fixture at crates/jobs/src/lib.rs:49-57 uses. It also states that model runs never commit through datafusion-iceberg's IcebergCommitExec, a plain fast_append with no head check and no snapshot properties.
  • The RFC names the tests the implementation must meet: a Fake stage failing with Conflict is not retried by run_in_process, one failing with an I/O Execution error is, and a two-table job reports two commits.
  • The RFC answers or defers these open questions: who recovers a JobId after a crash when there is no queue (the CLI from the table property, or a job file in the project); one ExecutionStage per model or per model and input partition (Phase 0 can run each model as one stage with one task through run_in_process and an in-process Runner); whether Phase 0 writes the per-attempt staging manifests that ADR 0006 and In-process runner: stage, run tasks, commit once #13 ask for or leaves them to the stage and commit split of Phase 2, and then where they live and who deletes them; whether the runner or the compiler guarantees that a rerun produces the same output; what records a job as done once the snapshot summaries carrying its job id expire; whether run_task may return Conflict or only the commit step may; and which principal the jobs in each Pool run as.
  • The RFC is merged with status Accepted after at least one week open for comment and approval by two maintainers (GOVERNANCE.md:28-29).

Points: 3 (D3)

Depends on: nothing
Blocks: #12 (the per-table outcome and the retry classification), #13, #82 (the attempt in the staged-file name only)

Source: design review of 2026-11-01, section 8.5, D3 (findings C11, C12, C13, C14 and C31, and sections 5.3 to 5.6).

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    The RFC is up for review in #102 as docs/rfcs/0002-job-and-runner-contracts.md. Every Done-when item except the last is covered there; the last (Accepted after a week open and two approvals) closes this issue when the status line changes.

    Decisions worth reading first, because they deviate from or add to the text above:

    • Storage errors are classed by iceberg-rust error kind, not by its retryable flag alone. The library sets that flag only on commit conflicts, which become Conflict, and maps every I/O error to kind Unexpected without it, so keeping only the flag would make storage I/O Retry::Never. Unexpected is Task; DataInvalid, PreconditionFailed, FeatureUnsupported and the already-exists kinds are Never; an unlisted kind keeps the library's flag. Storage's own errors split the same way: local I/O is Task, validation is Never.
    • TaskStats stays in crates/dist and gains rows_quarantined, bytes_in and bytes_out.
    • StagedFiles is plain data in crates/model (version, table, attempt key, schema id, partition spec id, and each DataFile in iceberg-rust's JSON form). dist carries it in TaskOutput without depending on storage, and Warehouse::stage writes it beside the data files as the per-attempt staging manifest of ADR 0006; commit_staged deletes it once the commit lands.
    • run_task and Runner::run are async through async-trait. Runner::run takes the RunId.
    • A JobId is assigned (random u64, never reused). Job carries the expected head, schema id and partition spec id of every output table. A model job is recorded at .infostash/jobs/<job-id>.json after preparation and before the first commit; the record is deleted after the last commit, or after a failed run that committed to no table, so only jobs with partial commits survive. infoctl run reruns any such record for its target before preparing a new job. A connector load commits once and keeps no record. A job is abandoned for a new id when its rerun finds a changed table state (Conflict), when a run fails with Retry::Never after a partial commit, or when the recorded project version is no longer current; this is an exception to the same-id rule that docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 should state.
    • NoChange means no input past its checkpoint, decided by a new jobs::prepare function that infoctl run calls before the runner. A rerun that finds every table already carrying its id returns Committed with every record flagged already_committed and zero metrics.
    • A retried task starts again immediately; RetryPolicy is #[non_exhaustive] so a wait can be added. A retryable storage error from commit_staged is retried in place by the runner, since no task exists at that point.
    • Execution is classified Task; InputLost (reserved) is Job; a Retry::Stage class is left open.
  3. added a commit that references this issue on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions