Skip to content

docs: add RFC 0002 for the job and runner contracts - #102

Merged
codecapo merged 6 commits into
mainfrom
docs/rfc-0002-job-contracts
Nov 10, 2026
Merged

codecapo merged 6 commits into
mainfrom
docs/rfc-0002-job-contracts

Conversation

@codecapo

Copy link
Copy Markdown
Owner

What

Adds docs/rfcs/0002-job-and-runner-contracts.md, the RFC that #74 asks for. It fixes the contracts between a job, the tasks that run it and the commits it makes, across crates/model, crates/jobs, crates/dist and crates/storage, before #13 builds the in-process runner on them.

How

  • JobOutcome::Committed carries one CommitRecord per output table (table, snapshot, whether a rerun found it already committed under the job id), in commit order. Failed carries an ErrorKind, a message and a Retry class. NoChange is narrowed to "no input past its checkpoint, decided during preparation".
  • Error::retry() is total: Never for Parse, Plan, Data, NotFound and non-retryable Storage; Task for Execution and retryable Storage; Job for Conflict and the reserved InputLost. Storage errors are classified by iceberg-rust error kind (Unexpected, which covers I/O, is retryable; DataInvalid, PreconditionFailed and the like are not; an unlisted kind keeps the library's flag), because iceberg-rust sets its own retryable flag only on commit conflicts. Storage's own errors follow the same split: local I/O is retryable, validation is not. run_in_process retries only Retry::Task, immediately, with no wait in Phase 0.
  • A new Error::Data variant takes cast, parse, overflow and divide-by-zero failures from the root of a DataFusionError and names the model when there is one. Arrow schema and argument errors map to Plan; a Plan error for a missing table stays NotFound, as feat(cli): infoctl query runs read-only SQL against the warehouse #46 maps it.
  • A JobId identifies one unit of work fixed by project version, target, pinned inputs, the expected state (head, schema id, partition spec id) of every output table and the clock pin Job::started_at. It is assigned by the control plane and never reused. Preparation becomes a named step, jobs::prepare, which decides NoChange. A model job is recorded in .infostash/jobs/<job-id>.json after preparation and before the first commit; the RFC defines when the record is written, rewritten and deleted for every outcome. A job is abandoned for a new id when its rerun finds a changed table state (Conflict), when a run fails with Retry::Never after a partial commit, or when the recorded project version is no longer the current one, so no target can be left permanently blocked. infoctl run reruns an unfinished record under its id before preparing a new job. A connector load commits once and keeps no record. A separate RunId is never reused and appears in staged file names, TaskContext and logs.
  • Warehouse::stage and Warehouse::commit_staged are built from the private halves of append_if_head. Staged files are named {job}-{run}-{stage}-{partition}-{attempt}-{n:05}-{uuid}.parquet; attempts count from 1, as run_in_process counts today. StagedFiles is a versioned plain-data record in crates/model carrying the attempt key, table, schema id, partition spec id and each DataFile in iceberg-rust's JSON form; stage writes it beside the data files as the per-attempt staging manifest ADR 0006 requires, and commit_staged deletes it after the commit. commit_staged only appends; replace and merge are left to storage: add the replace_if_head commit #88.
  • run_task and Runner::run become async (via async-trait, already used by crates/storage). run_task takes a TaskContext (job, run, task, attempt, with reserved exchange input, cancellation and limits) and returns a TaskOutput (stats plus staged files). TaskStats stays in crates/dist and gains quarantined-row and byte counts. run_in_process returns a TaskRun per task with its attempt count.
  • QueryContext holds one local principal and an on_behalf_of slot, is #[non_exhaustive] and is built by QueryContext::local(). Every pool runs as Principal::Local in Phase 0.
  • Job, JobOutcome, Error, TaskStats and the new types are #[non_exhaustive], and every one that an outside crate builds has a constructor.
  • Model runs never commit through datafusion-iceberg's IcebergCommitExec, which uses a plain fast_append with no head check and no snapshot properties.
  • Seven of the ticket's open questions are answered in the RFC; three remain listed as open.

Things to know

Tests

Docs only. typos passes, no line outside tables or code exceeds 80 characters, and the attribution check passes. The RFC names the tests the implementing tickets must add: a Fake stage failing with Conflict is not retried, one failing with an I/O Execution error is, a two-table job reports two commits, a staging manifest reads back equal to its StagedFiles, a rerun against a changed target schema fails with Conflict before any task runs, and more.

Fix the contracts between a job, its tasks and its commits before the
in-process runner is built, so that the remote runners of later phases
share the same types.

The RFC replaces the single snapshot in JobOutcome::Committed with one
CommitRecord per output table and gives Failed a kind, a message and a
Retry class. Every Error variant gets a retry class: Never for parse,
plan, data and not-found errors, Task for I/O and other execution
errors, Job for Conflict. Storage errors are mapped from the iceberg
error kind, since the library's own retryable flag is set only on
commit conflicts. A Data variant holds bad input values mapped from the
root of a DataFusion error. A JobId identifies one unit of work fixed by
project version, target, pinned inputs, expected table state and the
clock pin, assigned by the control plane and recorded before the first
commit with a lifecycle defined for every outcome; a separate RunId is
never reused. Warehouse gains public stage and commit_staged halves of
append_if_head, staged files are named by job, run, stage, partition
and attempt, and each attempt writes the staging manifest ADR 0006
requires. run_task and Runner::run become async, with a TaskContext in
and a TaskOutput out, and run_in_process retries only Retry::Task and
returns each task's output with its attempt count. QueryContext holds
one local principal and an on_behalf_of slot. Job, JobOutcome, Error
and TaskStats become non_exhaustive.

Refs #74

Signed-off-by: Mayur Kale <hello@mayurkale.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Explain Change →

Summary

Summary by CodeRabbit

  • Documentation
    • Added a draft proposal outlining job, task, runner and storage contracts, including retry and commit flows, recovery rules, compatibility considerations and open questions. These behaviours are described as proposals and are not implemented by this change.

This PR adds an RFC that documents proposed job, task, runner and storage contracts. It covers identities, retries, staged files, commits, error mappings, job recovery, tests, compatibility changes and open questions. It does not implement these contracts.

Changes: Job and runner contracts

Layer / File(s) Summary
Job and outcome contracts
docs/rfcs/0002-job-and-runner-contracts.md
The RFC describes proposed job and run identities, retry classifications, per-table commit outcomes, task statistics, metrics and query principals.
Staging, commit and error mappings
docs/rfcs/0002-job-and-runner-contracts.md
The RFC defines staged-file records and separate stage and commit_staged operations. It also documents storage and DataFusion error mappings.
Job lifecycle and runner execution
docs/rfcs/0002-job-and-runner-contracts.md
The RFC specifies job preparation, record recovery and reruns, plus task and runner interfaces, retry rules and commit ordering.
Validation and compatibility
docs/rfcs/0002-job-and-runner-contracts.md
The RFC lists implementation test expectations, alternatives, proposed compatibility changes across seven crates, and answered and open questions.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Possibly related issues

  • codecapo/infostash#49: This issue requests the job, task, runner, storage and retry contracts that the RFC defines.
  • codecapo/infostash#48: This issue assigns the Data error contract to this RFC, which specifies that error.
  • codecapo/infostash#69: This issue requests architecture documentation for the job, task and commit contracts described by the RFC.
  • docs: fix README, architecture and ROADMAP statements the code does not back #94: This issue requests architecture updates for contracts specified here; those documentation edits are not part of this PR.
  • codecapo/infostash#47: This issue's replacement route must preserve commit constraints that the RFC defines.
  • connectors: stage under a job prefix and run checks before the writer #82: This issue requires staged-file names to include run, task and attempt identifiers, which the RFC specifies.
  • codecapo/infostash#57: This issue requires staged-file names to include run, task and attempt identifiers, which the RFC specifies.

Possibly related PRs

  • codecapo/infostash#101: RFC 0001 defines the pinned run-start time; this RFC applies that rule to job and runner execution.
  • codecapo/infostash#41: ADR 0006 establishes ordered, per-table commits under a job ID; this RFC specifies those outcomes and related contracts.
  • codecapo/infostash#20: ADR 0004 defines task staging followed by a job commit; this RFC specifies separate staging and commit operations.

Suggested labels: rfc, area/docs

Merge Risk: Moderate · up to 45376

This RFC fixes contracts that later implementation work will follow. Before it is accepted, it should require the commit guard to check the prepared table state, require directory-level durability for job records, and say the lock is held through deletion. It should also fix the decimal overflow error mapping.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly identifies the addition of RFC 0002 on job and runner contracts.
Description check Passed The description explains the RFC’s scope, key contracts, and relationship to follow-up work.
Linked Issues check Passed #74 asks for an RFC that settles the job, task, runner and storage contracts. The RFC specifies per-table commit records, retry and error rules, job and run identity, staging manifests and APIs, task …
Out of Scope Changes check Passed The whole-PR diff adds only docs/rfcs/0002-job-and-runner-contracts.md. Its contract definitions, recovery rules and test requirements directly support #74. No unrelated change is present.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Linked Issues check

Explanation

#74 asks for an RFC that settles the job, task, runner and storage contracts. The RFC specifies per-table commit records, retry and error rules, job and run identity, staging manifests and APIs, task and runner signatures, QueryContext, and public type construction. It also names implementation tests and answers or defers the design questions. The latest changes clarify job-record recovery, locking and retention, and do not leave a coding requirement unmet. Acceptance status and maintainer approvals are governance steps, not coding requirements.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.14%. Comparing base (7b39d58) to head (45376fc).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #102   +/-   ##
=======================================
  Coverage   99.14%   99.14%           
=======================================
  Files           8        8           
  Lines        1048     1048           
=======================================
  Hits         1039     1039           
  Misses          9        9           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Mayur Kale <hello@mayurkale.com>
Signed-off-by: Mayur Kale <hello@mayurkale.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 6e13aa1 and 0851b22.

Files selected for processing: 1

  • docs/rfcs/0002-job-and-runner-contracts.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@codecapo

Copy link
Copy Markdown
Owner Author

ADR 0009 (PR #103) adds a Diverged error for a table whose infostash.job-id property and head snapshot summary disagree. It is never retried, so the retry() table in this RFC needs a Diverged row under Never once the ADR lands. The ADR carries the abandon exception and the retention addition from this RFC, citing it as proposed.

Mark the Committed and Failed variants non_exhaustive. Compare head,
schema id and partition spec id on the load the catalog update makes,
so a metadata-only change fails the swap and maps to Conflict. Write the
job record durably before the first commit and rename every rewrite into
place. Keep the record with its Committed outcome instead of deleting
it, and let the maintenance sweep remove it. Take an exclusive lock on
the jobs directory for the whole run. Return Committed early from the
precheck when every table already carries the id.

Refs #74

Signed-off-by: Mayur Kale <hello@mayurkale.com>
@codecapo codecapo mentioned this pull request Nov 10, 2026
3 tasks

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0851b22 and 45376fc.

Files selected for processing: 1

  • docs/rfcs/0002-job-and-runner-contracts.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

pub async fn commit_staged(
&self,
table: &TableRef,
expected_head: Option<SnapshotId>,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Heavy lift

Pass the prepared table state to the commit guard.

If the partition spec changes after the runner precheck but before staging, stage records the new spec ID. The commit checks then accept that ID because it matches the current spec. The job commits despite its prepared TableState being stale.

Pass the full prepared TableState to commit_staged and check it at the commit boundary. Add a test for a metadata-only change before staging.

| `SQL` | `Parse` |
| `Plan` naming a table that does not exist (as #46 maps it) | `NotFound` |
| `Plan`, `SchemaError`, `Configuration`, `NotImplemented` | `Plan` |
| `ArrowError(SchemaError)`, `ArrowError(InvalidArgumentError)` | `Plan` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Classify decimal cast overflow as Data.

DataFusion 55.2.0 returns ArrowError::InvalidArgumentError for a decimal cast whose input exceeds the target precision. This row maps that data failure to Plan, so callers receive the wrong error kind. (raw.githubusercontent.com)

Distinguish cast value failures from schema or argument failures, and add a decimal overflow mapping test.

@codecapo
codecapo merged commit b3d6a08 into main Nov 10, 2026
19 checks passed
@codecapo
codecapo deleted the docs/rfc-0002-job-contracts branch November 10, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3)

1 participant