You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs: add RFC 0002 for the job and runner contracts - #102
Adds docs/rfcs/0002-job-and-runner-contracts.md, the RFC that #74 asks for. It fixes the contracts between a job, the tasks that run it and the commits it makes, across crates/model, crates/jobs, crates/dist and crates/storage, before #13 builds the in-process runner on them.
How
JobOutcome::Committed carries one CommitRecord per output table (table, snapshot, whether a rerun found it already committed under the job id), in commit order. Failed carries an ErrorKind, a message and a Retry class. NoChange is narrowed to "no input past its checkpoint, decided during preparation".
Error::retry() is total: Never for Parse, Plan, Data, NotFound and non-retryable Storage; Task for Execution and retryable Storage; Job for Conflict and the reserved InputLost. Storage errors are classified by iceberg-rust error kind (Unexpected, which covers I/O, is retryable; DataInvalid, PreconditionFailed and the like are not; an unlisted kind keeps the library's flag), because iceberg-rust sets its own retryable flag only on commit conflicts. Storage's own errors follow the same split: local I/O is retryable, validation is not. run_in_process retries only Retry::Task, immediately, with no wait in Phase 0.
A new Error::Data variant takes cast, parse, overflow and divide-by-zero failures from the root of a DataFusionError and names the model when there is one. Arrow schema and argument errors map to Plan; a Plan error for a missing table stays NotFound, as feat(cli): infoctl query runs read-only SQL against the warehouse #46 maps it.
A JobId identifies one unit of work fixed by project version, target, pinned inputs, the expected state (head, schema id, partition spec id) of every output table and the clock pin Job::started_at. It is assigned by the control plane and never reused. Preparation becomes a named step, jobs::prepare, which decides NoChange. A model job is recorded in .infostash/jobs/<job-id>.json after preparation and before the first commit; the RFC defines when the record is written, rewritten and deleted for every outcome. A job is abandoned for a new id when its rerun finds a changed table state (Conflict), when a run fails with Retry::Never after a partial commit, or when the recorded project version is no longer the current one, so no target can be left permanently blocked. infoctl run reruns an unfinished record under its id before preparing a new job. A connector load commits once and keeps no record. A separate RunId is never reused and appears in staged file names, TaskContext and logs.
Warehouse::stage and Warehouse::commit_staged are built from the private halves of append_if_head. Staged files are named {job}-{run}-{stage}-{partition}-{attempt}-{n:05}-{uuid}.parquet; attempts count from 1, as run_in_process counts today. StagedFiles is a versioned plain-data record in crates/model carrying the attempt key, table, schema id, partition spec id and each DataFile in iceberg-rust's JSON form; stage writes it beside the data files as the per-attempt staging manifest ADR 0006 requires, and commit_staged deletes it after the commit. commit_staged only appends; replace and merge are left to storage: add the replace_if_head commit #88.
run_task and Runner::run become async (via async-trait, already used by crates/storage). run_task takes a TaskContext (job, run, task, attempt, with reserved exchange input, cancellation and limits) and returns a TaskOutput (stats plus staged files). TaskStats stays in crates/dist and gains quarantined-row and byte counts. run_in_process returns a TaskRun per task with its attempt count.
QueryContext holds one local principal and an on_behalf_of slot, is #[non_exhaustive] and is built by QueryContext::local(). Every pool runs as Principal::Local in Phase 0.
Job, JobOutcome, Error, TaskStats and the new types are #[non_exhaustive], and every one that an outside crate builds has a constructor.
Model runs never commit through datafusion-iceberg's IcebergCommitExec, which uses a plain fast_append with no head check and no snapshot properties.
Seven of the ticket's open questions are answered in the RFC; three remain listed as open.
Deviations from the ticket text, each explained in the RFC: StagedFiles lives in crates/model as plain data rather than wrapping iceberg's DataFile, so dist does not depend on storage; InputLost is classed Retry::Job rather than a stage rerun, because Phase 0 has one stage, and docs: fix README, architecture and ROADMAP statements the code does not back #94 should note that the dist doc comment and docs/architecture.md describe the later runner.
crates/storage gains a direct serde_json dependency for the staging manifest, and crates/cli gains one for the job record.
Docs only. typos passes, no line outside tables or code exceeds 80 characters, and the attribution check passes. The RFC names the tests the implementing tickets must add: a Fake stage failing with Conflict is not retried, one failing with an I/O Execution error is, a two-table job reports two commits, a staging manifest reads back equal to its StagedFiles, a rerun against a changed target schema fails with Conflict before any task runs, and more.
Fix the contracts between a job, its tasks and its commits before the
in-process runner is built, so that the remote runners of later phases
share the same types.
The RFC replaces the single snapshot in JobOutcome::Committed with one
CommitRecord per output table and gives Failed a kind, a message and a
Retry class. Every Error variant gets a retry class: Never for parse,
plan, data and not-found errors, Task for I/O and other execution
errors, Job for Conflict. Storage errors are mapped from the iceberg
error kind, since the library's own retryable flag is set only on
commit conflicts. A Data variant holds bad input values mapped from the
root of a DataFusion error. A JobId identifies one unit of work fixed by
project version, target, pinned inputs, expected table state and the
clock pin, assigned by the control plane and recorded before the first
commit with a lifecycle defined for every outcome; a separate RunId is
never reused. Warehouse gains public stage and commit_staged halves of
append_if_head, staged files are named by job, run, stage, partition
and attempt, and each attempt writes the staging manifest ADR 0006
requires. run_task and Runner::run become async, with a TaskContext in
and a TaskOutput out, and run_in_process retries only Retry::Task and
returns each task's output with its attempt count. QueryContext holds
one local principal and an on_behalf_of slot. Job, JobOutcome, Error
and TaskStats become non_exhaustive.
Refs #74
Signed-off-by: Mayur Kale <hello@mayurkale.com>
Added a draft proposal outlining job, task, runner and storage contracts, including retry and commit flows, recovery rules, compatibility considerations and open questions. These behaviours are described as proposals and are not implemented by this change.
This PR adds an RFC that documents proposed job, task, runner and storage contracts. It covers identities, retries, staged files, commits, error mappings, job recovery, tests, compatibility changes and open questions. It does not implement these contracts.
Changes: Job and runner contracts
Layer / File(s)
Summary
Job and outcome contracts docs/rfcs/0002-job-and-runner-contracts.md
The RFC describes proposed job and run identities, retry classifications, per-table commit outcomes, task statistics, metrics and query principals.
Staging, commit and error mappings docs/rfcs/0002-job-and-runner-contracts.md
The RFC defines staged-file records and separate stage and commit_staged operations. It also documents storage and DataFusion error mappings.
Job lifecycle and runner execution docs/rfcs/0002-job-and-runner-contracts.md
The RFC specifies job preparation, record recovery and reruns, plus task and runner interfaces, retry rules and commit ordering.
Validation and compatibility docs/rfcs/0002-job-and-runner-contracts.md
The RFC lists implementation test expectations, alternatives, proposed compatibility changes across seven crates, and answered and open questions.
codecapo/infostash#57: This issue requires staged-file names to include run, task and attempt identifiers, which the RFC specifies.
Possibly related PRs
codecapo/infostash#101: RFC 0001 defines the pinned run-start time; this RFC applies that rule to job and runner execution.
codecapo/infostash#41: ADR 0006 establishes ordered, per-table commits under a job ID; this RFC specifies those outcomes and related contracts.
codecapo/infostash#20: ADR 0004 defines task staging followed by a job commit; this RFC specifies separate staging and commit operations.
Suggested labels:rfc, area/docs
Merge Risk: Moderate · up to 45376
This RFC fixes contracts that later implementation work will follow. Before it is accepted, it should require the commit guard to check the prepared table state, require directory-level durability for job records, and say the lock is held through deletion. It should also fix the decimal overflow error mapping.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title clearly identifies the addition of RFC 0002 on job and runner contracts.
Description check
The description explains the RFC’s scope, key contracts, and relationship to follow-up work.
Linked Issues check
#74 asks for an RFC that settles the job, task, runner and storage contracts. The RFC specifies per-table commit records, retry and error rules, job and run identity, staging manifests and APIs, task …
Out of Scope Changes check
The whole-PR diff adds only docs/rfcs/0002-job-and-runner-contracts.md. Its contract definitions, recovery rules and test requirements directly support #74. No unrelated change is present.
Docstring Coverage
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Linked Issues check
Explanation
#74 asks for an RFC that settles the job, task, runner and storage contracts. The RFC specifies per-table commit records, retry and error rules, job and run identity, staging manifests and APIs, task and runner signatures, QueryContext, and public type construction. It also names implementation tests and answers or defers the design questions. The latest changes clarify job-record recovery, locking and retention, and do not leave a coding requirement unmet. Acceptance status and maintainer approvals are governance steps, not coding requirements.
Full details: Docstring Coverage
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Reviewing files that changed from the base of the PR and between 6e13aa1 and 0851b22.
Files selected for processing: 1
docs/rfcs/0002-job-and-runner-contracts.md
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
ADR 0009 (PR #103) adds a Diverged error for a table whose infostash.job-id property and head snapshot summary disagree. It is never retried, so the retry() table in this RFC needs a Diverged row under Never once the ADR lands. The ADR carries the abandon exception and the retention addition from this RFC, citing it as proposed.
Mark the Committed and Failed variants non_exhaustive. Compare head,
schema id and partition spec id on the load the catalog update makes,
so a metadata-only change fails the swap and maps to Conflict. Write the
job record durably before the first commit and rename every rewrite into
place. Keep the record with its Committed outcome instead of deleting
it, and let the maintenance sweep remove it. Take an exclusive lock on
the jobs directory for the whole run. Return Committed early from the
precheck when every table already carries the id.
Refs #74
Signed-off-by: Mayur Kale <hello@mayurkale.com>
Reviewing files that changed from the base of the PR and between 0851b22 and 45376fc.
Files selected for processing: 1
docs/rfcs/0002-job-and-runner-contracts.md
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Data Integrity & Integration · Heavy lift
Pass the prepared table state to the commit guard.
If the partition spec changes after the runner precheck but before staging, stage records the new spec ID. The commit checks then accept that ID because it matches the current spec. The job commits despite its prepared TableState being stale.
Pass the full prepared TableState to commit_staged and check it at the commit boundary. Add a test for a metadata-only change before staging.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
Classify decimal cast overflow as Data.
DataFusion 55.2.0 returns ArrowError::InvalidArgumentError for a decimal cast whose input exceeds the target precision. This row maps that data failure to Plan, so callers receive the wrong error kind. (raw.githubusercontent.com)
Distinguish cast value failures from schema or argument failures, and add a decimal overflow mapping test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
docs/rfcs/0002-job-and-runner-contracts.md, the RFC that #74 asks for. It fixes the contracts between a job, the tasks that run it and the commits it makes, acrosscrates/model,crates/jobs,crates/distandcrates/storage, before #13 builds the in-process runner on them.How
JobOutcome::Committedcarries oneCommitRecordper output table (table, snapshot, whether a rerun found it already committed under the job id), in commit order.Failedcarries anErrorKind, a message and aRetryclass.NoChangeis narrowed to "no input past its checkpoint, decided during preparation".Error::retry()is total:NeverforParse,Plan,Data,NotFoundand non-retryableStorage;TaskforExecutionand retryableStorage;JobforConflictand the reservedInputLost. Storage errors are classified by iceberg-rust error kind (Unexpected, which covers I/O, is retryable;DataInvalid,PreconditionFailedand the like are not; an unlisted kind keeps the library's flag), because iceberg-rust sets its own retryable flag only on commit conflicts. Storage's own errors follow the same split: local I/O is retryable, validation is not.run_in_processretries onlyRetry::Task, immediately, with no wait in Phase 0.Error::Datavariant takes cast, parse, overflow and divide-by-zero failures from the root of aDataFusionErrorand names the model when there is one. Arrow schema and argument errors map toPlan; aPlanerror for a missing table staysNotFound, as feat(cli): infoctl query runs read-only SQL against the warehouse #46 maps it.JobIdidentifies one unit of work fixed by project version, target, pinned inputs, the expected state (head, schema id, partition spec id) of every output table and the clock pinJob::started_at. It is assigned by the control plane and never reused. Preparation becomes a named step,jobs::prepare, which decidesNoChange. A model job is recorded in.infostash/jobs/<job-id>.jsonafter preparation and before the first commit; the RFC defines when the record is written, rewritten and deleted for every outcome. A job is abandoned for a new id when its rerun finds a changed table state (Conflict), when a run fails withRetry::Neverafter a partial commit, or when the recorded project version is no longer the current one, so no target can be left permanently blocked.infoctl runreruns an unfinished record under its id before preparing a new job. A connector load commits once and keeps no record. A separateRunIdis never reused and appears in staged file names,TaskContextand logs.Warehouse::stageandWarehouse::commit_stagedare built from the private halves ofappend_if_head. Staged files are named{job}-{run}-{stage}-{partition}-{attempt}-{n:05}-{uuid}.parquet; attempts count from 1, asrun_in_processcounts today.StagedFilesis a versioned plain-data record incrates/modelcarrying the attempt key, table, schema id, partition spec id and eachDataFilein iceberg-rust's JSON form;stagewrites it beside the data files as the per-attempt staging manifest ADR 0006 requires, andcommit_stageddeletes it after the commit.commit_stagedonly appends; replace and merge are left to storage: add the replace_if_head commit #88.run_taskandRunner::runbecome async (viaasync-trait, already used bycrates/storage).run_tasktakes aTaskContext(job, run, task, attempt, with reserved exchange input, cancellation and limits) and returns aTaskOutput(stats plus staged files).TaskStatsstays incrates/distand gains quarantined-row and byte counts.run_in_processreturns aTaskRunper task with its attempt count.QueryContextholds onelocalprincipal and anon_behalf_ofslot, is#[non_exhaustive]and is built byQueryContext::local(). Every pool runs asPrincipal::Localin Phase 0.Job,JobOutcome,Error,TaskStatsand the new types are#[non_exhaustive], and every one that an outside crate builds has a constructor.IcebergCommitExec, which uses a plainfast_appendwith no head check and no snapshot properties.Things to know
main, and states the clock pin and the compiler's determinism obligation itself rather than relying on the RFC 0001 revision in docs: revise RFC 0001 with the incremental rules #101. It refers to the rerun-after-partial-commit rule by its ticket, docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75, because that ADR does not exist yet, and names two things docs: add an ADR amending 0006 on position, reruns and expiry (decision D4) #75 should carry: the exception that a job made stale by aConflictis abandoned for a new id rather than rerun, and the retention rule that snapshot expiry must keep every snapshot a job with a live record committed.StagedFileslives incrates/modelas plain data rather than wrapping iceberg'sDataFile, so dist does not depend on storage;InputLostis classedRetry::Jobrather than a stage rerun, because Phase 0 has one stage, and docs: fix README, architecture and ROADMAP statements the code does not back #94 should note that the dist doc comment anddocs/architecture.mddescribe the later runner.crates/storagegains a directserde_jsondependency for the staging manifest, andcrates/cligains one for the job record.Tests
Docs only.
typospasses, no line outside tables or code exceeds 80 characters, and the attribution check passes. The RFC names the tests the implementing tickets must add: aFakestage failing withConflictis not retried, one failing with an I/OExecutionerror is, a two-table job reports two commits, a staging manifest reads back equal to itsStagedFiles, a rerun against a changed target schema fails withConflictbefore any task runs, and more.