You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs: revise RFC 0001 with the incremental rules - #101
Revises RFC 0001 with the incremental rules that the Phase 0
implementation tickets (#7, #12, #35, #36, #78, #84, #86, #87, #89, #90)
depend on. Closes#73.
How
Directives: the header is the run of blank and -- comment lines at
the top of the file; a -- @key: value line after the SQL starts is an
ordinary comment; @check repeats, may carry a name, and failing any
check quarantines the row.
Incremental eligibility is an allow-list over the unoptimised
DataFusion logical plan: TableScan of a declared append-only input, Filter, Projection, SubqueryAlias, Unnest and UNION ALL.
Other nodes are a compile error that offers @full_refresh: true.
Only Immutable functions may appear in incremental bodies, Stable
ones in checks and full-refresh models, Volatile ones nowhere.
Joins and aggregates are not incremental; the daily_revenue example
declares @full_refresh: true.
Raw columns are nullable text. Clean models type them with try_cast,
a strict cast over a Raw column is a compile error, failed casts
quarantine under an implicit cast:<col> check that runs on every input
row before the where, and a check that returns NULL fails the row.
Ordering: latest per key is the highest snapshot sequence number, then _source_file, then _source_row; ties within one snapshot fail the
run. Clock functions return the job start time. Quarantine rows carry _job_id.
A model version hash is stored under the infostash. property prefix.
A mismatch refuses an incremental run; a @full_refresh run skips the
comparison and commits the new hash.
The CSV position records the path pattern.
Records the ADR 0008 fallback (a model rebuilt in full without @full_refresh) and the interim compile-time rule that an incremental
model may not read a table declaring @unique_key or @full_refresh
until the plan-time check in engine: fall back to a full refresh at plan time and fix files_added_by #89.
@partition_by is removed while the RFC is a Draft. Tables are created
without a partition spec and the replace commit covers unpartitioned
tables only. A later revision adds partitioning back. The first
Done-when item of Partition tables with @partition_by #36 waits on that revision.
The version hash excludes @description and @full_refresh, with the
reasons stated in the RFC.
Open question 3 (CSV schema) is answered: every column is nullable text
and typing happens in Clean. This overlaps the discussion in RFC 0001: project file format #5.
Checks, written and implicit, are restricted to row-wise models over
one input in Phase 0, which also limits @unique_key to such models.
New open question 6 asks where checks sit on joins, aggregates and
unnests.
A check that reads the clock judges each row by the start time of the
run that first read it. This is the one stated exception to the rule
that incremental and single-pass rebuilds give the same table.
Only the RFC changes. No code, CI or check is touched.
Tests
Docs only. typos passes, no em or en dashes, lines wrap under 80
outside tables, and the attribution check passes on this text.
Write down the rules the Phase 0 implementation tickets depend on, so
they can be built and tested from one text.
Directives: the header is the run of blank and comment lines at the top
of a model file, a directive after the SQL starts is an ordinary
comment, @check repeats and may carry a name, and unknown or malformed
directives fail compilation.
Incremental eligibility: an allow-list over the unoptimised DataFusion
logical plan (TableScan of a declared append-only input, Filter,
Projection, SubqueryAlias, Unnest and UNION ALL). Anything else is a
compile error that offers @full_refresh: true. Only Immutable functions
may appear in incremental bodies, Stable ones in checks and full-refresh
models, Volatile ones nowhere. Joins and aggregates are therefore not
incremental, and the daily_revenue example declares @full_refresh.
Typing: Raw columns are nullable text, Clean models type them with
try_cast, a strict cast over a Raw column is a compile error, failed
casts quarantine under an implicit cast:<col> check, and a check that
returns NULL fails the row.
Ordering: latest per key means the highest snapshot sequence number,
then _source_file, then _source_row; ties within one snapshot fail the
run. Clock functions return the job start time. Quarantine rows carry
_job_id. A model version hash is stored under the infostash. property
prefix; a mismatch refuses an incremental run, and a full refresh
skips the comparison and commits the new hash. The CSV position
records the path pattern.
@partition_by is removed while the RFC is a Draft: tables are created
without a partition spec and the replace commit covers unpartitioned
tables only. Open questions 2, 3 and 4 are answered and a new question
6 asks where checks sit on models that are not row-wise.
Closes#73
Signed-off-by: Mayur Kale <hello@mayurkale.com>
Expanded the project-format specification with details on CSV typing, metadata, append-only loading, checkpoints, model directives, checks, and incremental processing.
Updated examples and clarified when incremental plans require a full refresh or rebuild, including row ordering, pinned time, and model-version tracking.
Removed @partition_by from the directive reference, documented implementation gaps and compatibility constraints, and recorded resolved and remaining open questions.
RFC 0001 adds rules for CSV sources, model directives and checks, incremental-plan eligibility, refresh and rebuild behaviour, row ordering, quarantine and model versions. It also updates examples and records implementation gaps, compatibility constraints and open questions.
Changes: Project-file contract
Layer / File(s)
Summary
Source and model rules docs/rfcs/0001-project-files.md
The RFC describes CSV Raw-column typing, metadata and pattern-bound checkpoints. It specifies directive parsing, named and implicit checks, quarantine conditions, and restrictions on strict casts. Updated examples use named checks and try_cast.
The RFC defines eligible incremental plans and function restrictions. It describes full refresh and rebuild conditions, row ordering and tie failures, quarantine contents, and model-version hash behaviour. The daily-revenue example uses @full_refresh: true.
Execution and compatibility docs/rfcs/0001-project-files.md
The RFC records alternatives, compatibility constraints and implementation gaps. It marks several questions as answered and adds an open question about checks on non-row-wise plans.
codecapo/infostash#43: Implements CSV loading and file-position tracking that the RFC specifies.
codecapo/infostash#41: Defines ordered table commits and read-position storage used by the RFC's checkpoint rules.
Suggested labels:rfc, area/docs
Merge Risk: Moderate · up to f151d
This is an RFC-only change. Before merging, specify how a forced rebuild replaces quarantine, and make the clock invariant consistent with the rebuild rule that the Time section now states.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title clearly identifies the RFC revision and its incremental-processing rules.
Description check
The description explains the RFC changes and their scope, so it relates directly to the changeset.
Linked Issues check
[#73] The only direct link is #73. Its deliverable is RFC text in docs/rfcs/0001-project-files.md, so its requirements are non-coding. No coding requirements apply. The text at head meets the verified…
Out of Scope Changes check
The PR changes only docs/rfcs/0001-project-files.md, as the PR description and raw summary state. No code, CI or check files are changed. The additions map to #73 Done-when items and to the findings i…
Docstring Coverage
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Linked Issues check
Explanation
[#73] The only direct link is #73. Its deliverable is RFC text in docs/rfcs/0001-project-files.md, so its requirements are non-coding. No coding requirements apply. The text at head meets the verified Done-when items:
Directives: header run of blank and -- lines, -- @key: value after SQL starts is ignored, repeated @check forms a list, any failing check quarantines the row, and a check may carry a name (open question 2 answered).
Eligibility: allow-list over the unoptimised plan (TableScan of a declared table, Filter, Projection, SubqueryAlias, Unnest, Union from union all, no subquery expressions). Other nodes are a compile error that offers @full_refresh: true. Immutable functions only in incremental bodies, Stable in @check and @full_refresh models, Volatile nowhere. The compiler performs the check before job creation.
Joins and all aggregates are non-incremental. daily_revenue declares @full_refresh: true. The 100-then-20 example gives (2026-10-29, 120).
@partition_by is removed from the directive table and the day line. Unknown keys are an error, so the directive is rejected.
clean/orders.sql uses try_cast. Strict casts over Raw columns are a compile error naming the column. Implicit cast:<col> checks run before the where and before written checks. The NULL rule is stated.
CSV position stores the path pattern. A pattern change fails the run. Earlier positions without the field are read as having none.
_source_row is defined as a 1-based row position.
Clock functions return the job start time, stored with the job id and reused on rerun.
Not verified directly: the file read truncated before the Ordering, version-hash, Compatibility and open-question sections. The ordering, _job_id and hash claims rest on the PR description and the raw summary, not on text I read. Also, the hash excludes @description and @full_refresh. Issue C16 names 'directives' generally. The PR gives a stated reason, which I could not check. Forward references to ADR 0007, ADR 0008 and ADR 0009 (#103) do not resolve at head. The PR states it merges after #100 for ADR 0007 and 0008.
Full details: Out of Scope Changes check
Explanation
The PR changes only docs/rfcs/0001-project-files.md, as the PR description and raw summary state. No code, CI or check files are changed. The additions map to #73 Done-when items and to the findings it cites: the eligibility allow-list, per-mode output rule, cast and quarantine rules, ordering and time rules, the @full_refresh and ADR 0008 fallback text (related to #90 and #84), CSV position pattern (#78), and open question 6 on checks over non-row-wise models. The CSV column description (_source_file, _loaded_at, _source_row) and the raw-text statement describe the Raw schema that the checks rely on. They are supporting RFC content, not unrelated changes. No unrelated change was demonstrated.
Full details: Docstring Coverage
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
A try_cast of a Raw column in the where clause yields null on a bad
value, the comparison yields null and the filter drops the row, so a
check that runs after the filter never sees it. Implicit cast checks
now run on every input row before the where, and two implicit checks
that would share a name are a compile error that asks for an alias.
Signed-off-by: Mayur Kale <hello@mayurkale.com>
Reviewing files that changed from the base of the PR and between 6e13aa1 and 550a61d.
Files selected for processing: 1
docs/rfcs/0001-project-files.md
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
Replace the quarantine table in its own commit before the target under
the same job id, with the reader rule covering the window between the
two commits. State that a rebuild judges rows by its own start time, as
the one way a rebuilt table can differ. Refuse a table that has rows but
no stored model version hash instead of running it.
Refs #73
Signed-off-by: Mayur Kale <hello@mayurkale.com>
Reviewing files that changed from the base of the PR and between 550a61d and f151dbd.
Files selected for processing: 1
docs/rfcs/0001-project-files.md
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
Define quarantine replacement for forced rebuilds.
A forced rebuild rechecks previously processed rows. If it appends quarantine, it can duplicate earlier failures or retain rows that now pass a clock check. Specify that it replaces quarantine as well as the target.
Suggested fix
cause ([ADR 0008](../adr/0008-incremental-readers-of-replaced-tables.md)).
+The quarantine table is replaced as in a full refresh, under the same job id+and before the target.
## What
Adds `docs/rfcs/0002-job-and-runner-contracts.md`, the RFC that #74 asks
for. It fixes the contracts between a job, the tasks that run it and the
commits it makes, across `crates/model`, `crates/jobs`, `crates/dist`
and `crates/storage`, before #13 builds the in-process runner on them.
## How
- `JobOutcome::Committed` carries one `CommitRecord` per output table
(table, snapshot, whether a rerun found it already committed under the
job id), in commit order. `Failed` carries an `ErrorKind`, a message and
a `Retry` class. `NoChange` is narrowed to "no input past its
checkpoint, decided during preparation".
- `Error::retry()` is total: `Never` for `Parse`, `Plan`, `Data`,
`NotFound` and non-retryable `Storage`; `Task` for `Execution` and
retryable `Storage`; `Job` for `Conflict` and the reserved `InputLost`.
Storage errors are classified by iceberg-rust error kind (`Unexpected`,
which covers I/O, is retryable; `DataInvalid`, `PreconditionFailed` and
the like are not; an unlisted kind keeps the library's flag), because
iceberg-rust sets its own retryable flag only on commit conflicts.
Storage's own errors follow the same split: local I/O is retryable,
validation is not. `run_in_process` retries only `Retry::Task`,
immediately, with no wait in Phase 0.
- A new `Error::Data` variant takes cast, parse, overflow and
divide-by-zero failures from the root of a `DataFusionError` and names
the model when there is one. Arrow schema and argument errors map to
`Plan`; a `Plan` error for a missing table stays `NotFound`, as #46 maps
it.
- A `JobId` identifies one unit of work fixed by project version,
target, pinned inputs, the expected state (head, schema id, partition
spec id) of every output table and the clock pin `Job::started_at`. It
is assigned by the control plane and never reused. Preparation becomes a
named step, `jobs::prepare`, which decides `NoChange`. A model job is
recorded in `.infostash/jobs/<job-id>.json` after preparation and before
the first commit; the RFC defines when the record is written, rewritten
and deleted for every outcome. A job is abandoned for a new id when its
rerun finds a changed table state (`Conflict`), when a run fails with
`Retry::Never` after a partial commit, or when the recorded project
version is no longer the current one, so no target can be left
permanently blocked. `infoctl run` reruns an unfinished record under its
id before preparing a new job. A connector load commits once and keeps
no record. A separate `RunId` is never reused and appears in staged file
names, `TaskContext` and logs.
- `Warehouse::stage` and `Warehouse::commit_staged` are built from the
private halves of `append_if_head`. Staged files are named
`{job}-{run}-{stage}-{partition}-{attempt}-{n:05}-{uuid}.parquet`;
attempts count from 1, as `run_in_process` counts today. `StagedFiles`
is a versioned plain-data record in `crates/model` carrying the attempt
key, table, schema id, partition spec id and each `DataFile` in
iceberg-rust's JSON form; `stage` writes it beside the data files as the
per-attempt staging manifest ADR 0006 requires, and `commit_staged`
deletes it after the commit. `commit_staged` only appends; replace and
merge are left to #88.
- `run_task` and `Runner::run` become async (via `async-trait`, already
used by `crates/storage`). `run_task` takes a `TaskContext` (job, run,
task, attempt, with reserved exchange input, cancellation and limits)
and returns a `TaskOutput` (stats plus staged files). `TaskStats` stays
in `crates/dist` and gains quarantined-row and byte counts.
`run_in_process` returns a `TaskRun` per task with its attempt count.
- `QueryContext` holds one `local` principal and an `on_behalf_of` slot,
is `#[non_exhaustive]` and is built by `QueryContext::local()`. Every
pool runs as `Principal::Local` in Phase 0.
- `Job`, `JobOutcome`, `Error`, `TaskStats` and the new types are
`#[non_exhaustive]`, and every one that an outside crate builds has a
constructor.
- Model runs never commit through datafusion-iceberg's
`IcebergCommitExec`, which uses a plain `fast_append` with no head check
and no snapshot properties.
- Seven of the ticket's open questions are answered in the RFC; three
remain listed as open.
## Things to know
- This PR does not close#74. Its last Done-when item is the RFC
reaching Accepted after at least one week open with two maintainer
approvals (GOVERNANCE.md), so the ticket closes when the status line
changes.
- The RFC cites ADR 0006 and RFC 0001 as they are on `main`, and states
the clock pin and the compiler's determinism obligation itself rather
than relying on the RFC 0001 revision in #101. It refers to the
rerun-after-partial-commit rule by its ticket, #75, because that ADR
does not exist yet, and names two things #75 should carry: the exception
that a job made stale by a `Conflict` is abandoned for a new id rather
than rerun, and the retention rule that snapshot expiry must keep every
snapshot a job with a live record committed.
- Deviations from the ticket text, each explained in the RFC:
`StagedFiles` lives in `crates/model` as plain data rather than wrapping
iceberg's `DataFile`, so dist does not depend on storage; `InputLost` is
classed `Retry::Job` rather than a stage rerun, because Phase 0 has one
stage, and #94 should note that the dist doc comment and
`docs/architecture.md` describe the later runner.
- `crates/storage` gains a direct `serde_json` dependency for the
staging manifest, and `crates/cli` gains one for the job record.
- Code changes follow in #12, #13, #81, #82 and #95. #29 keeps attempt
counting. #94 keeps the doc fixes.
## Tests
Docs only. `typos` passes, no line outside tables or code exceeds 80
characters, and the attribution check passes. The RFC names the tests
the implementing tickets must add: a `Fake` stage failing with
`Conflict` is not retried, one failing with an I/O `Execution` error is,
a two-table job reports two commits, a staging manifest reads back equal
to its `StagedFiles`, a rerun against a changed target schema fails with
`Conflict` before any task runs, and more.
---------
Signed-off-by: Mayur Kale <hello@mayurkale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Revises RFC 0001 with the incremental rules that the Phase 0
implementation tickets (#7, #12, #35, #36, #78, #84, #86, #87, #89, #90)
depend on. Closes #73.
How
--comment lines atthe top of the file; a
-- @key: valueline after the SQL starts is anordinary comment;
@checkrepeats, may carry a name, and failing anycheck quarantines the row.
DataFusion logical plan:
TableScanof a declared append-only input,Filter,Projection,SubqueryAlias,UnnestandUNION ALL.Other nodes are a compile error that offers
@full_refresh: true.Only
Immutablefunctions may appear in incremental bodies,Stableones in checks and full-refresh models,
Volatileones nowhere.daily_revenueexampledeclares
@full_refresh: true.try_cast,a strict cast over a Raw column is a compile error, failed casts
quarantine under an implicit
cast:<col>check that runs on every inputrow before the
where, and a check that returns NULL fails the row._source_file, then_source_row; ties within one snapshot fail therun. Clock functions return the job start time. Quarantine rows carry
_job_id.infostash.property prefix.A mismatch refuses an incremental run; a
@full_refreshrun skips thecomparison and commits the new hash.
pathpattern.@full_refresh) and the interim compile-time rule that an incrementalmodel may not read a table declaring
@unique_keyor@full_refreshuntil the plan-time check in engine: fall back to a full refresh at plan time and fix files_added_by #89.
Things to know
@partition_byis removed while the RFC is a Draft. Tables are createdwithout a partition spec and the replace commit covers unpartitioned
tables only. A later revision adds partitioning back. The first
Done-when item of Partition tables with @partition_by #36 waits on that revision.
@descriptionand@full_refresh, with thereasons stated in the RFC.
and typing happens in Clean. This overlaps the discussion in RFC 0001: project file format #5.
one input in Phase 0, which also limits
@unique_keyto such models.New open question 6 asks where checks sit on joins, aggregates and
unnests.
run that first read it. This is the one stated exception to the rule
that incremental and single-pass rebuilds give the same table.
Tests
Docs only.
typospasses, no em or en dashes, lines wrap under 80outside tables, and the attribution check passes on this text.