Skip to content

docs: revise RFC 0001 with the incremental rules - #101

Merged
codecapo merged 3 commits into
mainfrom
docs/rfc-0001-incremental-rules
Nov 10, 2026
Merged

codecapo merged 3 commits into
mainfrom
docs/rfc-0001-incremental-rules

Conversation

@codecapo

@codecapo codecapo commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

What

Revises RFC 0001 with the incremental rules that the Phase 0
implementation tickets (#7, #12, #35, #36, #78, #84, #86, #87, #89, #90)
depend on. Closes #73.

How

  • Directives: the header is the run of blank and -- comment lines at
    the top of the file; a -- @key: value line after the SQL starts is an
    ordinary comment; @check repeats, may carry a name, and failing any
    check quarantines the row.
  • Incremental eligibility is an allow-list over the unoptimised
    DataFusion logical plan: TableScan of a declared append-only input,
    Filter, Projection, SubqueryAlias, Unnest and UNION ALL.
    Other nodes are a compile error that offers @full_refresh: true.
    Only Immutable functions may appear in incremental bodies, Stable
    ones in checks and full-refresh models, Volatile ones nowhere.
  • Joins and aggregates are not incremental; the daily_revenue example
    declares @full_refresh: true.
  • Raw columns are nullable text. Clean models type them with try_cast,
    a strict cast over a Raw column is a compile error, failed casts
    quarantine under an implicit cast:<col> check that runs on every input
    row before the where, and a check that returns NULL fails the row.
  • Ordering: latest per key is the highest snapshot sequence number, then
    _source_file, then _source_row; ties within one snapshot fail the
    run. Clock functions return the job start time. Quarantine rows carry
    _job_id.
  • A model version hash is stored under the infostash. property prefix.
    A mismatch refuses an incremental run; a @full_refresh run skips the
    comparison and commits the new hash.
  • The CSV position records the path pattern.
  • Records the ADR 0008 fallback (a model rebuilt in full without
    @full_refresh) and the interim compile-time rule that an incremental
    model may not read a table declaring @unique_key or @full_refresh
    until the plan-time check in engine: fall back to a full refresh at plan time and fix files_added_by #89.

Things to know

  • Merge after docs: record the replace commit route (ADR 0007, ADR 0008) #100: this RFC links ADR 0007 and ADR 0008.
  • @partition_by is removed while the RFC is a Draft. Tables are created
    without a partition spec and the replace commit covers unpartitioned
    tables only. A later revision adds partitioning back. The first
    Done-when item of Partition tables with @partition_by #36 waits on that revision.
  • The version hash excludes @description and @full_refresh, with the
    reasons stated in the RFC.
  • Open question 3 (CSV schema) is answered: every column is nullable text
    and typing happens in Clean. This overlaps the discussion in RFC 0001: project file format #5.
  • Checks, written and implicit, are restricted to row-wise models over
    one input in Phase 0, which also limits @unique_key to such models.
    New open question 6 asks where checks sit on joins, aggregates and
    unnests.
  • A check that reads the clock judges each row by the start time of the
    run that first read it. This is the one stated exception to the rule
    that incremental and single-pass rebuilds give the same table.
  • Only the RFC changes. No code, CI or check is touched.

Tests

Docs only. typos passes, no em or en dashes, lines wrap under 80
outside tables, and the attribution check passes on this text.

Write down the rules the Phase 0 implementation tickets depend on, so
they can be built and tested from one text.

Directives: the header is the run of blank and comment lines at the top
of a model file, a directive after the SQL starts is an ordinary
comment, @check repeats and may carry a name, and unknown or malformed
directives fail compilation.

Incremental eligibility: an allow-list over the unoptimised DataFusion
logical plan (TableScan of a declared append-only input, Filter,
Projection, SubqueryAlias, Unnest and UNION ALL). Anything else is a
compile error that offers @full_refresh: true. Only Immutable functions
may appear in incremental bodies, Stable ones in checks and full-refresh
models, Volatile ones nowhere. Joins and aggregates are therefore not
incremental, and the daily_revenue example declares @full_refresh.

Typing: Raw columns are nullable text, Clean models type them with
try_cast, a strict cast over a Raw column is a compile error, failed
casts quarantine under an implicit cast:<col> check, and a check that
returns NULL fails the row.

Ordering: latest per key means the highest snapshot sequence number,
then _source_file, then _source_row; ties within one snapshot fail the
run. Clock functions return the job start time. Quarantine rows carry
_job_id. A model version hash is stored under the infostash. property
prefix; a mismatch refuses an incremental run, and a full refresh
skips the comparison and commits the new hash. The CSV position
records the path pattern.

@partition_by is removed while the RFC is a Draft: tables are created
without a partition spec and the replace commit covers unpartitioned
tables only. Open questions 2, 3 and 4 are answered and a new question
6 asks where checks sit on models that are not row-wise.

Closes #73

Signed-off-by: Mayur Kale <hello@mayurkale.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Explain Change →

Summary

Summary by CodeRabbit

  • Documentation
    • Expanded the project-format specification with details on CSV typing, metadata, append-only loading, checkpoints, model directives, checks, and incremental processing.
    • Updated examples and clarified when incremental plans require a full refresh or rebuild, including row ordering, pinned time, and model-version tracking.
    • Removed @partition_by from the directive reference, documented implementation gaps and compatibility constraints, and recorded resolved and remaining open questions.

RFC 0001 adds rules for CSV sources, model directives and checks, incremental-plan eligibility, refresh and rebuild behaviour, row ordering, quarantine and model versions. It also updates examples and records implementation gaps, compatibility constraints and open questions.

Changes: Project-file contract

Layer / File(s) Summary
Source and model rules
docs/rfcs/0001-project-files.md
The RFC describes CSV Raw-column typing, metadata and pattern-bound checkpoints. It specifies directive parsing, named and implicit checks, quarantine conditions, and restrictions on strict casts. Updated examples use named checks and try_cast.
Incremental eligibility
docs/rfcs/0001-project-files.md
The RFC defines eligible incremental plans and function restrictions. It describes full refresh and rebuild conditions, row ordering and tie failures, quarantine contents, and model-version hash behaviour. The daily-revenue example uses @full_refresh: true.
Execution and compatibility
docs/rfcs/0001-project-files.md
The RFC records alternatives, compatibility constraints and implementation gaps. It marks several questions as answered and adds an open question about checks on non-row-wise plans.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Possibly related issues

Possibly related PRs

Suggested labels: rfc, area/docs

Merge Risk: Moderate · up to f151d

This is an RFC-only change. Before merging, specify how a forced rebuild replaces quarantine, and make the clock invariant consistent with the rebuild rule that the Time section now states.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly identifies the RFC revision and its incremental-processing rules.
Description check Passed The description explains the RFC changes and their scope, so it relates directly to the changeset.
Linked Issues check Passed [#73] The only direct link is #73. Its deliverable is RFC text in docs/rfcs/0001-project-files.md, so its requirements are non-coding. No coding requirements apply. The text at head meets the verified…
Out of Scope Changes check Passed The PR changes only docs/rfcs/0001-project-files.md, as the PR description and raw summary state. No code, CI or check files are changed. The additions map to #73 Done-when items and to the findings i…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Linked Issues check

Explanation

[#73] The only direct link is #73. Its deliverable is RFC text in docs/rfcs/0001-project-files.md, so its requirements are non-coding. No coding requirements apply. The text at head meets the verified Done-when items:

  • Directives: header run of blank and -- lines, -- @key: value after SQL starts is ignored, repeated @check forms a list, any failing check quarantines the row, and a check may carry a name (open question 2 answered).
  • Eligibility: allow-list over the unoptimised plan (TableScan of a declared table, Filter, Projection, SubqueryAlias, Unnest, Union from union all, no subquery expressions). Other nodes are a compile error that offers @full_refresh: true. Immutable functions only in incremental bodies, Stable in @check and @full_refresh models, Volatile nowhere. The compiler performs the check before job creation.
  • Joins and all aggregates are non-incremental. daily_revenue declares @full_refresh: true. The 100-then-20 example gives (2026-10-29, 120).
  • @partition_by is removed from the directive table and the day line. Unknown keys are an error, so the directive is rejected.
  • clean/orders.sql uses try_cast. Strict casts over Raw columns are a compile error naming the column. Implicit cast:&lt;col&gt; checks run before the where and before written checks. The NULL rule is stated.
  • CSV position stores the path pattern. A pattern change fails the run. Earlier positions without the field are read as having none.
  • _source_row is defined as a 1-based row position.
  • Clock functions return the job start time, stored with the job id and reused on rerun.

Not verified directly: the file read truncated before the Ordering, version-hash, Compatibility and open-question sections. The ordering, _job_id and hash claims rest on the PR description and the raw summary, not on text I read. Also, the hash excludes @description and @full_refresh. Issue C16 names 'directives' generally. The PR gives a stated reason, which I could not check. Forward references to ADR 0007, ADR 0008 and ADR 0009 (#103) do not resolve at head. The PR states it merges after #100 for ADR 0007 and 0008.

Full details: Out of Scope Changes check

Explanation

The PR changes only docs/rfcs/0001-project-files.md, as the PR description and raw summary state. No code, CI or check files are changed. The additions map to #73 Done-when items and to the findings it cites: the eligibility allow-list, per-mode output rule, cast and quarantine rules, ordering and time rules, the @full_refresh and ADR 0008 fallback text (related to #90 and #84), CSV position pattern (#78), and open question 6 on checks over non-row-wise models. The CSV column description (_source_file, _loaded_at, _source_row) and the raw-text statement describe the Raw schema that the checks rely on. They are supporting RFC content, not unrelated changes. No unrelated change was demonstrated.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.14%. Comparing base (6e13aa1) to head (f151dbd).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #101   +/-   ##
=======================================
  Coverage   99.14%   99.14%           
=======================================
  Files           8        8           
  Lines        1048     1048           
=======================================
  Hits         1039     1039           
  Misses          9        9           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A try_cast of a Raw column in the where clause yields null on a bad
value, the comparison yields null and the filter drops the row, so a
check that runs after the filter never sees it. Implicit cast checks
now run on every input row before the where, and two implicit checks
that would share a name are a compile error that asks for an alias.

Signed-off-by: Mayur Kale <hello@mayurkale.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 6e13aa1 and 550a61d.

Files selected for processing: 1

  • docs/rfcs/0001-project-files.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment thread docs/rfcs/0001-project-files.md Resolved
Comment thread docs/rfcs/0001-project-files.md Resolved
Comment thread docs/rfcs/0001-project-files.md Resolved
Replace the quarantine table in its own commit before the target under
the same job id, with the reader rule covering the window between the
two commits. State that a rebuild judges rows by its own start time, as
the one way a rebuilt table can differ. Refuse a table that has rows but
no stored model version hash instead of running it.

Refs #73

Signed-off-by: Mayur Kale <hello@mayurkale.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 550a61d and f151dbd.

Files selected for processing: 1

  • docs/rfcs/0001-project-files.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment on lines +270 to +271
before any new job for the model, so a rebuild never appends the same
quarantined rows again.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Quick win

Supported by static analysis

Script executed:

sed -n '205,325p' docs/rfcs/0001-project-files.md

Repository: codecapo/infostash

Length of output: 7886


Script executed:

rg -n -i 'quarantine|rebuild|rerun|retry' docs/rfcs/0001-project-files.md docs/adr/0006* docs/adr/0008* docs/adr/0009* 2>/dev/null
cat -n docs/rfcs/0001-project-files.md | sed -n '115,205p;245,325p'
for f in docs/adr/0006* docs/adr/0008* docs/adr/0009*; do if [ -f "$f" ]; then echo "=== $f ==="; cat -n "$f"; fi; done

Repository: codecapo/infostash

Length of output: 22009


Define quarantine replacement for forced rebuilds.

A forced rebuild rechecks previously processed rows. If it appends quarantine, it can duplicate earlier failures or retain rows that now pass a clock check. Specify that it replaces quarantine as well as the target.

Suggested fix

 cause ([ADR 0008](../adr/0008-incremental-readers-of-replaced-tables.md)).
+The quarantine table is replaced as in a full refresh, under the same job id
+and before the target.

@codecapo
codecapo merged commit 7b39d58 into main Nov 10, 2026
18 of 19 checks passed
@codecapo
codecapo deleted the docs/rfc-0001-incremental-rules branch November 10, 2026 15:42
codecapo added a commit that referenced this pull request Nov 10, 2026
## What

Adds `docs/rfcs/0002-job-and-runner-contracts.md`, the RFC that #74 asks
for. It fixes the contracts between a job, the tasks that run it and the
commits it makes, across `crates/model`, `crates/jobs`, `crates/dist`
and `crates/storage`, before #13 builds the in-process runner on them.

## How

- `JobOutcome::Committed` carries one `CommitRecord` per output table
(table, snapshot, whether a rerun found it already committed under the
job id), in commit order. `Failed` carries an `ErrorKind`, a message and
a `Retry` class. `NoChange` is narrowed to "no input past its
checkpoint, decided during preparation".
- `Error::retry()` is total: `Never` for `Parse`, `Plan`, `Data`,
`NotFound` and non-retryable `Storage`; `Task` for `Execution` and
retryable `Storage`; `Job` for `Conflict` and the reserved `InputLost`.
Storage errors are classified by iceberg-rust error kind (`Unexpected`,
which covers I/O, is retryable; `DataInvalid`, `PreconditionFailed` and
the like are not; an unlisted kind keeps the library's flag), because
iceberg-rust sets its own retryable flag only on commit conflicts.
Storage's own errors follow the same split: local I/O is retryable,
validation is not. `run_in_process` retries only `Retry::Task`,
immediately, with no wait in Phase 0.
- A new `Error::Data` variant takes cast, parse, overflow and
divide-by-zero failures from the root of a `DataFusionError` and names
the model when there is one. Arrow schema and argument errors map to
`Plan`; a `Plan` error for a missing table stays `NotFound`, as #46 maps
it.
- A `JobId` identifies one unit of work fixed by project version,
target, pinned inputs, the expected state (head, schema id, partition
spec id) of every output table and the clock pin `Job::started_at`. It
is assigned by the control plane and never reused. Preparation becomes a
named step, `jobs::prepare`, which decides `NoChange`. A model job is
recorded in `.infostash/jobs/<job-id>.json` after preparation and before
the first commit; the RFC defines when the record is written, rewritten
and deleted for every outcome. A job is abandoned for a new id when its
rerun finds a changed table state (`Conflict`), when a run fails with
`Retry::Never` after a partial commit, or when the recorded project
version is no longer the current one, so no target can be left
permanently blocked. `infoctl run` reruns an unfinished record under its
id before preparing a new job. A connector load commits once and keeps
no record. A separate `RunId` is never reused and appears in staged file
names, `TaskContext` and logs.
- `Warehouse::stage` and `Warehouse::commit_staged` are built from the
private halves of `append_if_head`. Staged files are named
`{job}-{run}-{stage}-{partition}-{attempt}-{n:05}-{uuid}.parquet`;
attempts count from 1, as `run_in_process` counts today. `StagedFiles`
is a versioned plain-data record in `crates/model` carrying the attempt
key, table, schema id, partition spec id and each `DataFile` in
iceberg-rust's JSON form; `stage` writes it beside the data files as the
per-attempt staging manifest ADR 0006 requires, and `commit_staged`
deletes it after the commit. `commit_staged` only appends; replace and
merge are left to #88.
- `run_task` and `Runner::run` become async (via `async-trait`, already
used by `crates/storage`). `run_task` takes a `TaskContext` (job, run,
task, attempt, with reserved exchange input, cancellation and limits)
and returns a `TaskOutput` (stats plus staged files). `TaskStats` stays
in `crates/dist` and gains quarantined-row and byte counts.
`run_in_process` returns a `TaskRun` per task with its attempt count.
- `QueryContext` holds one `local` principal and an `on_behalf_of` slot,
is `#[non_exhaustive]` and is built by `QueryContext::local()`. Every
pool runs as `Principal::Local` in Phase 0.
- `Job`, `JobOutcome`, `Error`, `TaskStats` and the new types are
`#[non_exhaustive]`, and every one that an outside crate builds has a
constructor.
- Model runs never commit through datafusion-iceberg's
`IcebergCommitExec`, which uses a plain `fast_append` with no head check
and no snapshot properties.
- Seven of the ticket's open questions are answered in the RFC; three
remain listed as open.

## Things to know

- This PR does not close #74. Its last Done-when item is the RFC
reaching Accepted after at least one week open with two maintainer
approvals (GOVERNANCE.md), so the ticket closes when the status line
changes.
- The RFC cites ADR 0006 and RFC 0001 as they are on `main`, and states
the clock pin and the compiler's determinism obligation itself rather
than relying on the RFC 0001 revision in #101. It refers to the
rerun-after-partial-commit rule by its ticket, #75, because that ADR
does not exist yet, and names two things #75 should carry: the exception
that a job made stale by a `Conflict` is abandoned for a new id rather
than rerun, and the retention rule that snapshot expiry must keep every
snapshot a job with a live record committed.
- Deviations from the ticket text, each explained in the RFC:
`StagedFiles` lives in `crates/model` as plain data rather than wrapping
iceberg's `DataFile`, so dist does not depend on storage; `InputLost` is
classed `Retry::Job` rather than a stage rerun, because Phase 0 has one
stage, and #94 should note that the dist doc comment and
`docs/architecture.md` describe the later runner.
- `crates/storage` gains a direct `serde_json` dependency for the
staging manifest, and `crates/cli` gains one for the job record.
- Code changes follow in #12, #13, #81, #82 and #95. #29 keeps attempt
counting. #94 keeps the doc fixes.

## Tests

Docs only. `typos` passes, no line outside tables or code exceeds 80
characters, and the attribution check passes. The RFC names the tests
the implementing tickets must add: a `Fake` stage failing with
`Conflict` is not retried, one failing with an I/O `Execution` error is,
a two-table job reports two commits, a staging manifest reads back equal
to its `StagedFiles`, a rerun against a changed target schema fails with
`Conflict` before any task runs, and more.

---------

Signed-off-by: Mayur Kale <hello@mayurkale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: revise RFC 0001 with the incremental rules (decision D2)

1 participant