Skip to content

docs: revise RFC 0001 with the incremental rules (decision D2) #73

Closed

Description

@codecapo

RFC 0001 promises that models run incrementally by default and that a query which cannot run that way fails to compile (docs/rfcs/0001-project-files.md:124-128 at a0b5e6c), but it gives no rule for deciding this, and its own ready/daily_revenue.sql (:101-107) is a GROUP BY without @full_refresh that a sound rule must reject. It also leaves @partition_by without a grammar (:120), and the two @check lines of clean/orders.sql (:90-91) collapse to one in directives(), which keeps a single value per key (crates/compiler/src/lib.rs:9-16). The RFC is still a Draft (docs/rfcs/0001-project-files.md:4), and once it is accepted, changing what a directive means is breaking and needs a new RFC and a format bump (:156-158), so every rule that the compiler, the model runner and quarantine will depend on goes into one revision now. The rules keep ADR 0006: checkpoints and the model version hash are infostash. properties committed with the rows they describe (docs/adr/0006-ordered-job-commits.md:49-51, crates/storage/src/lib.rs:63), and a rerun reads the same pinned input snapshots so that it produces the same split between target and quarantine (docs/adr/0006-ordered-job-commits.md:71-72). Only the RFC text changes here: the code lands in the issues under Blocks, and the Data error for bad values outside the checks belongs to #74.

Where: docs/rfcs/0001-project-files.md

Done when

  • The directives paragraph (:112-114) says explicitly that a -- @key: value line after the SQL starts is ignored, not an error, and the @check row (:119) says that repeated checks form a list, that a row failing any of them is quarantined, and whether a check may carry a name (open question 2, :169-171), so that Compile a project into a dependency graph #7 parses one agreed grammar (C32).
  • A paragraph at :124-128 states eligibility as an allow-list over the unoptimized DataFusion logical plan (TableScan of a declared, append-only project input, Filter, Projection, SubqueryAlias, Unnest and UNION ALL, with no subquery expressions, and any other node a compile error that names it and asks for @full_refresh: true), allows only Immutable functions in an incremental body (Stable ones such as now() only in @check expressions and @full_refresh models, Volatile ones nowhere), and says whether the compiler or the runner checks it (C03).
  • A sentence next to :126 names joins and all aggregates as non-incremental in Phase 0, ready/daily_revenue.sql gains -- @full_refresh: true, a model's output is defined as its query over the inputs at the pinned snapshots minus quarantined rows (the latest row per key under @unique_key), so orders of 100 and then 20 for 2026-10-29 give one row (2026-10-29, 120), and the @unique_key (:118) and @full_refresh (:121) rows note that the compiler rejects both until a replace commit lands (storage: spike the replace commit route (decision D1) #72 decides whether one is built), or withdraw both if storage: spike the replace commit route (decision D1) #72 ends append-only (C02, C03, C18).
  • @partition_by (:120) either gets a grammar for Partition tables with @partition_by #36 to implement (its transforms, whether it takes several fields, and whether day at :103 names the output column at :104 or the Iceberg transform), or its row and the :103 line are removed and the compiler rejects it, since tables are created without a partition spec and written unpartitioned (crates/storage/src/lib.rs:127-131, :361-362) (C19).
  • The clean/orders.sql example (:93-97) uses try_cast, a lint on the analysed plan rejects a strict cast over a Raw column in a Clean model and names the column (Raw columns are text, crates/connectors/src/csv.rs:199), every cast column gets the implicit check x is null or try_cast(x) is not null, reported as cast:<col> even when the column also has an explicit @check, explicit checks run after it on the typed values, and the @check row (docs/rfcs/0001-project-files.md:119) states one NULL rule: a NULL result quarantines the row, or NULL is accepted and required values use is not null (C14, S6-08).
  • Open question 4 (:175-176) gets the ordering rule (no deletes in Phase 0, lexical file order, and the latest row wins by Raw snapshot sequence number, then _source_file, then a new _source_row, the 1-based row within its file from the counter at crates/connectors/src/csv.rs:39 and :158), and the RFC pins now() to one job start time stored with the job id and reused on rerun and gives quarantine rows a per-row _job_id (C15, C17).
  • A paragraph after :124-128 says that each model commits a hash of its normalized SQL, directives and input schema ids under an infostash. key in the same commit as its checkpoint (crates/storage/src/lib.rs:171-180), that a Phase 0 run refuses on a mismatch and names the model, that falling back to a full refresh (the @full_refresh row, docs/rfcs/0001-project-files.md:121) waits for storage: spike the replace commit route (decision D1) #72, and that additive Raw schema evolution stays a later opt-in (C16).
  • The Sources section (:80-83) says that the CSV position also stores the source's path pattern in an optional field that earlier positions parse without, and that a run fails when the pattern changes, because the position keys files by path alone and a file seen under a new path loads again (crates/connectors/src/csv.rs:7-10); the listing filters and the producer contract stay with connectors: glob once per job, skip dotfiles, symlinks, empty and fresh files #78 (C08).

Points: 3 (D2)

Depends on: #5 (fold the revision into the RFC before it is accepted if #5 is still open; otherwise revise it as GOVERNANCE.md allows), #72 (only for the @full_refresh and @unique_key text)
Blocks: #7 (the directive grammar), #8 (the example Clean and Ready models it writes), #12 (the NULL rule, check names, the cast check and _job_id), #16 (the daily_revenue result and the quarantined row it shows), #36 (the grammar, or the removal of the directive), #78 (the pattern field in the position only), #84 (the eligibility rule), #86 (the ordering rule), #87 (the version-hash rule), #90 (the @full_refresh text and the per-mode output rule)

Source: design review of 2026-11-01, section 8.5, D2 (findings C02, C03, C08, C14, C15, C16, C17, C18, C19 and C32, and design point S6-08).

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    Revision opened as #101.

    Decisions taken in it:

    • Directives: the header is the run of blank and -- comment lines at the top of the file; a directive line starts at column one with -- @; a -- @key: value line after the SQL starts is an ordinary comment; @check repeats and may carry a name; @full_refresh takes only true.
    • Eligibility is an allow-list over the unoptimised logical plan (TableScan of a declared append-only input, Filter, Projection, SubqueryAlias, Unnest, UNION ALL). Other nodes are a compile error that offers @full_refresh: true. Immutable functions only in incremental bodies, Stable in checks and full-refresh models, Volatile nowhere.
    • daily_revenue declares @full_refresh: true.
    • @partition_by is removed while the RFC is a Draft.
    • Raw columns are nullable text, strict casts over them are a compile error, failed try_casts quarantine under cast:<col>, and a NULL check result fails the row.
    • Ordering is by data sequence number, then _source_file, then _source_row; ties within one snapshot fail the run. Clock functions return the job start time. Quarantine rows carry _job_id.
    • The version hash covers the reprinted SQL, @unique_key, the checks and the input schema ids. A mismatch refuses an incremental run; a @full_refresh run skips the comparison and commits the new hash.
    • The CSV position records the path pattern.
    • Checks, written and implicit, are limited to row-wise models over one input in Phase 0, which also limits @unique_key to such models. New open question 6 covers the rest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/docsDocs, RFCs and ADRsrfcNeeds or is an RFC

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions