Repository navigation
Description
RFC 0001 promises that models run incrementally by default and that a query which cannot run that way fails to compile (docs/rfcs/0001-project-files.md:124-128 at a0b5e6c), but it gives no rule for deciding this, and its own ready/daily_revenue.sql (:101-107) is a GROUP BY without @full_refresh that a sound rule must reject. It also leaves @partition_by without a grammar (:120), and the two @check lines of clean/orders.sql (:90-91) collapse to one in directives(), which keeps a single value per key (crates/compiler/src/lib.rs:9-16). The RFC is still a Draft (docs/rfcs/0001-project-files.md:4), and once it is accepted, changing what a directive means is breaking and needs a new RFC and a format bump (:156-158), so every rule that the compiler, the model runner and quarantine will depend on goes into one revision now. The rules keep ADR 0006: checkpoints and the model version hash are infostash. properties committed with the rows they describe (docs/adr/0006-ordered-job-commits.md:49-51, crates/storage/src/lib.rs:63), and a rerun reads the same pinned input snapshots so that it produces the same split between target and quarantine (docs/adr/0006-ordered-job-commits.md:71-72). Only the RFC text changes here: the code lands in the issues under Blocks, and the Data error for bad values outside the checks belongs to #74.
Where: docs/rfcs/0001-project-files.md
Done when
- The directives paragraph (:112-114) says explicitly that a
-- @key: valueline after the SQL starts is ignored, not an error, and the@checkrow (:119) says that repeated checks form a list, that a row failing any of them is quarantined, and whether a check may carry a name (open question 2, :169-171), so that Compile a project into a dependency graph #7 parses one agreed grammar (C32). - A paragraph at :124-128 states eligibility as an allow-list over the unoptimized DataFusion logical plan (
TableScanof a declared, append-only project input,Filter,Projection,SubqueryAlias,UnnestandUNION ALL, with no subquery expressions, and any other node a compile error that names it and asks for@full_refresh: true), allows onlyImmutablefunctions in an incremental body (Stableones such asnow()only in@checkexpressions and@full_refreshmodels,Volatileones nowhere), and says whether the compiler or the runner checks it (C03). - A sentence next to :126 names joins and all aggregates as non-incremental in Phase 0,
ready/daily_revenue.sqlgains-- @full_refresh: true, a model's output is defined as its query over the inputs at the pinned snapshots minus quarantined rows (the latest row per key under@unique_key), so orders of 100 and then 20 for 2026-10-29 give one row (2026-10-29, 120), and the@unique_key(:118) and@full_refresh(:121) rows note that the compiler rejects both until a replace commit lands (storage: spike the replace commit route (decision D1) #72 decides whether one is built), or withdraw both if storage: spike the replace commit route (decision D1) #72 ends append-only (C02, C03, C18). -
@partition_by(:120) either gets a grammar for Partition tables with @partition_by #36 to implement (its transforms, whether it takes several fields, and whetherdayat :103 names the output column at :104 or the Iceberg transform), or its row and the :103 line are removed and the compiler rejects it, since tables are created without a partition spec and written unpartitioned (crates/storage/src/lib.rs:127-131,:361-362) (C19). - The
clean/orders.sqlexample (:93-97) usestry_cast, a lint on the analysed plan rejects a strict cast over a Raw column in a Clean model and names the column (Raw columns are text,crates/connectors/src/csv.rs:199), every cast column gets the implicit checkx is null or try_cast(x) is not null, reported ascast:<col>even when the column also has an explicit@check, explicit checks run after it on the typed values, and the@checkrow (docs/rfcs/0001-project-files.md:119) states one NULL rule: a NULL result quarantines the row, or NULL is accepted and required values useis not null(C14, S6-08). - Open question 4 (:175-176) gets the ordering rule (no deletes in Phase 0, lexical file order, and the latest row wins by Raw snapshot sequence number, then
_source_file, then a new_source_row, the 1-based row within its file from the counter atcrates/connectors/src/csv.rs:39and:158), and the RFC pinsnow()to one job start time stored with the job id and reused on rerun and gives quarantine rows a per-row_job_id(C15, C17). - A paragraph after :124-128 says that each model commits a hash of its normalized SQL, directives and input schema ids under an
infostash.key in the same commit as its checkpoint (crates/storage/src/lib.rs:171-180), that a Phase 0 run refuses on a mismatch and names the model, that falling back to a full refresh (the@full_refreshrow,docs/rfcs/0001-project-files.md:121) waits for storage: spike the replace commit route (decision D1) #72, and that additive Raw schema evolution stays a later opt-in (C16). - The Sources section (:80-83) says that the CSV position also stores the source's
pathpattern in an optional field that earlier positions parse without, and that a run fails when the pattern changes, because the position keys files by path alone and a file seen under a new path loads again (crates/connectors/src/csv.rs:7-10); the listing filters and the producer contract stay with connectors: glob once per job, skip dotfiles, symlinks, empty and fresh files #78 (C08).
Points: 3 (D2)
Depends on: #5 (fold the revision into the RFC before it is accepted if #5 is still open; otherwise revise it as GOVERNANCE.md allows), #72 (only for the @full_refresh and @unique_key text)
Blocks: #7 (the directive grammar), #8 (the example Clean and Ready models it writes), #12 (the NULL rule, check names, the cast check and _job_id), #16 (the daily_revenue result and the quarantined row it shows), #36 (the grammar, or the removal of the directive), #78 (the pattern field in the position only), #84 (the eligibility rule), #86 (the ordering rule), #87 (the version-hash rule), #90 (the @full_refresh text and the per-mode output rule)
Source: design review of 2026-11-01, section 8.5, D2 (findings C02, C03, C08, C14, C15, C16, C17, C18, C19 and C32, and design point S6-08).
Activity
codecapo commentedon Nov 10, 2026
Revision opened as #101.
Decisions taken in it:
--comment lines at the top of the file; a directive line starts at column one with-- @; a-- @key: valueline after the SQL starts is an ordinary comment;@checkrepeats and may carry a name;@full_refreshtakes onlytrue.TableScanof a declared append-only input,Filter,Projection,SubqueryAlias,Unnest,UNION ALL). Other nodes are a compile error that offers@full_refresh: true.Immutablefunctions only in incremental bodies,Stablein checks and full-refresh models,Volatilenowhere.daily_revenuedeclares@full_refresh: true.@partition_byis removed while the RFC is a Draft.try_casts quarantine undercast:<col>, and a NULL check result fails the row._source_file, then_source_row; ties within one snapshot fail the run. Clock functions return the job start time. Quarantine rows carry_job_id.@unique_key, the checks and the input schema ids. A mismatch refuses an incremental run; a@full_refreshrun skips the comparison and commits the new hash.pathpattern.@unique_keyto such models. New open question 6 covers the rest.docs: answer the review on RFC 0001
docs: record the replace commit route (ADR 0007, ADR 0008) (#100)