Repository navigation
Description
RFC 0001 promises that the compiler decides whether a model can run incrementally and that compiling fails with a request for @full_refresh: true when it cannot, with a window as its only example (docs/rfcs/0001-project-files.md:124-128 at a0b5e6c), but no code makes that decision: crates/compiler holds only directives() (crates/compiler/src/lib.rs:9-16) and declares no dependencies (crates/compiler/Cargo.toml:9-10). The incremental reader returns only the rows committed in a snapshot range, and its checks concern the snapshots rather than the query, refusing any snapshot that is not an append (crates/storage/src/lib.rs:231-238, :571-576), so once models run nothing would stop a window, a join or an aggregate from being evaluated over that delta and committing wrong rows without an error.
This ticket implements the rule that #73 adds to RFC 0001: a fail-closed allow-list over the unoptimized logical plan that #7 builds for each model with the workspace's DataFusion 55.2.0 (Cargo.toml:28), which accepts row-wise queries over declared append-only inputs and rejects everything else, every join and every aggregate included, with an error that names the construct. Functions need a scoped rule too. Under delta evaluation a Stable function such as now() in an incremental body is evaluated once for each row, in the run that first reads it, so a time-relative filter never ages a row out. A Volatile one such as random() returns a different value on each attempt, so a rerun can split rows between target and quarantine differently from the first attempt even though it reads the same pinned input snapshots, which ADR 0006 requires precisely so that the split is the same (docs/adr/0006-ordered-job-commits.md:71-72). An incremental body may therefore call only Immutable functions; Stable ones are allowed only in @check expressions and @full_refresh models, which each job evaluates at one pinned start time (pinning it is work for the engine in #10 and the runner in #13); and Volatile ones are rejected everywhere.
Crate: crates/compiler (or the runner of #13, if #73 places the check there)
Done when
- The classifier inspects the unoptimized logical plan that Compile a project into a dependency graph #7 exposes for each model, never an optimized one, since simplification folds
now()into a literal, and walks it withLogicalPlan::apply_with_subqueries, because plainapplynever visits the subquery expressions insideFilterandProjection. - An incremental model's plan may hold only
TableScan,Filter,Projection,SubqueryAlias,UnnestandUnion(fromUNION ALL) nodes, and any other node, such asJoin(which also coversINTERSECTandEXCEPT),Aggregatewith or withoutGROUP BY,Window,Distinct(which covers plainUNION),Sort,Limit,ValuesorEmptyRelation, or anyEXISTS,INor scalar subquery, fails compilation with an error that names the model file and the construct and suggests-- @full_refresh: true. - Every
TableScanresolves to one of the model's inputs in the project's table list from Compile a project into a dependency graph #7, so a scan of a table function that DataFusion registers by default, such asgenerate_seriesorrange, fails compilation although its node type is allowed. - In an incremental model body every
Expr::ScalarFunctionhasVolatility::Immutablein its signature, checked directly rather than withExpr::is_volatile, which flags onlyVolatilefunctions and so missesnow()andcurrent_date(bothStable). -
@checkexpressions may also callStablefunctions, and so may@full_refreshmodel bodies, which compiler: implement the @full_refresh directive #90 holds to these function rules but not to the node allow-list once it accepts the directive, while aVolatilefunction such asrandom()oruuid()fails compilation in every position,@checkincluded, with an error that does not offer@full_refreshas the fix. - Until the replace commit that storage: add the replace_if_head commit #88 builds exists, a model that declares
@unique_keyor@full_refresh(docs/rfcs/0001-project-files.md:118,:121) fails compilation with an error that names the directive, with a test for each; once storage: add the replace_if_head commit #88 has landed, Merge rows on @unique_key #35 lifts this rejection for@unique_keyand compiler: implement the @full_refresh directive #90 for@full_refresh. - A separate check over the whole DAG fails an incremental model that reads the table of a model declaring
@unique_keyor@full_refreshand names that input, so incremental models read only append-only tables; it stays in force after Merge rows on @unique_key #35 or compiler: implement the @full_refresh directive #90 lifts the rejection above, until the plan-time check of engine: fall back to a full refresh at plan time and fix files_added_by #89 replaces it, and a unit test covers it on its own, since while the rejection stands it fails the input model first. - Test: the
ready/daily_revenuequery (docs/rfcs/0001-project-files.md:104-106) without@full_refreshfails naming itsGROUP BY, and a window, an inner join,UNIONwithoutALL,LIMIT,now()in the select list, a scalar subquery and a select fromgenerate_serieseach fail naming that construct. - Test:
clean/orders.sql(docs/rfcs/0001-project-files.md:88-98) without its@unique_keyline, aUNION ALLof two row-wise selects and a model withnow()inside a@checkare all accepted by the classifier as incremental, and a model withrandom()inside a@checkis rejected.
Points: 5 (M2 classifier allow-list 5)
Depends on: #73 (the allow-list text and where the check runs), #7 (each model's directives, input tables and unoptimized plan)
Blocks: #89 (the check over the DAG that it removes), #90 (the interim rejection it lifts for @full_refresh, and the function rules), #35 (the interim rejection it lifts for @unique_key, and the node and function rules)
Source: design review of 2026-11-01, findings C03 and C18, with section 2.5 (points from section 8.5, M2).
Activity
docs: record the replace commit route (ADR 0007, ADR 0008) (#100)
docs: revise RFC 0001 with the incremental rules (#101)