Skip to content

compiler: add an incremental eligibility classifier with a function allow-list #84

Open

Description

@codecapo

RFC 0001 promises that the compiler decides whether a model can run incrementally and that compiling fails with a request for @full_refresh: true when it cannot, with a window as its only example (docs/rfcs/0001-project-files.md:124-128 at a0b5e6c), but no code makes that decision: crates/compiler holds only directives() (crates/compiler/src/lib.rs:9-16) and declares no dependencies (crates/compiler/Cargo.toml:9-10). The incremental reader returns only the rows committed in a snapshot range, and its checks concern the snapshots rather than the query, refusing any snapshot that is not an append (crates/storage/src/lib.rs:231-238, :571-576), so once models run nothing would stop a window, a join or an aggregate from being evaluated over that delta and committing wrong rows without an error.

This ticket implements the rule that #73 adds to RFC 0001: a fail-closed allow-list over the unoptimized logical plan that #7 builds for each model with the workspace's DataFusion 55.2.0 (Cargo.toml:28), which accepts row-wise queries over declared append-only inputs and rejects everything else, every join and every aggregate included, with an error that names the construct. Functions need a scoped rule too. Under delta evaluation a Stable function such as now() in an incremental body is evaluated once for each row, in the run that first reads it, so a time-relative filter never ages a row out. A Volatile one such as random() returns a different value on each attempt, so a rerun can split rows between target and quarantine differently from the first attempt even though it reads the same pinned input snapshots, which ADR 0006 requires precisely so that the split is the same (docs/adr/0006-ordered-job-commits.md:71-72). An incremental body may therefore call only Immutable functions; Stable ones are allowed only in @check expressions and @full_refresh models, which each job evaluates at one pinned start time (pinning it is work for the engine in #10 and the runner in #13); and Volatile ones are rejected everywhere.

Crate: crates/compiler (or the runner of #13, if #73 places the check there)

Done when

  • The classifier inspects the unoptimized logical plan that Compile a project into a dependency graph #7 exposes for each model, never an optimized one, since simplification folds now() into a literal, and walks it with LogicalPlan::apply_with_subqueries, because plain apply never visits the subquery expressions inside Filter and Projection.
  • An incremental model's plan may hold only TableScan, Filter, Projection, SubqueryAlias, Unnest and Union (from UNION ALL) nodes, and any other node, such as Join (which also covers INTERSECT and EXCEPT), Aggregate with or without GROUP BY, Window, Distinct (which covers plain UNION), Sort, Limit, Values or EmptyRelation, or any EXISTS, IN or scalar subquery, fails compilation with an error that names the model file and the construct and suggests -- @full_refresh: true.
  • Every TableScan resolves to one of the model's inputs in the project's table list from Compile a project into a dependency graph #7, so a scan of a table function that DataFusion registers by default, such as generate_series or range, fails compilation although its node type is allowed.
  • In an incremental model body every Expr::ScalarFunction has Volatility::Immutable in its signature, checked directly rather than with Expr::is_volatile, which flags only Volatile functions and so misses now() and current_date (both Stable).
  • @check expressions may also call Stable functions, and so may @full_refresh model bodies, which compiler: implement the @full_refresh directive #90 holds to these function rules but not to the node allow-list once it accepts the directive, while a Volatile function such as random() or uuid() fails compilation in every position, @check included, with an error that does not offer @full_refresh as the fix.
  • Until the replace commit that storage: add the replace_if_head commit #88 builds exists, a model that declares @unique_key or @full_refresh (docs/rfcs/0001-project-files.md:118, :121) fails compilation with an error that names the directive, with a test for each; once storage: add the replace_if_head commit #88 has landed, Merge rows on @unique_key #35 lifts this rejection for @unique_key and compiler: implement the @full_refresh directive #90 for @full_refresh.
  • A separate check over the whole DAG fails an incremental model that reads the table of a model declaring @unique_key or @full_refresh and names that input, so incremental models read only append-only tables; it stays in force after Merge rows on @unique_key #35 or compiler: implement the @full_refresh directive #90 lifts the rejection above, until the plan-time check of engine: fall back to a full refresh at plan time and fix files_added_by #89 replaces it, and a unit test covers it on its own, since while the rejection stands it fails the input model first.
  • Test: the ready/daily_revenue query (docs/rfcs/0001-project-files.md:104-106) without @full_refresh fails naming its GROUP BY, and a window, an inner join, UNION without ALL, LIMIT, now() in the select list, a scalar subquery and a select from generate_series each fail naming that construct.
  • Test: clean/orders.sql (docs/rfcs/0001-project-files.md:88-98) without its @unique_key line, a UNION ALL of two row-wise selects and a model with now() inside a @check are all accepted by the classifier as incremental, and a model with random() inside a @check is rejected.

Points: 5 (M2 classifier allow-list 5)

Depends on: #73 (the allow-list text and where the check runs), #7 (each model's directives, input tables and unoptimized plan)
Blocks: #89 (the check over the DAG that it removes), #90 (the interim rejection it lifts for @full_refresh, and the function rules), #35 (the interim rejection it lifts for @unique_key, and the node and function rules)

Source: design review of 2026-11-01, findings C03 and C18, with section 2.5 (points from section 8.5, M2).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/compilercrates/compilerblockedWaiting on another issue or a decisionenhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions