You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The design review of 2026-11-01 touches this issue. It keeps the directive parser, the dependency graph and the run order here, adds a DataFusion logical plan for each model, and leaves the incremental rules built on them to new tickets.
Suggested additions to the Done-when:
Once docs: revise RFC 0001 with the incremental rules (decision D2) #73 settles the directive grammar, tests cover it: the header of the RFC's own clean/orders gives both @check lines in file order (docs/rfcs/0001-project-files.md:90-91 at a0b5e6c), a -- @ line after the SQL starts is ignored (:112-114), and -- @check not_negative: amount >= 0 follows docs: revise RFC 0001 with the incremental rules (decision D2) #73's answer to open question 2 (:169-171). Today the parser reads every -- @ comment line in the file and keeps one value per key in a BTreeMap, so only the second check survives, and it splits at the first colon, so the named form becomes the key check not_negative (crates/compiler/src/lib.rs:9-16, :13-14).
-- @partition_by: day fails compilation with an error that names the directive, with a test, until Partition tables with @partition_by #36 implements partitioning; if docs: revise RFC 0001 with the incremental rules (decision D2) #73 removes the directive's row, it fails as an unknown key instead until a later revision adds it back. Today it is parsed into a map that only the unit test of directives() reads (crates/compiler/src/lib.rs:25), while tables are created without a partition spec and written unpartitioned (crates/storage/src/lib.rs:127-131, :361-362).
The exposed plan is analysed but not optimized, with a test for each side: the casts that type coercion inserts (where amount > 0 over a Raw column, which is nullable text, crates/connectors/src/csv.rs:199) and :: casts appear in it for the strict-cast lint (Row checks and quarantine tables #12), and now() stays a function call, which compiler: add an incremental eligibility classifier with a function allow-list #84 can reject in an incremental body, instead of a literal folded in by simplification.
Suggested re-score: 13 instead of the 5 in #19, an estimate. Section 8.5 of the design review of 2026-11-01 scores two milestone M2 items that stay in this issue, "directive parser" at 3 and "SQL to logical plan, DAG and layer rules" at 8. Together they are 11, more than the 5 that #19 lists, and the next number on the Fibonacci scale is 13. This is a suggestion only; #19 decides.
Source: design review of 2026-11-01, findings C03, C14, C16, C18, C19 and C32 (points from section 8.5, M2).
Starting this in two pull requests, to keep each under the size limit. The first covers the original Done-when list: loading infostash.toml, the sources and the models, the header grammar from RFC 0001 as revised in #73, the tables each model reads from the parsed SQL, the layer, name and cycle errors with file and line, and the run order. The second plans each model into an unoptimised DataFusion logical plan in run order; it needs a decision first on where a Raw table's columns come from at compile time, before any file has been loaded.
Activity
docs: add decision records and RFC 0001 for the project file format (#20
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. It keeps the directive parser, the dependency graph and the run order here, adds a DataFusion logical plan for each model, and leaves the incremental rules built on them to new tickets.
Suggested additions to the Done-when:
clean/ordersgives both@checklines in file order (docs/rfcs/0001-project-files.md:90-91at a0b5e6c), a-- @line after the SQL starts is ignored (:112-114), and-- @check not_negative: amount >= 0follows docs: revise RFC 0001 with the incremental rules (decision D2) #73's answer to open question 2 (:169-171). Today the parser reads every-- @comment line in the file and keeps one value per key in aBTreeMap, so only the second check survives, and it splits at the first colon, so the named form becomes the keycheck not_negative(crates/compiler/src/lib.rs:9-16,:13-14).-- @partition_by: dayfails compilation with an error that names the directive, with a test, until Partition tables with @partition_by #36 implements partitioning; if docs: revise RFC 0001 with the incremental rules (decision D2) #73 removes the directive's row, it fails as an unknown key instead until a later revision adds it back. Today it is parsed into a map that only the unit test ofdirectives()reads (crates/compiler/src/lib.rs:25), while tables are created without a partition spec and written unpartitioned (crates/storage/src/lib.rs:127-131,:361-362).crates/compiler/src/lib.rs:1-2,crates/compiler/Cargo.toml:3) but it has no dependencies yet (crates/compiler/Cargo.toml:9-10), so it gains the workspace's DataFusion 55.2.0 (Cargo.toml:28); if docs: revise RFC 0001 with the incremental rules (decision D2) #73 has the runner of In-process runner: stage, run tasks, commit once #13 classify with catalog access instead, this item moves to that runner.where amount > 0over a Raw column, which is nullable text,crates/connectors/src/csv.rs:199) and::casts appear in it for the strict-cast lint (Row checks and quarantine tables #12), andnow()stays a function call, which compiler: add an incremental eligibility classifier with a function allow-list #84 can reject in an incremental body, instead of a literal folded in by simplification.Work that moved to new tickets:
@partition_bykeeps its row, and settles whether the compiler or the runner checks incremental eligibility.Immutable.@full_refresh: truework once the replace commit exists.Suggested re-score: 13 instead of the 5 in #19, an estimate. Section 8.5 of the design review of 2026-11-01 scores two milestone M2 items that stay in this issue, "directive parser" at 3 and "SQL to logical plan, DAG and layer rules" at 8. Together they are 11, more than the 5 that #19 lists, and the next number on the Fibonacci scale is 13. This is a suggestion only; #19 decides.
Source: design review of 2026-11-01, findings C03, C14, C16, C18, C19 and C32 (points from section 8.5, M2).
docs: revise RFC 0001 with the incremental rules (#101)
feat(cli): infoctl query runs read-only SQL against the warehouse (#46)
codecapo commentedon Nov 12, 2026
Starting this in two pull requests, to keep each under the size limit. The first covers the original Done-when list: loading
infostash.toml, the sources and the models, the header grammar from RFC 0001 as revised in #73, the tables each model reads from the parsed SQL, the layer, name and cycle errors with file and line, and the run order. The second plans each model into an unoptimised DataFusion logical plan in run order; it needs a decision first on where a Raw table's columns come from at compile time, before any file has been loaded.