You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Agree the format of an Infostash project before building the compiler and infoctl init: infostash.toml, sources/*.toml, and SQL models in clean/ and ready/ with -- @key: value directives.
Where:docs/rfcs/0001-project-files.md
Done when
The RFC has been open for comment for at least one week
Its open questions are answered in the RFC
It is approved as set out in GOVERNANCE.md and merged with status Accepted
The design review of 2026-11-01 touches this issue. It collects the decisions that RFC 0001 has to settle into one revision, #73, which should be folded into the RFC before it is accepted, if the comment period allows; the RFC is still a Draft (docs/rfcs/0001-project-files.md:4 at a0b5e6c). Bare line numbers below refer to that file.
Suggested additions to the Done-when:
The docs: revise RFC 0001 with the incremental rules (decision D2) #73 revision is folded into the RFC before its status changes to Accepted, if the comment period allows (Tracking: Phase 0 (0.1.0) #19 gives its end as 2026-11-05). If the revision is not ready when the period ends, the RFC stays a Draft and the period runs longer, which GOVERNANCE.md:28-29 permits because one week open for comment is the minimum. The RFC is to be agreed before the compiler is written (:18-21), so the design review of 2026-11-01 counts the replace-commit and incremental-rule questions (findings C01 to C03) as blockers for its acceptance, and once it is accepted, changing what a directive means needs a new RFC and a format bump (:156-158).
The RFC settles what a relative source path is resolved against. It says relative to the project (:77), but the CSV connector hands the pattern to glob::glob as written (crates/connectors/src/csv.rs:82), so today the files found depend on the directory the process runs in. The position records each file by the path the glob yields (crates/connectors/src/csv.rs:7-10), so a run from another directory skips any file whose relative name is already recorded, even though it is a different file.
The RFC says whether one source per Raw table is a rule. It says that sources/ defines Raw tables and that each file is one source (:45-46), and its layout maps orders.toml to bronze.orders (:38), but it does not say that a Raw table has only one source. The position is held in one property of the Raw table, infostash.position (crates/connectors/src/lib.rs:19-21), so two sources feeding one table would overwrite each other's position. connectors: glob once per job, skip dotfiles, symlinks, empty and fresh files #78 also lists one source per Raw table in the producer contract it writes into the CSV connector's doc comment.
The @check row (:119) says that repeated checks form a list; today directives() keeps one value per key (crates/compiler/src/lib.rs:9-16, as :160-163 notes), so the two checks at :90-91 become one (C32).
The incremental rule at :124-128 is specified as an allow-list over the unoptimized DataFusion logical plan, and the RFC says whether the compiler or the runner checks it (C03).
Joins and all aggregates are named non-incremental in Phase 0, and ready/daily_revenue.sql (:101-107) gains -- @full_refresh: true (C02, C18).
The clean/orders.sql example (:93-97) uses try_cast, a lint rejects strict casts over Raw columns in Clean models, since Raw columns are all text (crates/connectors/src/csv.rs:3), and every cast column gets an implicit check (C14).
Explicit checks run after the cast check on the typed output columns, so amount >= 0 (:90) tests the decimal from :96 and not the Raw text, and the @check row (:119) either keeps sending a NULL result to quarantine or switches once to accepting NULL, with is not null for required values (C14, S6-08).
Open question 4 (:175-176) gets an ordering rule built on the Raw snapshot sequence number and a new _source_row column (the row number within its file), and now() is pinned to one start time per job and reused on rerun (C15, C17).
Each model commits a hash of its normalized SQL, directives and input schema ids with its checkpoint, and a run refuses on a mismatch (C16).
The CSV position also stores the source's path pattern, and a run fails when the pattern changes (C08).
Source: design review of 2026-11-01, findings C01, C02, C03, C08, C14, C15, C16, C17, C18, C19 and C32, design point S6-08, decision D2, and the open questions of its sections 3.6 and 6.5.
#101 answers open question 3 of RFC 0001: every CSV column is read as nullable text and typing happens in Clean models with try_cast. A declared schema in the source file can come later as a connector key.
Activity
codecapo commentedon Oct 29, 2026
Draft RFC is up for comment in #20.
docs: add decision records and RFC 0001 for the project file format (#20
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. It collects the decisions that RFC 0001 has to settle into one revision, #73, which should be folded into the RFC before it is accepted, if the comment period allows; the RFC is still a Draft (
docs/rfcs/0001-project-files.md:4at a0b5e6c). Bare line numbers below refer to that file.Suggested additions to the Done-when:
GOVERNANCE.md:28-29permits because one week open for comment is the minimum. The RFC is to be agreed before the compiler is written (:18-21), so the design review of 2026-11-01 counts the replace-commit and incremental-rule questions (findings C01 to C03) as blockers for its acceptance, and once it is accepted, changing what a directive means needs a new RFC and aformatbump (:156-158).pathis resolved against. It says relative to the project (:77), but the CSV connector hands the pattern toglob::globas written (crates/connectors/src/csv.rs:82), so today the files found depend on the directory the process runs in. The position records each file by the path the glob yields (crates/connectors/src/csv.rs:7-10), so a run from another directory skips any file whose relative name is already recorded, even though it is a different file.sources/defines Raw tables and that each file is one source (:45-46), and its layout mapsorders.tomltobronze.orders(:38), but it does not say that a Raw table has only one source. The position is held in one property of the Raw table,infostash.position(crates/connectors/src/lib.rs:19-21), so two sources feeding one table would overwrite each other's position. connectors: glob once per job, skip dotfiles, symlinks, empty and fresh files #78 also lists one source per Raw table in the producer contract it writes into the CSV connector's doc comment.Work that moved to new tickets:
@checkrow (:119) says that repeated checks form a list; todaydirectives()keeps one value per key (crates/compiler/src/lib.rs:9-16, as :160-163 notes), so the two checks at :90-91 become one (C32).ready/daily_revenue.sql(:101-107) gains-- @full_refresh: true(C02, C18).@unique_key(:118) and@full_refresh(:121) rows say the compiler rejects both until a replace commit lands, which storage: spike the replace commit route (decision D1) #72 decides, or both are withdrawn if storage: spike the replace commit route (decision D1) #72 ends append-only (C01, C02).@partition_by(:120) gets a grammar for Partition tables with @partition_by #36, or its row and its use at :103 are removed (C19).clean/orders.sqlexample (:93-97) usestry_cast, a lint rejects strict casts over Raw columns in Clean models, since Raw columns are all text (crates/connectors/src/csv.rs:3), and every cast column gets an implicit check (C14).amount >= 0(:90) tests the decimal from :96 and not the Raw text, and the@checkrow (:119) either keeps sending a NULL result to quarantine or switches once to accepting NULL, withis not nullfor required values (C14, S6-08)._source_rowcolumn (the row number within its file), andnow()is pinned to one start time per job and reused on rerun (C15, C17).pathpattern, and a run fails when the pattern changes (C08).Source: design review of 2026-11-01, findings C01, C02, C03, C08, C14, C15, C16, C17, C18, C19 and C32, design point S6-08, decision D2, and the open questions of its sections 3.6 and 6.5.
codecapo commentedon Nov 10, 2026
#101 answers open question 3 of RFC 0001: every CSV column is read as nullable text and typing happens in Clean models with
try_cast. A declared schema in the source file can come later as a connector key.docs: revise RFC 0001 with the incremental rules (#101)
feat(cli): infoctl query runs read-only SQL against the warehouse (#46)