You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The design review of 2026-11-01 touches this issue. The Ready model that init writes, ready/daily_revenue, is a GROUP BY aggregate that gives wrong results when run over only the new rows, so it has to carry -- @full_refresh: true, and the sample data should set up the case that shows the difference.
Suggested additions to the Done-when:
The Ready model that init writes matches the RFC 0001 example as revised in docs: revise RFC 0001 with the incremental rules (decision D2) #73: ready/daily_revenue.sql keeps its name and query, because the quick start queries gold.daily_revenue (README.md:42 at a0b5e6c), and carries -- @full_refresh: true in its header, which the init test checks. The example today (docs/rfcs/0001-project-files.md:101-107) has no such line, so under the append and incremental defaults (:118, :124-128) a second run would read only the newly committed rows (crates/storage/tests/iceberg_roundtrip.rs:77-79) and append a second row for a day the table already holds. If storage: spike the replace commit route (decision D1) #72 ends with an append-only 0.1.0, docs: revise RFC 0001 with the incremental rules (decision D2) #73 withdraws the directive instead, the RFC examples are rewritten without it, and the Ready model follows the rewritten example.
The sample CSV that init writes holds an order for a day on which the CSV file added before the quick start's second run (Five-minute quick start, tested in CI #16) adds a second order, for example 100 and then 20 for 2026-10-29, so that after the second run gold.daily_revenue holds exactly one row for that day, with revenue 120. The added file and that check belong to Five-minute quick start, tested in CI #16. The two orders have different order_id values, so the result does not depend on a merge in Clean; a correction of the first order would need the @unique_key merge of Merge rows on @unique_key #35.
Activity
docs: add decision records and RFC 0001 for the project file format (#20
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. The Ready model that
initwrites,ready/daily_revenue, is aGROUP BYaggregate that gives wrong results when run over only the new rows, so it has to carry-- @full_refresh: true, and the sample data should set up the case that shows the difference.Suggested additions to the Done-when:
initwrites matches the RFC 0001 example as revised in docs: revise RFC 0001 with the incremental rules (decision D2) #73:ready/daily_revenue.sqlkeeps its name and query, because the quick start queriesgold.daily_revenue(README.md:42at a0b5e6c), and carries-- @full_refresh: truein its header, which theinittest checks. The example today (docs/rfcs/0001-project-files.md:101-107) has no such line, so under the append and incremental defaults (:118, :124-128) a second run would read only the newly committed rows (crates/storage/tests/iceberg_roundtrip.rs:77-79) and append a second row for a day the table already holds. If storage: spike the replace commit route (decision D1) #72 ends with an append-only 0.1.0, docs: revise RFC 0001 with the incremental rules (decision D2) #73 withdraws the directive instead, the RFC examples are rewritten without it, and the Ready model follows the rewritten example.initwrites holds an order for a day on which the CSV file added before the quick start's second run (Five-minute quick start, tested in CI #16) adds a second order, for example 100 and then 20 for 2026-10-29, so that after the second rungold.daily_revenueholds exactly one row for that day, with revenue 120. The added file and that check belong to Five-minute quick start, tested in CI #16. The two orders have differentorder_idvalues, so the result does not depend on a merge in Clean; a correction of the first order would need the@unique_keymerge of Merge rows on @unique_key #35.Work that moved to new tickets:
daily_revenueexample carries-- @full_refresh: trueand states what each kind of model outputs.@full_refreshdirective, whichinfoctl runneeds before it can build the Ready model in the projectinitcreates.Source: design review of 2026-11-01, finding C02 and its worked example in section 2.1, with finding C01 for the append-only alternative.