You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Run a model with DataFusion and append to its table #10
A spike proved this issue end to end, including a 1B-row run (raw to clean to ready in 17.3 min, 5.7 GB peak RSS, every count and sum exact).
Layer names: a CatalogProvider that wraps IcebergCatalogProvider and maps names through Layer::from_namespace makes raw.orders and bronze.orders resolve to one table. Look tables up lazily rather than snapshotting the list.
Write path: plan the SQL, create the target with its schema, then INSERT INTO. That streams through IcebergWriteExec and makes one commit. Don't go through Warehouse::append, which needs every batch in memory.
Never use CREATE TABLE AS. It silently drops rows: the upstream ensure_table_is_empty only checks partition 0. Reject CTAS in model SQL. To be reported upstream.
Timestamps:cast(x as timestamp) gives nanoseconds, which needs Iceberg format v3. Normalise outputs to microseconds.
The design review of 2026-11-01 touches this issue. It asks the engine to plan every model under one start time per job and never to depend on the order rows arrive in, while reading pinned input ranges, the new _source_row column and the RFC text move to new tickets.
Suggested additions to the Done-when:
The engine plans and runs each model under a start time its caller supplies, on DataFusion 55.2 (Cargo.toml:28 at a0b5e6c): it sets query_execution_start_time through SessionState::execution_props_mut(), plans with SessionState::create_logical_plan, wraps the plan in DataFrame::new on that same state and executes that DataFrame, never through SessionContext::sql or SessionContext::execute_logical_plan, which both take a fresh copy of the state, with its start time reset to the wall clock, on every call. now(), current_timestamp and current_date in @check expressions (Row checks and quarantine tables #12) then fold to values taken from the job's start time, which a rerun needs to repeat the split between target and quarantine (docs/adr/0006-ordered-job-commits.md:71-72); storing that time with the job id and supplying it again on a rerun belongs with the runner (In-process runner: stage, run tasks, commit once #13), under the rule docs: revise RFC 0001 with the incremental rules (decision D2) #73 adds to RFC 0001.
Test: a query selecting now() and current_date, planned twice under the same supplied start time with a pause in between, returns identical values both times, and now() equals the supplied start time.
Nothing in the engine relies on the order rows arrive in or on _loaded_at: scan_at hands every data file to one iceberg reader at its default concurrency (crates/storage/src/lib.rs:644-647), so rows from different files can arrive in a different order on each read, and _loaded_at is one wall-clock reading per read batch (crates/connectors/src/csv.rs:207-211, :228). Tests compare sorted rows, and a step that needs an order (Row checks and quarantine tables #12, Merge rows on @unique_key #35) uses the RFC 0001 ordering key from docs: revise RFC 0001 with the incremental rules (decision D2) #73: Raw snapshot sequence number, then _source_file, then _source_row.
Work that moved to new tickets:
engine: add a StreamingTable delta provider over scan_range #85: reads each input of a model run through a StreamingTable over Warehouse::scan_range for a pinned snapshot range, registered under the same layer name, instead of through a provider that reads the table head.
Activity
[-]Run a model with DataFusion and write its table[/-][+]Run a model with DataFusion and append to its table[/+]codecapo commentedon Oct 30, 2026
Spike findings (engine)
A spike proved this issue end to end, including a 1B-row run (raw to clean to ready in 17.3 min, 5.7 GB peak RSS, every count and sum exact).
CatalogProviderthat wrapsIcebergCatalogProviderand maps names throughLayer::from_namespacemakesraw.ordersandbronze.ordersresolve to one table. Look tables up lazily rather than snapshotting the list.INSERT INTO. That streams through IcebergWriteExec and makes one commit. Don't go throughWarehouse::append, which needs every batch in memory.CREATE TABLE AS. It silently drops rows: the upstreamensure_table_is_emptyonly checks partition 0. Reject CTAS in model SQL. To be reported upstream.cast(x as timestamp)gives nanoseconds, which needs Iceberg format v3. Normalise outputs to microseconds.Points: 8 -> 5. Added two criteria below.
codecapo commentedon Nov 6, 2026
The design review of 2026-11-01 touches this issue. It asks the engine to plan every model under one start time per job and never to depend on the order rows arrive in, while reading pinned input ranges, the new
_source_rowcolumn and the RFC text move to new tickets.Suggested additions to the Done-when:
Cargo.toml:28at a0b5e6c): it setsquery_execution_start_timethroughSessionState::execution_props_mut(), plans withSessionState::create_logical_plan, wraps the plan inDataFrame::newon that same state and executes thatDataFrame, never throughSessionContext::sqlorSessionContext::execute_logical_plan, which both take a fresh copy of the state, with its start time reset to the wall clock, on every call.now(),current_timestampandcurrent_datein@checkexpressions (Row checks and quarantine tables #12) then fold to values taken from the job's start time, which a rerun needs to repeat the split between target and quarantine (docs/adr/0006-ordered-job-commits.md:71-72); storing that time with the job id and supplying it again on a rerun belongs with the runner (In-process runner: stage, run tasks, commit once #13), under the rule docs: revise RFC 0001 with the incremental rules (decision D2) #73 adds to RFC 0001.now()andcurrent_date, planned twice under the same supplied start time with a pause in between, returns identical values both times, andnow()equals the supplied start time._loaded_at:scan_athands every data file to one iceberg reader at its default concurrency (crates/storage/src/lib.rs:644-647), so rows from different files can arrive in a different order on each read, and_loaded_atis one wall-clock reading per read batch (crates/connectors/src/csv.rs:207-211,:228). Tests compare sorted rows, and a step that needs an order (Row checks and quarantine tables #12, Merge rows on @unique_key #35) uses the RFC 0001 ordering key from docs: revise RFC 0001 with the incremental rules (decision D2) #73: Raw snapshot sequence number, then_source_file, then_source_row.Work that moved to new tickets:
StreamingTableoverWarehouse::scan_rangefor a pinned snapshot range, registered under the same layer name, instead of through a provider that reads the table head._source_row, each row's 1-based position within its file, to Raw tables; models read it like any other column.Source: design review of 2026-11-01, findings C15 and C17, with section 2.5.