Repository navigation
Description
RFC 0001 lets a model rebuild its whole table on every run with @full_refresh: true (docs/rfcs/0001-project-files.md:121 at a0b5e6c). Its ready/daily_revenue example, a GROUP BY over clean.orders (:104-106), is only correct when rebuilt. Run over the delta, a second order of 20 for a day that already holds 100 adds the row (day, 20) beside (day, 100), because an incremental read returns only the new commit's rows (crates/storage/tests/iceberg_roundtrip.rs:77-79). Nothing implements the directive today: crates/compiler holds only directives() (crates/compiler/src/lib.rs:9-16), both Warehouse::append and append_if_head commit through fast_append (crates/storage/src/lib.rs:143-148, :205-211), and nothing public in the pinned iceberg-rust makes an overwrite (crates/storage/src/lib.rs:682-683).
This issue makes the directive work on top of replace_if_head from #88. The compiler accepts it, the model's job reads every input whole as of a pinned snapshot (from = None, which crates/model/src/lib.rs:79-80 defines as a full build), and the result replaces the target instead of being appended to it. The commit keeps ADR 0006: one atomic commit for the target that carries infostash.job-id and each input's last processed snapshot, and that fails rather than rebasing if the head moved since the job read it (docs/adr/0006-ordered-job-commits.md:39-40, :49-54). Recomputing only the changed days belongs to Phase 1 and its own RFC, so a full refresh always rebuilds the whole table. The work is needed only if #72 decides to build the replace commit. If #72 chooses an append-only 0.1.0 instead, #73 withdraws the directive from the RFC and this issue closes as not planned.
Crates: crates/compiler (accepting the directive), crates/engine (running the query over whole inputs and committing through replace_if_head), and the input range step of #13 in crates/jobs (pinning from = None)
Done when
- The compiler accepts
@full_refresh: true. The interim rejection from compiler: add an incremental eligibility classifier with a function allow-list #84 is lifted for this directive only (lifting it for@unique_keyis left to Merge rows on @unique_key #35), and such a model is checked against compiler: add an incremental eligibility classifier with a function allow-list #84's function rules but not its incremental node allow-list. So thedaily_revenuequery with the directive compiles, while a@full_refreshbody that calls aVolatilefunction such asrandom()still fails compilation. - Lifting the rejection leaves compiler: add an incremental eligibility classifier with a function allow-list #84's check over the DAG in place: an incremental model that reads a
@full_refreshtable still fails compilation until the plan-time check of engine: fall back to a full refresh at plan time and fix files_added_by #89 replaces that check. A test shows the error naming that input. - The job of a
@full_refreshmodel pins every input tofrom = Nonewith a pinnedto, soscan_rangereads each input whole as oftoand never callsappends_between(crates/storage/src/lib.rs:246-252). A test runs a@full_refreshmodel over another@full_refreshmodel twice, and the second run reads the replaced upstream table without hitting the refusal at:571-576. - The result never goes through an append, because appending a full recompute would duplicate every row already in the target. It replaces the target in one
replace_if_headcommit from storage: add the replace_if_head commit #88 that marks every live data file removed and adds the new files. That commit carries the job id and each input'stoas its checkpoint, in the same properties that the append commit of In-process runner: stage, run tasks, commit once #13 writes (docs/adr/0006-ordered-job-commits.md:39-40,:49-51). - The target is never dropped and recreated, which a downstream reader would see as
NotFound(crates/storage/src/lib.rs:566,:581-583), so its table UUID and snapshot history survive every full refresh. - A full refresh whose query returns no rows still commits a snapshot that removes every live file and carries the job id. It must not reuse the empty
fast_appendmarker ofdocs/adr/0006-ordered-job-commits.md:43-46, which would leave the old rows in place. A test over a target holding rows shows it empty afterwards. - Test: the
ready/daily_revenuequery (docs/rfcs/0001-project-files.md:104-106) with-- @full_refresh: trueruns over aclean.orderswithout@unique_key, so the test does not wait for Merge rows on @unique_key #35. It runs once after an order of 100 for 2026-10-29 and again after a second order of 20 for that day. The table then holds exactly one row (2026-10-29, 120), keeps the UUID of the first run, and still lists the first run's snapshot in its history.
Points: 2 (M3 @full_refresh 2)
Depends on: #72 (whether a replace commit is built at all), #73 (the @full_refresh text and the per-mode output rule in RFC 0001), #84 (the interim rejection this lifts, the function rules and the rule for incremental readers), #88 (the replace commit), #7 (the directive parser), #13 (the input range step and the model commit path)
Blocks: #8 (its Ready model, ready/daily_revenue), #16 (the quick start queries gold.daily_revenue, README.md:42, and must get one row per day after its second run)
Source: design review of 2026-11-01, findings C02 and C04, with finding C01 for the replace commit and C03 for the function rules (points from section 8.5, M3).
Activity
docs: revise RFC 0001 with the incremental rules (#101)
refactor(storage): keep live_files out of the public API
feat(storage): replace live data files in one overwrite snapshot (#106)