Repository navigation
Description
An incremental model's read fails, and nothing falls back, when an input's range cannot be read as appends. appends_between refuses an overwrite, delete or replace snapshot in (from, to] with a generic Error::Storage (crates/storage/src/lib.rs:570-576 at a0b5e6c), and a from lost to snapshot expiry fails with the same NotFound variant as a snapshot that was never an ancestor (:564-566, :581-583). Nothing public in the pinned iceberg-rust can write a non-append snapshot today, which is why the test an_overwrite_in_the_range_is_refused adds its overwrite to the metadata by hand (:682-683). Once #88 can replace a table, though, every incremental model that reads a merged (#35) or @full_refresh (#90) table would fail on its first run after that table's next merge or rebuild, so #84 refuses to compile such a model in the meantime. This ticket moves that check from compile time to plan time, run by run. Before a job is created, the scheduler checks each input's range, and where the read would fail it plans a full refresh: a new job that rebuilds the target and replaces it in one commit carrying its job id. That keeps the ADR 0006 rules that a job makes one commit per output table and that a rerun reads the same pinned input snapshots (docs/adr/0006-ordered-job-commits.md:39-40, :71-72). This ticket also fixes files_added_by, which skips every manifest written outside the range and keeps only Added entries (crates/storage/src/lib.rs:602-605, :612-613). As soon as any rule lets a delta read step over a replace, that would silently drop the rows of a file that a compaction or manifest rewrite carries forward as an Existing entry. The fallback rule itself is recorded in the short ADR that #72 writes after the replace-commit decision, and #75 sets the retention rule that keeps truncated history rare. If #72 chooses an append-only 0.1.0, this ticket waits until a replace commit exists.
Crates: crates/jobs, crates/engine, crates/storage, crates/model
Done when
- Before In-process runner: stage, run tasks, commit once #13's scheduler creates a job, it reads the operation of every snapshot in each input's
(from, to]from table metadata alone, through one newcrates/storagefunction that is the only new storage API here, and if any range holds a snapshot that is not anappend, or its history is truncated beforefrom, it plans a full refresh and reports the input, the snapshot and the reason. - A full refresh is a new job with newly pinned inputs, never a rerun under a failed job's id (
docs/adr/0006-ordered-job-commits.md:47-48,:71-72). It runs the model (Run a model with DataFusion and append to its table #10) withfrom = Nonefor every input, because the rebuilt target must equal the model evaluated over each input as of itsto. It replaces the target rather than appending to it, throughreplace_if_headfrom storage: add the replace_if_head commit #88 as a@full_refreshmodel (compiler: implement the @full_refresh directive #90) does, and that one commit carries its job id and per-input checkpoints (docs/adr/0006-ordered-job-commits.md:39-40,:49-51). - The snapshot walk that
appends_betweenand the new function share stops as soon as a snapshot's parent isfrom(crates/storage/src/lib.rs:579), and a missing snapshot other thantoreturns a newErrorvariant naming truncated history instead of theNotFoundat:564-566. The lookup-first order at:563and the non-ancestorNotFoundat:581-583stay as they are, and the new variant goes ininfostash_model::Error(crates/model/src/lib.rs:147-156). - The compile-time check over the DAG that compiler: add an incremental eligibility classifier with a function allow-list #84 adds (no incremental model may read a table declaring
@unique_keyor@full_refresh) is removed, and the refusal atcrates/storage/src/lib.rs:570-576stays as a guard, so a job that still reaches it ends asJobOutcome::Failed { retryable: false }(crates/model/src/lib.rs:140-143) instead of being retried the waycrates/dist/src/lib.rs:106retries any error. -
files_added_byselects, from every manifest thattolists, the live entries (statusAddedorExisting) whoseentry.snapshot_id()is in the range, instead of skipping each manifest written outside the range (crates/storage/src/lib.rs:602-605) and keeping onlyAddedentries (:612-613). - Test: when
tolists a manifest, written by a later snapshot, that carries a file added inside the range as anExistingentry,files_added_byreturns that file, and it returns noDeletedentry and no file added at or beforefrom. - Test: when an input is rebuilt by an overwrite (S6) and then appended to (S7) after a consumer's checkpoint (S5), the scheduler plans a full refresh for the consumer, and running it leaves each expected row in the target exactly once (the planning half can use an overwrite built by hand, as
crates/storage/src/lib.rs:682-703does). - Test: when expiry removes a consumer's checkpoint snapshot and the snapshot after it, reading the range returns the truncated-history error and the scheduler plans a full refresh, while removing only older snapshots, or the checkpoint alone with its child kept, lets the next run proceed incrementally.
Points: 5 (M3 plan-time fallback and the files_added_by fix 5)
Depends on: #72 (the replace-commit decision and the short ADR on the fallback rule), #84 (only the removal of its compile-time check over the DAG), #88 (the full-refresh commit only; the files_added_by fix and the appends_between changes can land before it), #13 (the scheduler that runs the plan-time check)
Blocks: nothing
Source: design review of 2026-11-01, findings C04 and C05 (points from section 8.5, M3), with finding C01 for the replace commit.
Activity
codecapo commentedon Nov 10, 2026
ADR 0009 (PR #103) relies on the truncated-history variant this ticket adds to
infostash_model::Error: a reader whosefromsnapshot was expired must get it, distinct from theNotFoundof a snapshot that was never an ancestor, and it is never retried. The ADR also names the test that expiry past a checkpoint gives this error and the planned fallback of ADR 0008.docs: record the replace commit route (ADR 0007, ADR 0008) (#100)
docs: revise RFC 0001 with the incremental rules (#101)
docs: add ADR 0009 amending 0006 on position, reruns and expiry (#103)
codecapo commentedon Nov 12, 2026
#115 covers the storage half: the truncated-history error variant, the walk that stops at the child of
from, andfiles_added_byreading live entries from every manifest, with the Existing-entry test and the storage side of the expiry test. The plan-time range function, the full refresh, the removal of the #84 compile-time check and the non-retryable outcome wait for the scheduler in #13.