Skip to content

engine: fall back to a full refresh at plan time and fix files_added_by #89

Open

Description

@codecapo

An incremental model's read fails, and nothing falls back, when an input's range cannot be read as appends. appends_between refuses an overwrite, delete or replace snapshot in (from, to] with a generic Error::Storage (crates/storage/src/lib.rs:570-576 at a0b5e6c), and a from lost to snapshot expiry fails with the same NotFound variant as a snapshot that was never an ancestor (:564-566, :581-583). Nothing public in the pinned iceberg-rust can write a non-append snapshot today, which is why the test an_overwrite_in_the_range_is_refused adds its overwrite to the metadata by hand (:682-683). Once #88 can replace a table, though, every incremental model that reads a merged (#35) or @full_refresh (#90) table would fail on its first run after that table's next merge or rebuild, so #84 refuses to compile such a model in the meantime. This ticket moves that check from compile time to plan time, run by run. Before a job is created, the scheduler checks each input's range, and where the read would fail it plans a full refresh: a new job that rebuilds the target and replaces it in one commit carrying its job id. That keeps the ADR 0006 rules that a job makes one commit per output table and that a rerun reads the same pinned input snapshots (docs/adr/0006-ordered-job-commits.md:39-40, :71-72). This ticket also fixes files_added_by, which skips every manifest written outside the range and keeps only Added entries (crates/storage/src/lib.rs:602-605, :612-613). As soon as any rule lets a delta read step over a replace, that would silently drop the rows of a file that a compaction or manifest rewrite carries forward as an Existing entry. The fallback rule itself is recorded in the short ADR that #72 writes after the replace-commit decision, and #75 sets the retention rule that keeps truncated history rare. If #72 chooses an append-only 0.1.0, this ticket waits until a replace commit exists.

Crates: crates/jobs, crates/engine, crates/storage, crates/model

Done when

  • Before In-process runner: stage, run tasks, commit once #13's scheduler creates a job, it reads the operation of every snapshot in each input's (from, to] from table metadata alone, through one new crates/storage function that is the only new storage API here, and if any range holds a snapshot that is not an append, or its history is truncated before from, it plans a full refresh and reports the input, the snapshot and the reason.
  • A full refresh is a new job with newly pinned inputs, never a rerun under a failed job's id (docs/adr/0006-ordered-job-commits.md:47-48, :71-72). It runs the model (Run a model with DataFusion and append to its table #10) with from = None for every input, because the rebuilt target must equal the model evaluated over each input as of its to. It replaces the target rather than appending to it, through replace_if_head from storage: add the replace_if_head commit #88 as a @full_refresh model (compiler: implement the @full_refresh directive #90) does, and that one commit carries its job id and per-input checkpoints (docs/adr/0006-ordered-job-commits.md:39-40, :49-51).
  • The snapshot walk that appends_between and the new function share stops as soon as a snapshot's parent is from (crates/storage/src/lib.rs:579), and a missing snapshot other than to returns a new Error variant naming truncated history instead of the NotFound at :564-566. The lookup-first order at :563 and the non-ancestor NotFound at :581-583 stay as they are, and the new variant goes in infostash_model::Error (crates/model/src/lib.rs:147-156).
  • The compile-time check over the DAG that compiler: add an incremental eligibility classifier with a function allow-list #84 adds (no incremental model may read a table declaring @unique_key or @full_refresh) is removed, and the refusal at crates/storage/src/lib.rs:570-576 stays as a guard, so a job that still reaches it ends as JobOutcome::Failed { retryable: false } (crates/model/src/lib.rs:140-143) instead of being retried the way crates/dist/src/lib.rs:106 retries any error.
  • files_added_by selects, from every manifest that to lists, the live entries (status Added or Existing) whose entry.snapshot_id() is in the range, instead of skipping each manifest written outside the range (crates/storage/src/lib.rs:602-605) and keeping only Added entries (:612-613).
  • Test: when to lists a manifest, written by a later snapshot, that carries a file added inside the range as an Existing entry, files_added_by returns that file, and it returns no Deleted entry and no file added at or before from.
  • Test: when an input is rebuilt by an overwrite (S6) and then appended to (S7) after a consumer's checkpoint (S5), the scheduler plans a full refresh for the consumer, and running it leaves each expected row in the target exactly once (the planning half can use an overwrite built by hand, as crates/storage/src/lib.rs:682-703 does).
  • Test: when expiry removes a consumer's checkpoint snapshot and the snapshot after it, reading the range returns the truncated-history error and the scheduler plans a full refresh, while removing only older snapshots, or the checkpoint alone with its child kept, lets the next run proceed incrementally.

Points: 5 (M3 plan-time fallback and the files_added_by fix 5)

Depends on: #72 (the replace-commit decision and the short ADR on the fallback rule), #84 (only the removal of its compile-time check over the DAG), #88 (the full-refresh commit only; the files_added_by fix and the appends_between changes can land before it), #13 (the scheduler that runs the plan-time check)
Blocks: nothing

Source: design review of 2026-11-01, findings C04 and C05 (points from section 8.5, M3), with finding C01 for the replace commit.

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    ADR 0009 (PR #103) relies on the truncated-history variant this ticket adds to infostash_model::Error: a reader whose from snapshot was expired must get it, distinct from the NotFound of a snapshot that was never an ancestor, and it is never retried. The ADR also names the test that expiry past a checkpoint gives this error and the planned fallback of ADR 0008.

  3. codecapo commented on Nov 12, 2026

    @codecapo
    OwnerAuthor

    #115 covers the storage half: the truncated-history error variant, the walk that stops at the child of from, and files_added_by reading live entries from every manifest, with the Existing-entry test and the storage side of the expiry test. The plan-time range function, the full refresh, the removal of the #84 compile-time check and the non-retryable outcome wait for the scheduler in #13.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/enginecrates/enginearea/storagecrates/storageblockedWaiting on another issue or a decisionenhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions