Skip to content

engine: add a StreamingTable delta provider over scan_range #85

Open

Description

@codecapo

An incremental model run should read only the rows each input gained since its last run, and the storage crate already yields those rows as a lazy stream: Warehouse::scan_range returns the rows committed after range.from up to and including a pinned range.to, and refuses a range that holds a non-append snapshot or whose from is not an ancestor of to (crates/storage/src/lib.rs:231-262 at a0b5e6c). Nothing hands that stream to DataFusion yet: crates/engine is only a doc comment (crates/engine/src/lib.rs:1-2) with its DataFusion dependencies marked unused (crates/engine/Cargo.toml:13-16), and the other delta read, read_since, collects every row into a Vec and reads up to the current head without returning the to it used (crates/storage/src/lib.rs:264-282), so a runner built on it would hold the whole delta in memory and could not record the per-input checkpoint that ADR 0006 commits with the rows (docs/adr/0006-ordered-job-commits.md:49-51). This ticket puts scan_range behind a DataFusion StreamingTable, so a model's SQL reads its delta one batch at a time under the input's usual name, from a range the caller pinned, and a rerun given the same ranges reads the same rows, as ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:71-72). Parallel scans and pushdown stay with #38, whose provider can take over delta reads when it lands; the eligibility classifier is #84, the fallback for an input that holds a non-append snapshot is #89, measuring the delta read at the Phase 0 target scale is #76, and storing each input's checkpoint and returning NoChange stay with #13.

Crates: crates/engine (the provider), crates/storage (a read-only accessor for a table's schema at a snapshot, because Warehouse::load is private at crates/storage/src/lib.rs:296; not needed if the public load() of PR #46 merges first)

Done when

  • crates/engine builds the delta provider from DataFusion's StreamingTable with one PartitionStream, without a new TableProvider type: the partition holds an Arc<Warehouse> with a hand-written Debug (Warehouse derives none, crates/storage/src/lib.rs:82-85), the input's TableRef and its SnapshotRange, and its synchronous execute returns a RecordBatchStreamAdapter over the scan_range future flattened into the stream it returns.
  • The provider reads exactly the SnapshotRange it is built with and never replaces to with the table's current head, so the to it reads is the snapshot the caller pinned and will commit as that input's checkpoint under In-process runner: stage, run tasks, commit once #13.
  • The provider declares its schema before any row is read, as the table's schema at snapshot to converted with schema_to_arrow_schema the way write_files does (crates/storage/src/lib.rs:344-345), because the scan is pinned to that snapshot (crates/storage/src/lib.rs:629-630) and an empty delta yields no batches (crates/storage/src/lib.rs:255-256).
  • An error from scan_range, including one returned before any stream exists, arrives as an item of the stream and fails the query with a DataFusion error that wraps the original infostash_model::Error, and a test shows a range whose from is not an ancestor of to failing with that NotFound (crates/storage/src/lib.rs:581-583, as in crates/storage/tests/iceberg_roundtrip.rs:247-252) instead of returning rows.
  • The provider never collects the delta: it uses neither read_since, which gathers every row into a Vec (crates/storage/src/lib.rs:281), nor a MemTable, so the read side does not repeat what prepare does on ingestion, where every batch is held until the connector is done (crates/connectors/src/lib.rs:79-84, which connectors: cap each job at a budget and load a larger backlog in chunks #77 bounds with a budget).
  • When the engine runs a model, it registers each input through this provider under the same layer-qualified name Run a model with DataFusion and append to its table #10 registers, or in the per-layer schema provider of PR feat(cli): infoctl query runs read-only SQL against the warehouse #46 if that has merged, instead of through a provider that reads the table head, so the model SQL is the same on a first run (from = None, the whole table as of to) and on every later run.
  • Test: over three appends, a SELECT * over the input registered through the provider returns the same rows as scan_range over the same SnapshotRange, compared after sorting so the test does not rely on row order, for a range that starts after the first append, for a to pinned before the last append, and for from = None (the cases of crates/storage/tests/iceberg_roundtrip.rs:211-217).
  • Test: an empty range (from == to) and a range holding only a 0-row snapshot, made with append_if_head, no batches and a property as in crates/storage/src/lib.rs:906-914, each return zero rows with the table's columns.

Points: 8 (M2 StreamingTable delta provider over scan_range 8)

Depends on: #10
Blocks: #13 (per-input delta reads)

Source: design review of 2026-11-01, finding C03 (the StreamingTable route of section 2.5, points from section 8.5, M2), with finding C06 for the memory argument.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions