Repository navigation
Description
An incremental model run should read only the rows each input gained since its last run, and the storage crate already yields those rows as a lazy stream: Warehouse::scan_range returns the rows committed after range.from up to and including a pinned range.to, and refuses a range that holds a non-append snapshot or whose from is not an ancestor of to (crates/storage/src/lib.rs:231-262 at a0b5e6c). Nothing hands that stream to DataFusion yet: crates/engine is only a doc comment (crates/engine/src/lib.rs:1-2) with its DataFusion dependencies marked unused (crates/engine/Cargo.toml:13-16), and the other delta read, read_since, collects every row into a Vec and reads up to the current head without returning the to it used (crates/storage/src/lib.rs:264-282), so a runner built on it would hold the whole delta in memory and could not record the per-input checkpoint that ADR 0006 commits with the rows (docs/adr/0006-ordered-job-commits.md:49-51). This ticket puts scan_range behind a DataFusion StreamingTable, so a model's SQL reads its delta one batch at a time under the input's usual name, from a range the caller pinned, and a rerun given the same ranges reads the same rows, as ADR 0006 requires (docs/adr/0006-ordered-job-commits.md:71-72). Parallel scans and pushdown stay with #38, whose provider can take over delta reads when it lands; the eligibility classifier is #84, the fallback for an input that holds a non-append snapshot is #89, measuring the delta read at the Phase 0 target scale is #76, and storing each input's checkpoint and returning NoChange stay with #13.
Crates: crates/engine (the provider), crates/storage (a read-only accessor for a table's schema at a snapshot, because Warehouse::load is private at crates/storage/src/lib.rs:296; not needed if the public load() of PR #46 merges first)
Done when
-
crates/enginebuilds the delta provider from DataFusion'sStreamingTablewith onePartitionStream, without a newTableProvidertype: the partition holds anArc<Warehouse>with a hand-writtenDebug(Warehousederives none,crates/storage/src/lib.rs:82-85), the input'sTableRefand itsSnapshotRange, and its synchronousexecutereturns aRecordBatchStreamAdapterover thescan_rangefuture flattened into the stream it returns. - The provider reads exactly the
SnapshotRangeit is built with and never replacestowith the table's current head, so thetoit reads is the snapshot the caller pinned and will commit as that input's checkpoint under In-process runner: stage, run tasks, commit once #13. - The provider declares its schema before any row is read, as the table's schema at snapshot
toconverted withschema_to_arrow_schemathe waywrite_filesdoes (crates/storage/src/lib.rs:344-345), because the scan is pinned to that snapshot (crates/storage/src/lib.rs:629-630) and an empty delta yields no batches (crates/storage/src/lib.rs:255-256). - An error from
scan_range, including one returned before any stream exists, arrives as an item of the stream and fails the query with a DataFusion error that wraps the originalinfostash_model::Error, and a test shows a range whosefromis not an ancestor oftofailing with thatNotFound(crates/storage/src/lib.rs:581-583, as incrates/storage/tests/iceberg_roundtrip.rs:247-252) instead of returning rows. - The provider never collects the delta: it uses neither
read_since, which gathers every row into aVec(crates/storage/src/lib.rs:281), nor aMemTable, so the read side does not repeat whatpreparedoes on ingestion, where every batch is held until the connector is done (crates/connectors/src/lib.rs:79-84, which connectors: cap each job at a budget and load a larger backlog in chunks #77 bounds with a budget). - When the engine runs a model, it registers each input through this provider under the same layer-qualified name Run a model with DataFusion and append to its table #10 registers, or in the per-layer schema provider of PR feat(cli): infoctl query runs read-only SQL against the warehouse #46 if that has merged, instead of through a provider that reads the table head, so the model SQL is the same on a first run (
from = None, the whole table as ofto) and on every later run. - Test: over three appends, a
SELECT *over the input registered through the provider returns the same rows asscan_rangeover the sameSnapshotRange, compared after sorting so the test does not rely on row order, for a range that starts after the first append, for atopinned before the last append, and forfrom = None(the cases ofcrates/storage/tests/iceberg_roundtrip.rs:211-217). - Test: an empty range (
from == to) and a range holding only a 0-row snapshot, made withappend_if_head, no batches and a property as incrates/storage/src/lib.rs:906-914, each return zero rows with the table's columns.
Points: 8 (M2 StreamingTable delta provider over scan_range 8)
Depends on: #10
Blocks: #13 (per-input delta reads)
Source: design review of 2026-11-01, finding C03 (the StreamingTable route of section 2.5, points from section 8.5, M2), with finding C06 for the memory argument.
Activity