You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Incremental scan: read rows added between two snapshots #11
Read only the rows added to a table between two snapshots. iceberg-rust has no incremental scan, so we walk the snapshot history and manifests ourselves.
Crate:crates/storage
Done when
A scan for a SnapshotRange returns only data files added after from up to to
from == None returns the whole table
Clear error if the range includes any snapshot whose operation is not append (a copy-on-write overwrite would otherwise double-count rows)
files_added_since already exists privately in crates/storage. Make it public as a SnapshotRange scan that returns FileScanTasks or a stream, not Vec<RecordBatch>. The engine wraps it in a DataFusion StreamingTable, registered as a per-job override so the model SQL is unchanged.
The overwrite error is required, not optional. For a copy-on-write merge (Merge rows on @unique_key #35), files_added_since reports the rewritten survivors as new rows and misses the replaced row. A consumer would double-count. Any snapshot in the range whose operation is not append must fail.
Activity
codecapo commentedon Oct 30, 2026
Spike findings (engine and merge)
files_added_sincealready exists privately incrates/storage. Make it public as aSnapshotRangescan that returnsFileScanTasks or a stream, notVec<RecordBatch>. The engine wraps it in a DataFusionStreamingTable, registered as a per-job override so the model SQL is unchanged.files_added_sincereports the rewritten survivors as new rows and misses the replaced row. A consumer would double-count. Any snapshot in the range whose operation is notappendmust fail.Points: 8 -> 3.