Skip to content

Incremental scan: read rows added between two snapshots #11

Closed

Description

@codecapo

Read only the rows added to a table between two snapshots. iceberg-rust has no incremental scan, so we walk the snapshot history and manifests ourselves.

Crate: crates/storage

Done when

  • A scan for a SnapshotRange returns only data files added after from up to to
  • from == None returns the whole table
  • Clear error if the range includes any snapshot whose operation is not append (a copy-on-write overwrite would otherwise double-count rows)
  • Clear error if from is not an ancestor of to (covers storage: read_since must fail for an unknown snapshot #30)
  • Returns FileScanTasks or a stream, not Vec<RecordBatch>, so the engine can scan it lazily
  • Tests: three appends, scan of the last two returns exactly their rows

Depends on: #6
Blocks: #13

Activity

  1. added this to the 0.1.0 milestone on Oct 29, 2026
  2. codecapo commented on Oct 30, 2026

    @codecapo
    OwnerAuthor

    Spike findings (engine and merge)

    • files_added_since already exists privately in crates/storage. Make it public as a SnapshotRange scan that returns FileScanTasks or a stream, not Vec<RecordBatch>. The engine wraps it in a DataFusion StreamingTable, registered as a per-job override so the model SQL is unchanged.
    • The overwrite error is required, not optional. For a copy-on-write merge (Merge rows on @unique_key #35), files_added_since reports the rewritten survivors as new rows and misses the replaced row. A consumer would double-count. Any snapshot in the range whose operation is not append must fail.
    • Also covers storage: read_since must fail for an unknown snapshot #30: an unknown snapshot id must fail rather than return the whole table.

    Points: 8 -> 3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions