Skip to content

feat: abstract Iceberg file discovery with FileScanTaskProvider - #113

Merged
gabotechs merged 11 commits into
apache:mainfrom
alexanderbianchi:prototype/pr109-table-free-source
Nov 19, 2026
Merged

gabotechs merged 11 commits into
apache:mainfrom
alexanderbianchi:prototype/pr109-table-free-source

Conversation

@alexanderbianchi

@alexanderbianchi alexanderbianchi commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

This PR supersedes #67.

Planning before execution

Add opt-in eager file planning with iceberg.eager_file_planning=true. Both catalog and static providers select FileScanTasks during TableProvider::scan(), then expose up to target_partitions local output partitions. Lazy planning remains the default.

                           Iceberg pruning + delete matching
Lazy:   execute ──────────► task stream ─────────────────────► reader
Eager:  plan ─────────────► retained tasks → local partitions
        execute ───────────────────────────────────────────► reader

Eager planning performs metadata I/O and retains the selected work before execution, including for queries that stop early. It does not read Parquet data during planning. Statistics remain unknown, and this PR makes no performance claim.

Architecture

Use one IcebergDataSource for reading, with a FileScanTaskProvider extension point for how file tasks arrive:

                    IcebergDataSource
              shared reader, fetch, metrics
                            │
               Arc<dyn FileScanTaskProvider>
                  ┌─────────┴─────────┐
     LazyFileScanTaskProvider   EagerFileScanTaskProvider
     table + deferred discovery   planned file groups

The source itself is table-free: only lazy discovery retains a Table. Eager planning consumes that discovery through into_eager(), while sources can also be reconstructed synchronously from supplied file groups and a process-local reader.

Builders apply session configuration and select eager/lazy planning. Custom task providers can use the same source—for example, DFD's work-unit feeds could deliver tasks locally or from a coordinator without separate data-source implementations. The DFD port and statistics remain separate work.

Compared to #109

I like #109's shared data source and task-provider split. This version takes that separation a little further:

  1. Planning stays out of the data-source builder. No try_with_eagerly_planned_file_scan_tasks setter; discovery and eager conversion belong to task providers.
  2. Discovery state moves out of the source and its builder. The source no longer retains a Table, and snapshot, projection and predicate settings belong to discovery rather than source construction.
  3. No collected_file_scan_tasks in the source builder. It accepts a task provider directly. Planned file groups stay inside the eager provider instead of leaking its representation into source construction.
  4. Lazy feeds eager. into_eager() consumes the lazy provider's task stream, reusing snapshot, projection and filter setup instead of maintaining a second discovery path.
  5. Table providers don't branch on eager_file_planning. They use the same builder chain in either mode; the task-provider builder handles the configuration choice.

Give task-provider construction its context up front and return the projected schema with the configured eager or lazy provider. Move eager collection to the consuming into_eager method.

Apply session reader settings in IcebergDataSource::builder, preserve explicit reader overrides, and remove build_source. Direct provider reconstruction remains synchronous and table-free.

Validate constructor settings in both modes and explicit-reader precedence without a current runtime. No transforms, factories or statistics changes.

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexanderbianchi this looks really good!

If we tidy up the tests, I think we can go for this

Comment thread crates/datafusion/src/read/scan.rs Resolved
Comment thread crates/datafusion/src/read/scan.rs Resolved
Use the existing taxi harness for SQL snapshots and focused source-composition checks. Remove the custom fixture and I/O recorder. Keep composable configuration and execute the displayed physical plan.
@alexanderbianchi
alexanderbianchi force-pushed the prototype/pr109-table-free-source branch from db81b50 to c30de72 Compare November 18, 2026 23:17
Report retained task predicates in eager EXPLAIN output and label both providers. Update plan snapshots and remove the manual reconstruction/thread smoke test. Discovery and reading are unchanged.
@alexanderbianchi
alexanderbianchi marked this pull request as ready for review November 19, 2026 00:14
Make the provider a required constructor argument rather than an optional builder field. Remove the builder setter and missing-provider runtime error, document the compile-time requirement, and check replacement fetch preservation with a plan snapshot.
Preserve main's snapshot estimates through task providers without restoring a Table dependency in the common source. Retain eager conversion estimates and bound fetch statistics per partition. Merge constructor, predicate and limit fixes while keeping the provider API and EXPLAIN labels.

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice work! left some suggestions, but none blocking. Thanks @alexanderbianchi.

Comment thread crates/datafusion/src/read/scan.rs Resolved
@alexanderbianchi
alexanderbianchi force-pushed the prototype/pr109-table-free-source branch from a7488a0 to dee7db1 Compare November 19, 2026 16:00
@gabotechs
gabotechs merged commit f6a6baf into apache:main Nov 19, 2026
5 checks passed
@NoahKusaba

Copy link
Copy Markdown
Contributor

Massive work! Thank you for seeing this through

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants