Repository navigation
feat: abstract Iceberg file discovery with FileScanTaskProvider - #113
Merged
gabotechs merged 11 commits intoNov 19, 2026
Merged
Conversation
Give task-provider construction its context up front and return the projected schema with the configured eager or lazy provider. Move eager collection to the consuming into_eager method. Apply session reader settings in IcebergDataSource::builder, preserve explicit reader overrides, and remove build_source. Direct provider reconstruction remains synchronous and table-free. Validate constructor settings in both modes and explicit-reader precedence without a current runtime. No transforms, factories or statistics changes.
gabotechs
reviewed
Nov 18, 2026
gabotechs
left a comment
Contributor
There was a problem hiding this comment.
@alexanderbianchi this looks really good!
If we tidy up the tests, I think we can go for this
Use the existing taxi harness for SQL snapshots and focused source-composition checks. Remove the custom fixture and I/O recorder. Keep composable configuration and execute the displayed physical plan.
alexanderbianchi
force-pushed
the
prototype/pr109-table-free-source
branch
from
November 18, 2026 23:17
db81b50 to
c30de72
Compare
Report retained task predicates in eager EXPLAIN output and label both providers. Update plan snapshots and remove the manual reconstruction/thread smoke test. Discovery and reading are unchanged.
alexanderbianchi
marked this pull request as ready for review
November 19, 2026 00:14
Make the provider a required constructor argument rather than an optional builder field. Remove the builder setter and missing-provider runtime error, document the compile-time requirement, and check replacement fetch preservation with a plan snapshot.
This was referenced Nov 19, 2026
Preserve main's snapshot estimates through task providers without restoring a Table dependency in the common source. Retain eager conversion estimates and bound fetch statistics per partition. Merge constructor, predicate and limit fixes while keeping the provider API and EXPLAIN labels.
gabotechs
approved these changes
Nov 19, 2026
gabotechs
left a comment
Contributor
There was a problem hiding this comment.
Really nice work! left some suggestions, but none blocking. Thanks @alexanderbianchi.
alexanderbianchi
force-pushed
the
prototype/pr109-table-free-source
branch
from
November 19, 2026 16:00
a7488a0 to
dee7db1
Compare
Contributor
|
Massive work! Thank you for seeing this through |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR supersedes #67.
Planning before execution
Add opt-in eager file planning with
iceberg.eager_file_planning=true. Both catalog and static providers selectFileScanTasks duringTableProvider::scan(), then expose up totarget_partitionslocal output partitions. Lazy planning remains the default.Eager planning performs metadata I/O and retains the selected work before execution, including for queries that stop early. It does not read Parquet data during planning. Statistics remain unknown, and this PR makes no performance claim.
Architecture
Use one
IcebergDataSourcefor reading, with aFileScanTaskProviderextension point for how file tasks arrive:The source itself is table-free: only lazy discovery retains a
Table. Eager planning consumes that discovery throughinto_eager(), while sources can also be reconstructed synchronously from supplied file groups and a process-local reader.Builders apply session configuration and select eager/lazy planning. Custom task providers can use the same source—for example, DFD's work-unit feeds could deliver tasks locally or from a coordinator without separate data-source implementations. The DFD port and statistics remain separate work.
Compared to #109
I like #109's shared data source and task-provider split. This version takes that separation a little further:
try_with_eagerly_planned_file_scan_taskssetter; discovery and eager conversion belong to task providers.Table, and snapshot, projection and predicate settings belong to discovery rather than source construction.collected_file_scan_tasksin the source builder. It accepts a task provider directly. Planned file groups stay inside the eager provider instead of leaking its representation into source construction.into_eager()consumes the lazy provider's task stream, reusing snapshot, projection and filter setup instead of maintaining a second discovery path.eager_file_planning. They use the same builder chain in either mode; the task-provider builder handles the configuration choice.