Skip to content

feat: abstract Iceberg file discovery behind a FileScanTaskProvider trait - #109

Closed
gabotechs wants to merge 1 commit into
apache:mainfrom
gabotechs:file-scan-task-provider
Closed

gabotechs wants to merge 1 commit into
apache:mainfrom
gabotechs:file-scan-task-provider

Conversation

@gabotechs

@gabotechs gabotechs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Alternative to #67 for supporting both lazy and eager execution modes.

The idea here is to, rather than using the datafusion::DataSource as the trait that allows switching lazy and eager execution mode, to have a single DataSource implementation and create a new extension point hosted in this project for customizing FileScanTask streaming.

                      IcebergDataSource
        reader, limit, metrics, EXPLAIN prefix, partitioning
                              │
                 Arc<dyn FileScanTaskProvider>
                    ┌─────────┴──────────┐
     LazyFileScanTaskProvider      EagerFileScanTaskProvider
     discovers files on first      file groups planned up front,
     poll, 1 partition             N partitions, memory reserved

This way, enabling eager planning is just a matter of switching the trait FileScanTaskProviderimplementation. This new trait should not only serve as the granular extension point for abstracting lazy VS eager behavior over the same singleIcebergDataSource, but also serve as a public extension point for people to inject their own custom FileScanTask` stream providers.

For example, in DFD, we have the IcebergWorkUnitFeed that can seamlessly stream FileScanTasks either in-memory in singe-node or over the wire through a coordinator->worker connection. We could make IcebergWorkUnitFeed implement FileScanTaskProvider and finally stopping to host our custom DataSource implementations in DFD.

/// (source.file_scan_task_provider() as &dyn Any).downcast_ref::<EagerFileScanTaskProvider>()
/// }
/// ```
pub trait FileScanTaskProvider: Any + Debug + Send + Sync {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexanderbianchi this could be hour hooking point in DFD for injecting our WorkUnitFeed, so the idea here is to kill two birds with one stone:

  • providing polymorphism that supports both eager and lazy execution modes
  • provide a public hooking point for injecting custom FileScanTask streams, like DFD's WorkUnitFeed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that was the idea in #107. Similar API, I like yours returning the partitioning instead of the count more. I don't know I'd considered it also a solution for the eager path (or if I did, I had forgotten 😆)

pub trait FileScanTaskSource: Clone + Debug + Send + Sync + 'static {
    /// Number of local output partitions, not a worker or distributed task count.
    fn partition_count(&self) -> usize;

    /// Opens a partition's stream during execution, without collecting it.
    ///
    /// Discovery can start here or when the stream is polled. Calls on one
    /// instance may coordinate discovery; a repeated claim of the same partition
    /// may be rejected. Implementations must not block waiting for tasks here.
    fn tasks(
        &self,
        partition: usize,
        context: Arc<TaskContext>,
    ) -> Result<FileScanTaskStream>;

    /// Appends task-source details to the data source's explain output.
    fn fmt_as(&self, _f: &mut Formatter<'_>) -> std::fmt::Result {
        Ok(())
    }
}

@gabotechs
gabotechs force-pushed the file-scan-task-provider branch 2 times, most recently from 0e38c7f to 715afa7 Compare November 17, 2026 14:23
@gabotechs
gabotechs force-pushed the file-scan-task-provider branch from 715afa7 to e6583d2 Compare November 17, 2026 14:32
let iceberg_cfg = IcebergDataFusionConfig::from_config_or_default(state.config());
if iceberg_cfg.execution.eager_file_planning {
builder = builder
.try_with_eagerly_planned_file_scan_tasks(state)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just be passing IcebergDataFusionConfig into the builder instead of if/else in each table provider? For example ArrowOptions I imagine will also need to be passed in, as would most options that IcebergDataFusionConfig eventually has.


pub struct FileScanTaskProviderContext<'a> {
pub task_context: &'a Arc<TaskContext>,
pub table: &'a Table,

@alexanderbianchi alexanderbianchi Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like if we also had a way to construct from not a table.

IMO there should be a way to pass in just FileScanTask and construct a data source,

  • FileScanTask is a public API in iceberg rust
  • Comet also passes FileScanTasks directly to DF, bypassing the Iceberg Catalog->Table->manifset planning path.
  • The Table in iceberg rust contains a lot. (FileIO, Cache, runtime)
  • It's needed for reconstruction, although I amm supportive of offering one codec from this library that could be shared between DFD / Ballista.

@alexanderbianchi alexanderbianchi left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good improvement and should be the foundation of the eager datasource + increase composability of the lazy path. Still some details to hash out that I think can be iterated on safely.

if iceberg_cfg.execution.eager_file_planning {
builder = builder
.try_with_eagerly_planned_file_scan_tasks(state)
.await?;

@alexanderbianchi alexanderbianchi Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just make build() async instead of having an async configuration? IMO more clear, and if anything in the builder impacts the manifest pruning it'll be available in Build() and we won't have to worry about the "state" at the time try_with_eagerly_planned_file_scan_tasks. Alternatively we can have a build_eagerly() or something like it that's async, but always called at the end. For example .with_snapshot_id(...) impacts try_with_eagerly_planned_file_scan_tasks so ordering in the builder matters.

/// Discovers files while executing, in a single partition: the files of a
/// snapshot that may match a predicate.
#[derive(Debug, Clone)]
pub struct LazyFileScanTaskProvider {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might be able to push this even further

  struct LazyFileScanTaskProvider {
      table: Table,
      snapshot_id: Option<i64>,
      projection: Vec<String>,
      predicate: Option<Predicate>,
  }

and

  struct EagerFileScanTaskProvider {
      groups: Vec<Vec<FileScanTask>>,
  }

Such that we can remove some of the logic from the data source, and it's builder.

   let discovery = Arc::new(LazyFileScanTaskProvider {
       table,
       snapshot_id,
       projection,
       predicate,
   });

   let provider: Arc<dyn FileScanTaskProvider> =
       if iceberg_config.execution.eager_file_planning {
           Arc::new(
               EagerFileScanTaskProvider::plan(discovery, context).await?,
           )
       } else {
           discovery
       };

   let source = IcebergDataSourceBuilder::new(schema, file_io, reader)
       .with_file_scan_task_provider(provider)
       .with_fetch(limit)
       .build()?;

Something like this, isolates the async out of the data source builder and removes the Table from the data source.

@alexanderbianchi

Copy link
Copy Markdown
Contributor

Playing around with this approach + my comments here #113

snapshot_id: Option<i64>,
predicate: Option<Predicate>,
fetch: Option<usize>,
collected_file_scan_tasks: Option<Vec<Vec<FileScanTask>>>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is kinda an abstraction leak from the FileScanTaskProvider. We don't need the Table after we have the fileScanTasks and the Table itself is needed only for Lazy.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I skimmed this code and it makes sense to me, though I am not a super expect

@alexanderbianchi 's comments also look spot on -- I think they are worth addressing before merge

/// instead, with fresh metrics. The
/// tasks must read this source's projected columns.
pub fn with_file_scan_task_provider(
&self,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend taking this by value rather than forcing a clone

if the user wants a clone, thy could call .clone() explicitly

@gabotechs

Copy link
Copy Markdown
Contributor Author

@alamb, sorry, I should have closed this, we merged an ironed out version of this in #113, so this one is no longer needed.

Thanks for taking a look though!

@gabotechs gabotechs closed this Nov 19, 2026
@alamb

alamb commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

that is great news

I was just trying to help move thing along in this repo -- let me know if you need some other high level reviews

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants