Skip to content

feat: accept and store expressions (including dynamic filters) in Iceberg scan nodes - #855

Merged
alexanderbianchi merged 9 commits into
datafusion-contrib:mainfrom
Perseus:anirudh.singh/iceberg-dyn-filtering-2
Nov 18, 2026
Merged

alexanderbianchi merged 9 commits into
datafusion-contrib:mainfrom
Perseus:anirudh.singh/iceberg-dyn-filtering-2

Conversation

@Perseus

@Perseus Perseus commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Note

This is part of a stack of PRs that implement dynamic filtering for Iceberg. The previous PR can be found here - #850
Relevant changes will also be ported to datafusion-iceberg.

Summary

This PR adds support for accepting and storing expressions, including dynamic filters, in both Iceberg datasources (Eager and Lazy).

Based on a config, iceberg.dynamic_filtering_enabled, operators like HashJoinExec are also informed that the datasource is a consumer of these dynamic filters.

The dynamic filters also show up in EXPLAIN ANALYZE plans in the DataSource in the following format:

DataSourceExec: format=iceberg, table=default.taxi, projection=[trip_distance, pickup_date], predicate=trip_distance >= 2, dynamic_filters=DynamicFilter [ pickup_date@1 >= 2024-01-10 AND pickup_date@1 <= 2024-01-10 AND pickup_date@1 IN ([1 elements]) ]

These filters are not yet used for any optimizations - they're only stored and reported.

Future PRs will use them to improve data pruning in the following order -

  • Convert the dynamic filterPhysicalExpr to iceberg::Predicate
  • Use predicates containing partition keys to skip processing FileScanTasks
  • Use predicates containing other columns to skip row groups as soon as the filters become available (dependency on #3351 in iceberg-rust)

Changes

  • Both the Lazy and Eager datasources implement try_pushdown_filters - they extract and store the filters within the scan config, and report PushedDown::No to DF so that it continues apply its own filters
  • Adds a new iceberg-specific dynamic_filtering_enabled flag
  • Depending on the value of the flag, apply_expressions registers its use of dynamic filters, which in turn makes operators like HashJoinExec actually build them at runtime

@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-2 branch 5 times, most recently from 6bdc438 to 58786d0 Compare November 6, 2026 13:36
Comment thread iceberg/tests/limit_pushdown.rs Resolved
@@ -18,6 +18,8 @@ extensions_options! {
pub column_stats_enabled: bool, default = false
/// Discover and retain file tasks during physical planning instead of execution.
pub eager_file_planning: bool, default = false
/// (Experimental) Whether to enable dynamic filtering.
pub dynamic_filtering_enabled: bool, default = false

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was added so that a rolling deploy for workers doesn't cause errors if the coordinator starts producing dynamic filters that the workers don't consume yet - I'm not sure if that's necessarily an issue in practice and this adds something to the public API of this datasource that might not be easily removed later, so I'm open to changing the approach to something better

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have another mechanism for this, worker versioning. doesn't need to be done per pr like this. I think dynamic filtering should be configurable datafusion-wide. Although I don't mind a data source specific toggle.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are largely just checking that the plumbing works since the filters don't do anything yet

@Perseus
Perseus marked this pull request as ready for review November 7, 2026 17:23
DataFusion offers each scan the filters above it, including the dynamic
filters that joins, top-N sorts and aggregates fill in while the query runs.
Iceberg scans didn't keep them. Keeping them is the first step toward
skipping files with them.

Both scans now keep every pushed filter, ANDed into one predicate. They
answer that they don't apply them, so DataFusion still checks every row.
EXPLAIN shows the dynamic ones on the scan as dynamic_filters=, and the
codecs carry all of them to workers.

With iceberg.dynamic_filtering_enabled on, scans also report the filters
through apply_expressions. Joins and aggregates fill in a dynamic filter only
when a scan reports it, and the coordinator sends updates to the workers
holding those scans. The setting is off by default, and the plan carries it
to workers. Enable it only once every worker supports it: an older worker can
fail a query whose scan reports a dynamic filter.
…ltering is on

With iceberg.dynamic_filtering_enabled off, scans aren't consumers of the
dynamic filters pushed into them, so those filters stay empty. Plans no
longer print them in that case.
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-2 branch from d927948 to cab7f00 Compare November 10, 2026 10:21
A join filter can list thousands of keys, which filled the plan. An IN list
of more than 5 values now prints as its count, for example
`pickup_date@0 IN ([7 elements])`. Only the printed plan changes.
Every IN list now prints as its count, so a plan stays short however many
keys a join filter lists. The rewrite moves into fmt_dynamic_filters, and
the plan test for long lists goes away.
.collect::<Result<Vec<_>>>()
})
.collect::<Result<Vec<_>>>()?;
Ok(DataSourceExec::from_data_source(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably just allow passing the whole source.scan struct into this so it can pick the properties it needs, but from_file_groups is a public function and changing it might break things for consumers; I can do the refactor but since this is only for two properties right now, left it as-is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be reworked through the ongoing datafusion-iceberg workstream. agreed that seems to make sense though.

Removes the comment about decoding filters with the plan's converter, since
that is the usual way to decode them. The lazy codec test builds its filter
with DataFusion's binary and col helpers.
Comment on lines +113 to +131
if filter.snapshot_generation() == 1 {
write!(f, "DynamicFilter [ empty ]")?;
continue;
}
let current = snapshot_physical_expr(filter)
.and_then(|current| {
current.transform_up(|expr| match expr.downcast_ref::<InListExpr>() {
Some(in_list) => {
let not = if in_list.negated() { "NOT " } else { "" };
let text = format!(
"{} {not}IN ([{} elements])",
in_list.expr(),
in_list.len()
);
Ok(Transformed::yes(lit(text)))
}
None => Ok(Transformed::no(expr)),
})
})

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some testing with large tables, printing all the filters was resulting in a massive plan because of the number of elements being passed in the dynamic filter; so I made it print DynamicFilter[ 10 elements ] instead.

I'm also ok to remove this from the plan entirely, but it was useful while testing the changes in this PR so maybe there's a better way to represent them

@alexanderbianchi alexanderbianchi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We will want to port this to datafusion-iceberg once apache/datafusion-iceberg#67 this lands. for now there's no reason to delay adding features, but once we are closer to removing this data source we will pause development here to focus on porting to datafusion-iceberg.

.collect::<Result<Vec<_>>>()
})
.collect::<Result<Vec<_>>>()?;
Ok(DataSourceExec::from_data_source(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be reworked through the ongoing datafusion-iceberg workstream. agreed that seems to make sense though.

@alexanderbianchi
alexanderbianchi merged commit 82b547b into datafusion-contrib:main Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants