Skip to content

feat: add support for skipping partitions using dynamic filters - #869

Open
Perseus wants to merge 15 commits into
datafusion-contrib:mainfrom
Perseus:anirudh.singh/iceberg-dyn-filtering-3
Open

Perseus wants to merge 15 commits into
datafusion-contrib:mainfrom
Perseus:anirudh.singh/iceberg-dyn-filtering-3

Conversation

@Perseus

@Perseus Perseus commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Note

This is part of a stack of PRs that implement dynamic filtering for Iceberg.
The previous PRs can be found here - #850 , #855
Please review this PR with this diff - https://github.com/datafusion-contrib/datafusion-distributed/pull/869/changes/1d7e7cc6755589b3aed84a2a3f1ba151b9f85719..2f771cac0fe91d8f0c487784596dfed51c1fe2de
Relevant changes will also be ported to datafusion-iceberg.

Summary

This PR

  • Adds PhysicalExpr -> Iceberg::Predicate translation
  • Uses dynamic filters to skip processing data files based on the partition key(s)

The translator changes have also been raised in datafusion-iceberg here - apache/datafusion-iceberg#106. If that merges after this PR, I will raise a follow-up to delete the code and use the public methods exposed there.

Some of the code in this PR duplicates functionality from iceberg-rust due to those APIs not being public. I've opened an issue here - apache/iceberg-rust#3675 to expose the required functionality and we can delete this code once that is done.

Details

The previous PR #855 added support for storing filters that DF pushes down into the data source.
The changes in this PR use the stored dynamic filters to skip FileScanTasks if the filters are on Iceberg partition columns, and if the given FileScanTask does not satisfy those filters.

PhysicalExpr -> Iceberg::Predicate

Before we're able to use the filters provided by DF, we need to convert them into an iceberg::Predicate so that iceberg-rust can understand and evaluate them. This is handled in dynamic_filters/physical_predicate.rs‎.

The conversions follow a few rules -

  • If a DataFusion filter cannot be represented by an iceberg::Predicate, we drop the filter
  • If DataFusion's expression evaluation semantics disagree with Iceberg or iceberg-rust's implementation, we drop the filter
  • Conversions are allowed to widen, i.e, admit more rows than DF's filter would allow, if the expression can only be partially represented as an iceberg::Predicate

Dropping a filter means that it is not added to the conjunction of filters that are used for pruning data. This means that iceberg-rust can return more rows than DF's filter evaluation would - this is okay because the datasources return PushedDown::No in try_pushdown_filters, which means that DF will retain its FilterExec node over the scan and filter out any additional rows.

Pruning

The new DynamicFilterPruner struct at datafusion/distributed-iceberg/src/common/dynamic_filters/pruner.rs is the central place where the filters and tasks are evaluated to prune data.

It keeps track of dynamic filters, caching their snapshots based on their generation (which is the counter that DF increments every time it updates a dynamic filter).

This PR only implements pruning FileScanTasks based on partition columns. If a dynamic filter contains a column that was used in the partitioning strategy for an Iceberg table, we can use that filter to skip a FileScanTask that does not contain data that the filter can match.

To evaluate the filter, we need to follow a few steps -

Check whether the current FileScanTask has a partition spec that uses that column

Iceberg allows changing the partition spec of a table without rewriting existing data; so there can be some data files that have partition spec A and some that have partition spec B. To decide whether a FileScanTask can be excluded based on the partition filter, we need to look at it's spec and check if it uses that column or not.

Convert the filter expression to use the partition spec's filter

Iceberg's partition specs aren't limited to just column values like PARTITION BY x; they can also have expressions. Common examples are bucketing PARTITION BY bucket(x, 4) or datetime conversions PARTITION BY day(ts).

Data files contain the raw values of these columns, while the evaluation of the partition filter needs to happen against the table's defined partition expression that's stored in the table's metadata.

@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch 3 times, most recently from 77858c6 to c59f43e Compare November 11, 2026 14:35

@Perseus Perseus Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire file should be replaced with an API call to iceberg-rust like this - apache/iceberg-rust#3675

@alexanderbianchi

Copy link
Copy Markdown
Collaborator

I had my AI do a in depth review checking iceberg rust and datafusion behavior because really most of this PR is just the predicate conversions.

Adds a translator from a DataFusion physical filter to an iceberg
predicate. It handles comparisons with a value, IN lists, IS NULL, LIKE
prefixes, isnan, starts_with, AND, OR, NOT and CASE. It also handles the
filters hash joins build while the query runs: IN lists, struct(a, b) IN
(...) for joins on several columns, and the CASE a partitioned join builds
with one branch per partition.

The predicate matches every row the filter keeps, and can match more, so
the filter still runs on every row. An AND can drop a side it can't
convert. A NOT is converted only when its inside was converted fully.

Floats and times aren't converted, and neither are LIKE prefixes with
non-ASCII text. iceberg-rust handles these in ways that could skip
matching rows or fail the read.

Nothing calls it yet. Partition skipping will use it next.
…ters

The converter only depends on DataFusion and iceberg, so it belongs with
the shared helpers. The partition skipper in common/ no longer imports
from table_provider/.
The partition skipper now reports files_pruned_by_partition, which shows
how many files it checked and how many it kept.
Runs joins and top-N queries over identity, truncate, month, bucket and
day partitions, with dynamic filtering on and off, and checks they
return the same rows. The cases use keys on partition edges, keys in no
partition, and nulls. A snapshot shows how many files each case kept.
Explain the two forms of CASE, what LIKE patterns convert, and why only
ASCII prefixes are converted.
…pper

The skipper becomes DynamicFilterPruner, with one prune method that
returns the task to read, or None to skip it. Later uses of the dynamic
filters on each file can go in the same place.

The code copied from iceberg-rust moves to partition_filter.rs, behind
the same PartitionFilter API that iceberg-rust is adding. Once the
crate's iceberg-rust version has it, that file can be deleted and the
import changed.
DataFusion's dynamic filters use comparisons, IN lists, IS NULL,
IS NOT NULL, AND, OR and CASE. This removes the conversions for NOT,
NOT IN, LIKE, isnan, starts_with and bare boolean columns. Without
NOT, nothing reads the `exact` flag, so the `Converted` wrapper goes
too.

`datum` no longer converts partition values from int to long or from
float to double. Files written before a column was widened now hit
the error arm, so we don't skip them.
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 1b1af3d to 249fd05 Compare November 18, 2026 09:44
@alexanderbianchi

Copy link
Copy Markdown
Collaborator

cc: @jayshrivastava might be nice for you to review, most of the diff is the predicate conversions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants