Skip to content

feat: add support for parquet-level pruning using dynamic filters - #889

Open
Perseus wants to merge 17 commits into
datafusion-contrib:mainfrom
Perseus:anirudh.singh/iceberg-dyn-filtering-4
Open

Perseus wants to merge 17 commits into
datafusion-contrib:mainfrom
Perseus:anirudh.singh/iceberg-dyn-filtering-4

Conversation

@Perseus

@Perseus Perseus commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Note

This is part of a stack of PRs that implement dynamic filtering for Iceberg. The earlier PRs are 850, 855 and 869.
Until they merge, this PR also shows their commits. Review the last two commits with this link: https://github.com/datafusion-contrib/datafusion-distributed/pull/889/changes/249fd05a8060d25de9b5249ff30ba8ecfa2d1dda..f0b5fb39e4e7a2860d674085c0ec80a1652476ce
Relevant changes will also be ported to datafusion-iceberg.

What this does

#869 uses dynamic filters to skip whole files whose partition value can't match. This PR also gives the filters to iceberg-rust's ArrowReader, so it can skip row groups inside the files it reads. That helps joins on columns that aren't partition columns, and top-N queries.

How it works

A FileScanTask describes one data file being read, along with filters that ArrowReader uses to skip row groups.

Just before ArrowReader opens each file, the scan takes the newest version of the dynamic filters:

newest dynamic filters
 ├─ the file's partition value can't match   → skip the file (PR 869)
 └─ otherwise, leave out the IN lists if they have too many values
     └─ AND the rest onto the task's filter  → ArrowReader skips row groups and drops rows

While adding the new dynamic filters onto the task's existing filters, if there are any errors ornew predicate, we bail early and allow the file to be read as-is. This can result in more rowsbeing read from storage, and DF will filter them out as part of the query execution.

What iceberg-rust is missing

Most of the code here works around two gaps in iceberg-rust.

Changing a task's filter (task_filter.rs)

iceberg-rust can't change a task's filter, and can't AND two bound filters. So we turn the task's filter back into an unbound one, AND ours onto it, bind the result, and copy the task with the new filter. We can delete this
once iceberg-rust has a way to change a task's filter. No PR exists for that yet.

IN list cap (row_group_filter.rs)

ArrowReader checks an IN list one value at a time, so long lists are slow. We count the values e filter. Above 200, we leave the lists out and keep the min/max bounds. DataFusion already limitseach join IN list to 150 values. But a partitioned join's filter ORs one list per partition, so 16 partitions can give 2,400 values. We use 200 because iceberg-rust's own row group checks ignore longer lists. We can remove
the cap once ArrowReader checks IN lists with a hash set.

How to review

  1. f1d1660 adds task_filter.rs. It's plumbing, so a skim is enough. Its tests check that a filtding again, and that the copy keeps every task field.
  2. d5bec23 adds the IN list cap and wires everything in: DynamicFilterPruner::with_reader_filter, the config and README docs, and the integration tests.

The integration tests in tests/row_group_skipping.rs show the behavior:

  • the_reader_skips_row_groups_the_join_filter_rules_out: a join reads 1 of the 40 row groups, and bytes read drop more than 5x. Results match a run with dynamic filtering off.
  • eager_planning_workers_give_the_reader_the_join_filter: workers get their files from the coorReader` the join filter.
  • the_top_n_filter_reaches_files_that_start_later: once the top 3 are known, ArrowReader drops every row of the second file.
  • the_reader_keeps_the_where_filter_too: adding the join filter keeps the WHERE filter.

Known issues

ArrowReader has bugs that can drop matching rows. They already affect WHERE filters. With this, joins/top K/aggregates can hit them too. Each one has a fix in iceberg-rust:

Case What happens Fixed by
Identity partition column the file doesn't store ArrowReader reads the column as NULL, so a in the file iceberg-rust PR 3302 (issue 3298)
Column added later with a default value Same, for files written before the column existed iceberg-rust PR 3302 (issue 3298)
Column widened from int to long A filter value outside the int range drops every row of the fe iceberg-rust PR 3540 (issue 3122)
Timestamps stored in milliseconds ArrowReader drops matching rows. Files written outside Iceberg can store timestamps this way. iceberg-rust PR 3623 (issue 3617)
Page skipping on a widened column, with iceberg.row_selection_enabled on It can skip pages option is off by default. iceberg-rust PR 3507, which merged after our pinned version, and PR3521

All of these PRs are still open, except 3507.

Benchmarks

TODO

@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch 7 times, most recently from 0162fe2 to d5bec23 Compare November 17, 2026 18:52
@Perseus
Perseus marked this pull request as ready for review November 17, 2026 23:20
Adds a translator from a DataFusion physical filter to an iceberg
predicate. It handles comparisons with a value, IN lists, IS NULL, LIKE
prefixes, isnan, starts_with, AND, OR, NOT and CASE. It also handles the
filters hash joins build while the query runs: IN lists, struct(a, b) IN
(...) for joins on several columns, and the CASE a partitioned join builds
with one branch per partition.

The predicate matches every row the filter keeps, and can match more, so
the filter still runs on every row. An AND can drop a side it can't
convert. A NOT is converted only when its inside was converted fully.

Floats and times aren't converted, and neither are LIKE prefixes with
non-ASCII text. iceberg-rust handles these in ways that could skip
matching rows or fail the read.

Nothing calls it yet. Partition skipping will use it next.
…ters

The converter only depends on DataFusion and iceberg, so it belongs with
the shared helpers. The partition skipper in common/ no longer imports
from table_provider/.
The partition skipper now reports files_pruned_by_partition, which shows
how many files it checked and how many it kept.
Runs joins and top-N queries over identity, truncate, month, bucket and
day partitions, with dynamic filtering on and off, and checks they
return the same rows. The cases use keys on partition edges, keys in no
partition, and nulls. A snapshot shows how many files each case kept.
Explain the two forms of CASE, what LIKE patterns convert, and why only
ASCII prefixes are converted.
…pper

The skipper becomes DynamicFilterPruner, with one prune method that
returns the task to read, or None to skip it. Later uses of the dynamic
filters on each file can go in the same place.

The code copied from iceberg-rust moves to partition_filter.rs, behind
the same PartitionFilter API that iceberg-rust is adding. Once the
crate's iceberg-rust version has it, that file can be deleted and the
import changed.
DataFusion's dynamic filters use comparisons, IN lists, IS NULL,
IS NOT NULL, AND, OR and CASE. This removes the conversions for NOT,
NOT IN, LIKE, isnan, starts_with and bare boolean columns. Without
NOT, nothing reads the `exact` flag, so the `Converted` wrapper goes
too.

`datum` no longer converts partition values from int to long or from
float to double. Files written before a column was widened now hit
the error arm, so we don't skip them.
The next commit gives the Iceberg reader the scan's dynamic filters, so it
can skip row groups. The reader only uses the filter stored on each
FileScanTask, and iceberg-rust has no way to change that filter.

and_predicate works around this. It turns the task's filter back into an
unbound predicate, ANDs the new predicate onto it, binds the result and
builds a copy of the task with it. Nothing calls it yet.

The copy sets each of the task's fields by hand. A snapshot test lists the
task's fields, so it fails if iceberg-rust adds a field the copy would
leave out.
With iceberg.dynamic_filtering_enabled on, the scan now gives the reader
the newest dynamic filters as it starts each file. The reader uses them to
skip row groups, and to drop the rows that don't match. This helps joins on
columns that aren't partition columns, and top-N.

The reader checks every row once per IN list value. When the IN lists hold
more than 200 values in total, we leave them out and keep the min/max
bounds.

The reader reads a column the file doesn't store as NULL. So a filter on an
identity partition column, or on a column added later with a default value,
can drop every row of such a file. We accept this for now.

Any error leaves the file's task as it was.
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch from d5bec23 to f0b5fb3 Compare November 18, 2026 09:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant