Repository navigation
Conversation
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-4
branch
7 times, most recently
from
November 17, 2026 18:52
0162fe2 to
d5bec23
Compare
Perseus
marked this pull request as ready for review
November 17, 2026 23:20
Adds a translator from a DataFusion physical filter to an iceberg predicate. It handles comparisons with a value, IN lists, IS NULL, LIKE prefixes, isnan, starts_with, AND, OR, NOT and CASE. It also handles the filters hash joins build while the query runs: IN lists, struct(a, b) IN (...) for joins on several columns, and the CASE a partitioned join builds with one branch per partition. The predicate matches every row the filter keeps, and can match more, so the filter still runs on every row. An AND can drop a side it can't convert. A NOT is converted only when its inside was converted fully. Floats and times aren't converted, and neither are LIKE prefixes with non-ASCII text. iceberg-rust handles these in ways that could skip matching rows or fail the read. Nothing calls it yet. Partition skipping will use it next.
…ters The converter only depends on DataFusion and iceberg, so it belongs with the shared helpers. The partition skipper in common/ no longer imports from table_provider/.
The partition skipper now reports files_pruned_by_partition, which shows how many files it checked and how many it kept.
Runs joins and top-N queries over identity, truncate, month, bucket and day partitions, with dynamic filtering on and off, and checks they return the same rows. The cases use keys on partition edges, keys in no partition, and nulls. A snapshot shows how many files each case kept.
Explain the two forms of CASE, what LIKE patterns convert, and why only ASCII prefixes are converted.
…pper The skipper becomes DynamicFilterPruner, with one prune method that returns the task to read, or None to skip it. Later uses of the dynamic filters on each file can go in the same place. The code copied from iceberg-rust moves to partition_filter.rs, behind the same PartitionFilter API that iceberg-rust is adding. Once the crate's iceberg-rust version has it, that file can be deleted and the import changed.
DataFusion's dynamic filters use comparisons, IN lists, IS NULL, IS NOT NULL, AND, OR and CASE. This removes the conversions for NOT, NOT IN, LIKE, isnan, starts_with and bare boolean columns. Without NOT, nothing reads the `exact` flag, so the `Converted` wrapper goes too. `datum` no longer converts partition values from int to long or from float to double. Files written before a column was widened now hit the error arm, so we don't skip them.
The next commit gives the Iceberg reader the scan's dynamic filters, so it can skip row groups. The reader only uses the filter stored on each FileScanTask, and iceberg-rust has no way to change that filter. and_predicate works around this. It turns the task's filter back into an unbound predicate, ANDs the new predicate onto it, binds the result and builds a copy of the task with it. Nothing calls it yet. The copy sets each of the task's fields by hand. A snapshot test lists the task's fields, so it fails if iceberg-rust adds a field the copy would leave out.
With iceberg.dynamic_filtering_enabled on, the scan now gives the reader the newest dynamic filters as it starts each file. The reader uses them to skip row groups, and to drop the rows that don't match. This helps joins on columns that aren't partition columns, and top-N. The reader checks every row once per IN list value. When the IN lists hold more than 200 values in total, we leave them out and keep the min/max bounds. The reader reads a column the file doesn't store as NULL. So a filter on an identity partition column, or on a column added later with a default value, can drop every row of such a file. We accept this for now. Any error leaves the file's task as it was.
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-4
branch
from
November 18, 2026 09:44
d5bec23 to
f0b5fb3
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This is part of a stack of PRs that implement dynamic filtering for Iceberg. The earlier PRs are 850, 855 and 869.
Until they merge, this PR also shows their commits. Review the last two commits with this link: https://github.com/datafusion-contrib/datafusion-distributed/pull/889/changes/249fd05a8060d25de9b5249ff30ba8ecfa2d1dda..f0b5fb39e4e7a2860d674085c0ec80a1652476ce
Relevant changes will also be ported to datafusion-iceberg.
What this does
#869 uses dynamic filters to skip whole files whose partition value can't match. This PR also gives the filters to iceberg-rust's
ArrowReader, so it can skip row groups inside the files it reads. That helps joins on columns that aren't partition columns, and top-N queries.How it works
A
FileScanTaskdescribes one data file being read, along with filters thatArrowReaderuses to skip row groups.Just before
ArrowReaderopens each file, the scan takes the newest version of the dynamic filters:While adding the new dynamic filters onto the task's existing filters, if there are any errors ornew predicate, we bail early and allow the file to be read as-is. This can result in more rowsbeing read from storage, and DF will filter them out as part of the query execution.
What iceberg-rust is missing
Most of the code here works around two gaps in iceberg-rust.
Changing a task's filter (
task_filter.rs)iceberg-rust can't change a task's filter, and can't AND two bound filters. So we turn the task's filter back into an unbound one, AND ours onto it, bind the result, and copy the task with the new filter. We can delete this
once iceberg-rust has a way to change a task's filter. No PR exists for that yet.
IN list cap (
row_group_filter.rs)ArrowReaderchecks an IN list one value at a time, so long lists are slow. We count the values e filter. Above 200, we leave the lists out and keep the min/max bounds. DataFusion already limitseach join IN list to 150 values. But a partitioned join's filter ORs one list per partition, so 16 partitions can give 2,400 values. We use 200 because iceberg-rust's own row group checks ignore longer lists. We can removethe cap once
ArrowReaderchecks IN lists with a hash set.How to review
task_filter.rs. It's plumbing, so a skim is enough. Its tests check that a filtding again, and that the copy keeps every task field.DynamicFilterPruner::with_reader_filter, the config and README docs, and the integration tests.The integration tests in
tests/row_group_skipping.rsshow the behavior:the_reader_skips_row_groups_the_join_filter_rules_out: a join reads 1 of the 40 row groups, and bytes read drop more than 5x. Results match a run with dynamic filtering off.eager_planning_workers_give_the_reader_the_join_filter: workers get their files from the coorReader` the join filter.the_top_n_filter_reaches_files_that_start_later: once the top 3 are known,ArrowReaderdrops every row of the second file.the_reader_keeps_the_where_filter_too: adding the join filter keeps theWHEREfilter.Known issues
ArrowReaderhas bugs that can drop matching rows. They already affectWHEREfilters. With this, joins/top K/aggregates can hit them too. Each one has a fix in iceberg-rust:ArrowReaderreads the column as NULL, so a in the fileArrowReaderdrops matching rows. Files written outside Iceberg can store timestamps this way.iceberg.row_selection_enabledonAll of these PRs are still open, except 3507.
Benchmarks
TODO