Repository navigation
feat: accept and store expressions (including dynamic filters) in Iceberg scan nodes - #855
Conversation
6bdc438 to
58786d0
Compare
| @@ -18,6 +18,8 @@ extensions_options! { | |||
| pub column_stats_enabled: bool, default = false | |||
| /// Discover and retain file tasks during physical planning instead of execution. | |||
| pub eager_file_planning: bool, default = false | |||
| /// (Experimental) Whether to enable dynamic filtering. | |||
| pub dynamic_filtering_enabled: bool, default = false | |||
There was a problem hiding this comment.
This was added so that a rolling deploy for workers doesn't cause errors if the coordinator starts producing dynamic filters that the workers don't consume yet - I'm not sure if that's necessarily an issue in practice and this adds something to the public API of this datasource that might not be easily removed later, so I'm open to changing the approach to something better
There was a problem hiding this comment.
We have another mechanism for this, worker versioning. doesn't need to be done per pr like this. I think dynamic filtering should be configurable datafusion-wide. Although I don't mind a data source specific toggle.
There was a problem hiding this comment.
These tests are largely just checking that the plumbing works since the filters don't do anything yet
DataFusion offers each scan the filters above it, including the dynamic filters that joins, top-N sorts and aggregates fill in while the query runs. Iceberg scans didn't keep them. Keeping them is the first step toward skipping files with them. Both scans now keep every pushed filter, ANDed into one predicate. They answer that they don't apply them, so DataFusion still checks every row. EXPLAIN shows the dynamic ones on the scan as dynamic_filters=, and the codecs carry all of them to workers. With iceberg.dynamic_filtering_enabled on, scans also report the filters through apply_expressions. Joins and aggregates fill in a dynamic filter only when a scan reports it, and the coordinator sends updates to the workers holding those scans. The setting is off by default, and the plan carries it to workers. Enable it only once every worker supports it: an older worker can fail a query whose scan reports a dynamic filter.
…ltering is on With iceberg.dynamic_filtering_enabled off, scans aren't consumers of the dynamic filters pushed into them, so those filters stay empty. Plans no longer print them in that case.
d927948 to
cab7f00
Compare
A join filter can list thousands of keys, which filled the plan. An IN list of more than 5 values now prints as its count, for example `pickup_date@0 IN ([7 elements])`. Only the printed plan changes.
Every IN list now prints as its count, so a plan stays short however many keys a join filter lists. The rewrite moves into fmt_dynamic_filters, and the plan test for long lists goes away.
| .collect::<Result<Vec<_>>>() | ||
| }) | ||
| .collect::<Result<Vec<_>>>()?; | ||
| Ok(DataSourceExec::from_data_source( |
There was a problem hiding this comment.
We should probably just allow passing the whole source.scan struct into this so it can pick the properties it needs, but from_file_groups is a public function and changing it might break things for consumers; I can do the refactor but since this is only for two properties right now, left it as-is
There was a problem hiding this comment.
This will be reworked through the ongoing datafusion-iceberg workstream. agreed that seems to make sense though.
Removes the comment about decoding filters with the plan's converter, since that is the usual way to decode them. The lazy codec test builds its filter with DataFusion's binary and col helpers.
| if filter.snapshot_generation() == 1 { | ||
| write!(f, "DynamicFilter [ empty ]")?; | ||
| continue; | ||
| } | ||
| let current = snapshot_physical_expr(filter) | ||
| .and_then(|current| { | ||
| current.transform_up(|expr| match expr.downcast_ref::<InListExpr>() { | ||
| Some(in_list) => { | ||
| let not = if in_list.negated() { "NOT " } else { "" }; | ||
| let text = format!( | ||
| "{} {not}IN ([{} elements])", | ||
| in_list.expr(), | ||
| in_list.len() | ||
| ); | ||
| Ok(Transformed::yes(lit(text))) | ||
| } | ||
| None => Ok(Transformed::no(expr)), | ||
| }) | ||
| }) |
There was a problem hiding this comment.
In some testing with large tables, printing all the filters was resulting in a massive plan because of the number of elements being passed in the dynamic filter; so I made it print DynamicFilter[ 10 elements ] instead.
I'm also ok to remove this from the plan entirely, but it was useful while testing the changes in this PR so maybe there's a better way to represent them
alexanderbianchi
left a comment
There was a problem hiding this comment.
LGTM. We will want to port this to datafusion-iceberg once apache/datafusion-iceberg#67 this lands. for now there's no reason to delay adding features, but once we are closer to removing this data source we will pause development here to focus on porting to datafusion-iceberg.
| .collect::<Result<Vec<_>>>() | ||
| }) | ||
| .collect::<Result<Vec<_>>>()?; | ||
| Ok(DataSourceExec::from_data_source( |
There was a problem hiding this comment.
This will be reworked through the ongoing datafusion-iceberg workstream. agreed that seems to make sense though.
Note
This is part of a stack of PRs that implement dynamic filtering for Iceberg. The previous PR can be found here - #850
Relevant changes will also be ported to datafusion-iceberg.
Summary
This PR adds support for accepting and storing expressions, including dynamic filters, in both Iceberg datasources (Eager and Lazy).
Based on a config,
iceberg.dynamic_filtering_enabled, operators likeHashJoinExecare also informed that the datasource is a consumer of these dynamic filters.The dynamic filters also show up in
EXPLAIN ANALYZEplans in the DataSource in the following format:These filters are not yet used for any optimizations - they're only stored and reported.
Future PRs will use them to improve data pruning in the following order -
PhysicalExprtoiceberg::PredicateFileScanTasksChanges
LazyandEagerdatasources implementtry_pushdown_filters- they extract and store the filters within the scan config, and reportPushedDown::Noto DF so that it continues apply its own filtersdynamic_filtering_enabledflagapply_expressionsregisters its use of dynamic filters, which in turn makes operators likeHashJoinExecactually build them at runtime