Repository navigation
feat: accept and store expressions (including dynamic filters) in Iceberg scan nodes #855
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
5aab3de
860d45a
8e03679
ccc2d92
038a867
cab7f00
4dad432
0800324
1d7e7cc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,6 +18,10 @@ extensions_options! { | |
| pub column_stats_enabled: bool, default = false | ||
| /// Discover and retain file tasks during physical planning instead of execution. | ||
| pub eager_file_planning: bool, default = false | ||
| /// (Experimental) Whether Iceberg scans are consumers of the dynamic filters pushed | ||
| /// into them, so joins and aggregates compute those filters. Scans don't skip data | ||
| /// with them yet. | ||
| pub dynamic_filtering_enabled: bool, default = false | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was added so that a rolling deploy for workers doesn't cause errors if the coordinator starts producing dynamic filters that the workers don't consume yet - I'm not sure if that's necessarily an issue in practice and this adds something to the public API of this datasource that might not be easily removed later, so I'm open to changing the approach to something better
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We have another mechanism for this, worker versioning. doesn't need to be done per pr like this. I think dynamic filtering should be configurable datafusion-wide. Although I don't mind a data source specific toggle. |
||
| } | ||
| } | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -95,16 +95,17 @@ impl PhysicalExtensionCodec for EagerIcebergCodec { | |
| .collect::<Result<Vec<_>>>() | ||
| }) | ||
| .collect::<Result<Vec<_>>>()?; | ||
| Ok(DataSourceExec::from_data_source( | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should probably just allow passing the whole
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This will be reworked through the ongoing datafusion-iceberg workstream. agreed that seems to make sense though. |
||
| EagerIcebergDataSource::from_file_groups( | ||
| scan.schema, | ||
| groups, | ||
| scan.iceberg_file_io, | ||
| scan.iceberg_runtime, | ||
| scan.fetch, | ||
| ctx, | ||
| )?, | ||
| )) | ||
| let mut source = EagerIcebergDataSource::from_file_groups( | ||
| scan.schema, | ||
| groups, | ||
| scan.iceberg_file_io, | ||
| scan.iceberg_runtime, | ||
| scan.fetch, | ||
| ctx, | ||
| )?; | ||
| source.scan.filters = scan.filters; | ||
| source.scan.dynamic_filtering_enabled = scan.dynamic_filtering_enabled; | ||
| Ok(DataSourceExec::from_data_source(source)) | ||
| } | ||
|
|
||
| fn try_encode( | ||
|
|
@@ -175,14 +176,22 @@ impl EagerIcebergCodec { | |
| proto.custom_file_io.as_deref(), | ||
| ctx, | ||
| )?; | ||
| Ok(ScanConfig::new( | ||
| let filters = proto | ||
| .filters | ||
| .as_ref() | ||
| .map(|filters| proto_converter.proto_to_physical_expr(filters, &schema, &decode_ctx)) | ||
| .transpose()?; | ||
| let mut scan = ScanConfig::new( | ||
| schema, | ||
| partitioning, | ||
| fetch, | ||
| iceberg_file_io, | ||
| self.iceberg_runtime.clone(), | ||
| ctx, | ||
| )) | ||
| ); | ||
| scan.filters = filters; | ||
| scan.dynamic_filtering_enabled = proto.dynamic_filtering_enabled; | ||
| Ok(scan) | ||
| } | ||
|
|
||
| fn encode_config( | ||
|
|
@@ -203,6 +212,13 @@ impl EagerIcebergCodec { | |
| fetch: source.fetch().map(|value| value as u64), | ||
| storage_properties: file_io.properties, | ||
| custom_file_io: file_io.custom, | ||
| filters: source | ||
| .scan | ||
| .filters | ||
| .as_ref() | ||
| .map(|filters| proto_converter.physical_expr_to_proto(filters, self)) | ||
| .transpose()?, | ||
| dynamic_filtering_enabled: source.scan.dynamic_filtering_enabled, | ||
| }) | ||
| } | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In some testing with large tables, printing all the filters was resulting in a massive plan because of the number of elements being passed in the dynamic filter; so I made it print
DynamicFilter[ 10 elements ]instead.I'm also ok to remove this from the plan entirely, but it was useful while testing the changes in this PR so maybe there's a better way to represent them