You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
GH-55447: [C++][Python] Allow creating a UnionDataset from filtered datasets - #55665
Dataset.filter() stores a filter on the Python Dataset wrapper, which is later used to build scan options. But a UnionDataset is represented in C++, so filtered Datasets could not be unioned.
This adds a new FilteredDataset class that holds a filter in the C++ layer, and converts any datasets with filters to a FilteredDataset when they are used to create a UnionDataset.
The Dataset's filter is combined with any filter in the scan options for predicate pushdown. Scanning a FilteredDataset generates FilteredFragment instances, which apply the filter when generating batches. The row-level filter doesn't account for any additional filter in the scan options, because that's always applied after the scan, so this would just duplicate work.
A possible future optimization could be to implement what was proposed in #28944 so a combined filter could be applied once within the fragment, and then a guarantee declared that uses the combined filter expression, so the later filter could be simplified.
Are these changes tested?
Yes, this includes new C++and Python unit tests.
Are there any user-facing changes?
This adds new types to the public API and new functionality. You can now create a UnionDataset using Datasets that have filters, which would previously raise an error.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Human
AI
(I used Claude Opus 5.5 for the initial implementation, then reviewed and edited everything myself)
The reason will be displayed to describe this comment to others. Learn more.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
Dataset.filter() stores a filter on the Python Dataset wrapper, but a
UnionDataset is represented in C++, so filtered Datasets could not be
unioned.
This adds a new FilteredDataset class that holds a filter in the C++ layer,
and converts any datasets with filters to a FilteredDataset when they
are used to create a UnionDataset.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Active API issues remain: the new public C++ types lack forward declarations, and the Python property shadows Dataset.filter, preventing chained filtering.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Pickling a dataset after applying the inherited filter API can omit the additional predicate, causing the restored dataset to return rows that violate the filter.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Pickling a doubly filtered dataset drops the added filter
python/pyarrow/_dataset.pyx:1144
This drops a second filter added through the inherited Dataset.filter() API: that method stores the extra predicate in _scan_options, but __reduce__ serializes only the C++ predicate returned by filter_expression. Pickling filtered.filter(expr) therefore restores the original filtered dataset and can return rows that no longer satisfy expr; include the pending scan filter when reconstructing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Dataset.filter()stores a filter on the Python Dataset wrapper, which is later used to build scan options. But aUnionDatasetis represented in C++, so filtered Datasets could not be unioned.See #55447 for more context.
What changes are included in this PR?
This adds a new
FilteredDatasetclass that holds a filter in the C++ layer, and converts any datasets with filters to aFilteredDatasetwhen they are used to create aUnionDataset.The Dataset's filter is combined with any filter in the scan options for predicate pushdown. Scanning a
FilteredDatasetgeneratesFilteredFragmentinstances, which apply the filter when generating batches. The row-level filter doesn't account for any additional filter in the scan options, because that's always applied after the scan, so this would just duplicate work.A possible future optimization could be to implement what was proposed in #28944 so a combined filter could be applied once within the fragment, and then a guarantee declared that uses the combined filter expression, so the later filter could be simplified.
Are these changes tested?
Yes, this includes new C++and Python unit tests.
Are there any user-facing changes?
This adds new types to the public API and new functionality. You can now create a
UnionDatasetusing Datasets that have filters, which would previously raise an error.Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
(I used Claude Opus 5.5 for the initial implementation, then reviewed and edited everything myself)
Reviewed before submission by: