Skip to content

GH-55447: [C++][Python] Allow creating a UnionDataset from filtered datasets - #55665

Open
adamreeve wants to merge 4 commits into
apache:mainfrom
adamreeve:filtered-dataset
Open

adamreeve wants to merge 4 commits into
apache:mainfrom
adamreeve:filtered-dataset

Conversation

@adamreeve

@adamreeve adamreeve commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Dataset.filter() stores a filter on the Python Dataset wrapper, which is later used to build scan options. But a UnionDataset is represented in C++, so filtered Datasets could not be unioned.

See #55447 for more context.

What changes are included in this PR?

This adds a new FilteredDataset class that holds a filter in the C++ layer, and converts any datasets with filters to a FilteredDataset when they are used to create a UnionDataset.

The Dataset's filter is combined with any filter in the scan options for predicate pushdown. Scanning a FilteredDataset generates FilteredFragment instances, which apply the filter when generating batches. The row-level filter doesn't account for any additional filter in the scan options, because that's always applied after the scan, so this would just duplicate work.

A possible future optimization could be to implement what was proposed in #28944 so a combined filter could be applied once within the fragment, and then a guarantee declared that uses the combined filter expression, so the later filter could be simplified.

Are these changes tested?

Yes, this includes new C++and Python unit tests.

Are there any user-facing changes?

This adds new types to the public API and new functionality. You can now create a UnionDataset using Datasets that have filters, which would previously raise an error.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

(I used Claude Opus 5.5 for the initial implementation, then reviewed and edited everything myself)

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Dataset.filter() stores a filter on the Python Dataset wrapper, but a
UnionDataset is represented in C++, so filtered Datasets could not be
unioned.

This adds a new FilteredDataset class that holds a filter in the C++ layer,
and converts any datasets with filters to a FilteredDataset when they
are used to create a UnionDataset.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active API issues remain: the new public C++ types lack forward declarations, and the Python property shadows Dataset.filter, preventing chained filtering.

2 open issues

Review effort: Lite

Comment thread python/pyarrow/_dataset.pyx Resolved
Comment thread cpp/src/arrow/dataset/dataset.h Resolved
@adamreeve
adamreeve marked this pull request as ready for review November 18, 2026 07:57
Copilot AI lite review requested due to automatic review settings November 18, 2026 07:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Pickling a dataset after applying the inherited filter API can omit the additional predicate, causing the restored dataset to return rows that violate the filter.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Pickling a doubly filtered dataset drops the added filter

python/​pyarrow/​_dataset.pyx:1144

This drops a second filter added through the inherited Dataset.filter() API: that method stores the extra predicate in _scan_options, but __reduce__ serializes only the C++ predicate returned by filter_expression. Pickling filtered.filter(expr) therefore restores the original filtered dataset and can return rows that no longer satisfy expr; include the pending scan filter when reconstructing.

2 resolved since last review

Review effort: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants