Repository navigation
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Hi @m4ru1, thank you for your contribution! This is a nice simple implementation, but I think there are a few things to keep in mind:
|
Jadiouo
left a comment
There was a problem hiding this comment.
Thanks for this, @m4ru1. I compared base f492423 and head b2ffe8c on a small local Hive dataset (V2 reader, 3 files x 5 row groups, with nulls and an all-null row group, year as int and as string), counting the bytes read through a wrapped filesystem.
For 20 OR filters the rows were identical between base and PR. (year==2020)|(x>12000) read 9 of 15 row groups (2.43 MB instead of 3.65 MB), and (year==2099)|(x>12000) read 5. Reversed operands and string partitions prune the same way. Nested AND, NOT, is_null, is_in, != and 3-way ORs fall back to a full scan, still correct.
Tests: 154 + 99 pass on the PR. On base the unit file fails to import and two logical tests fail, but test_parquet_mixed_or.py passes on base too, since it only compares rows.
Two small things:
- Could one e2e test assert that fewer row groups or bytes are read? Today only the unit tests would catch a pruning regression.
- The two new tests in
test_read_files_logical.pysit after theif __name__ == "__main__":block.
@AarryaSaraf's points on the PR (pruning at listing time, a pushed-down filter disabling this, wider coverage) seem worth settling first.
|
@AarryaSaraf, in reply to your comment: Thanks, Aarrya. I have moved the pruning decision into listing. Partition substitution now uses the existing column-substitution visitor in a common helper, with the Parquet adapter checking partition values and schema. An existing pushed data filter no longer turns it off: listing applies that filter first, then intersects the remaining groups with those selected after partition substitution. Both filters stay in place. Nested AND/OR/NOT, leaf null checks and bounded literal IN are covered now. Narrow-integer IN and floating-point residual predicates still fall back; the description has a coverage table. The helper is format-independent, but I have kept admission in the source adapter: it must supply the values the reader actually returns, match the statistics to the reader’s field types, and preserve the empty-result schema. For example, MCAP keeps a message’s own The part I would like your input on is the empty result. Keeping a data group just for its schema still caused an unnecessary read. Instead, listing emits at most one schema-only manifest entry per verified shard; the reader uses it to emit a typed empty table, with normal partition columns and projection, before opening data files. Since #67037 also changes these shared interfaces, would you carry that instruction in the manifest? Skipping a group also skips the work that could raise an error. A pushed expression such as division by zero can stop raising when the groups that would evaluate it are pruned. A task containing only schema-only entries does not construct the data scanner, so options checked there, such as |
|
@Jadiouo, in reply to your review: Thanks for checking the base too, Jadiouo. You caught the gap in the e2e test: it checked the answer but could not tell whether pruning happened. The new tests observe actual read workers and compare against the same code with the listing hint disabled. For the original issue’s query, Arrow 17.0.0 and 23.0.1 both consumed 25 rather than 160 groups, from five rather than 32 files. The test checks the complete multiset of 5,000 I moved the planning regressions into the listing rule’s test module, before its main guard. The focused tests were rerun on final The performance comparison was rerun on the final integrated commit, using one 59.75 GB dataset with filters keeping 100%, 50% and 12.5% of the row groups. Each case has two pairs with reversed version order. Query time fell by 55.0–62.6% when half the groups remained and 94.2–95.9% when one eighth remained. Keeping every group gave mixed results: +5.48% and −0.42% query time. The description includes both pairs for each large-dataset case. Full output and schema matched in separate checks, but caches were not cleared, so these timings do not establish cold-cache performance or a repeatable gain. Your exact 20-query set has not been rerun. If you still have the script, could you share it? The cases that fell back in your run would be useful to compare on the same input. Pruning is now in listing; the design question I raised with Aarrya is how the manifest should carry the empty-result schema without retaining a data group just for it. |
|
Thanks for moving this to listing, but this is now 6.3k lines (5.2k of them tests), and I think a much smaller version could cover more. Right now pruning only kicks in for a bare Something like this should be enough:
|
|
Thanks for the update, @m4ru1. I re-ran an equivalent of my earlier check on 17bbe59 against aa00fdb: a local Hive dataset (3 files x 5 row groups, with nulls and an all-null group; int and string Rows and schema match base in all of them. Both of my earlier points are addressed. You asked about my script: I no longer have the original, so here is the rebuilt one: https://gist.github.com/Jadiouo/35fcb95a0aee582953449dc44c65648f. It counts reads with strace, because my first version went through a wrapped filesystem, where this revision doesn't prune at all (that matches the local-only scope in the description). |
8f0a595 to
17bbe59
Compare
|
Thanks, @AarryaSaraf. I’ve kept the smaller revision local for now. It uses Arrow for listing-time pruning, keeps the existing filters, and leaves the empty-result schema fix separate. I ran into two cases when passing the residual directly to statistics pruning with PyArrow 23.0.1. Both have one row group under
There is related upstream work in Arrow #54915 for signed-zero membership and #50807 for NaN-aware pruning. Both are still open. For now, the smaller revision skips extra residual pruning when it cannot establish safety, rather than rewriting parts of the expression. This can read more groups for mixed expressions, while preserving the existing pushed filter and final residual evaluation. I’d prefer to keep the underlying fixes in Arrow and avoid expanding this PR. How would you think about this? |
|
Hi @Jadiouo, sorry for the review-request noise. During my latest update, the PR temporarily included unrelated upstream changes in its diff and appears to have triggered additional CODEOWNERS requests. I should have checked the PR’s actual comparison before pushing. I’ve reverted the update, but the extra requests remain. I’m discussing a more suitable approach with Aarrya. Could you help remove the requests below, or point me to a maintainer who can?
I tried removing them through the API and GitHub UI. The API returns 422 when resolving a team node, and the UI change does not persist after refreshing. Thanks for your help, and sorry for the extra work. |
|
@m4ru1 The two cases you mentioned are already pruned during listing on master, with a plain data filter and no partition column. For example, |
|
Thanks @AarryaSaraf and @Jadiouo for taking the time to work through this with me. Your feedback helped make the design much simpler. I’ve pushed the revision using Arrow’s existing pruning path. It’s much smaller now and also works with S3 and operations after the filter. I’ve kept the tests focused on actual worker reads and rechecked the queries from Jadiouo’s reproducer. Would you both have time to take another look when you get a chance? |
|
Thanks @AarryaSaraf for reviewing this again and helping me see what could be simpler. I’ve pushed the changes, and I really appreciate your patience throughout these revisions. |
|
Thanks @AarryaSaraf for taking another look. I’ve used your suggested comment and removed the duplicate S3 test. |
| residual is not None | ||
| and isinstance(scanner, ParquetScanner) | ||
| and scanner.limit is None | ||
| and scanner.partitioning is not None |
There was a problem hiding this comment.
This needs a rebase onto master. Since #67079, filters on synthesized columns like path stay in the residual filter, and this hint then prunes on them using the folder value or the file's stored statistics, though the reader only fills those columns in after reading. On master plus this PR, where root/year=2024/data.parquet stores its own path column:
ds = ray.data.read_parquet(root, include_paths=True)
ds.filter(expr=(col("path") == file) | (col("year") == "1999")).take_all()[]
Master returns both rows. A folder named path=foo (master's test_synthesized_path_filters_use_output_values[path-hive]) also returns [].
|
Could you update the description? The Testing section still says the tests cover wrapped filesystems and that S3 coverage uses Moto, but both of those tests are gone now. |
**Description** This PR lets Datasource V2 use a file’s partition values to prune Parquet row groups during listing, addressing the full scan in [ray-project#67567](ray-project#67567). For `(year == "2023") | (x < 1000)`, neither side alone is a valid pushdown condition, so Ray keeps the complete filter above the read and can scan every candidate row group. With the partition value available, Arrow can simplify the expression for each file: it is true under `year=2023`, and becomes `x < 1000` elsewhere. Footer statistics can then exclude groups before they become read tasks. Required footer reads still happen, and the complete filter remains above the read. Thanks @AarryaSaraf for suggesting the simpler Arrow-based approach and helping narrow the scope. This revision passes partition values through `make_fragment(partition_expression=...)` and lets `split_by_row_group` handle the residual using the reader schema. The selected groups are intersected with those allowed by any existing pushed data filter, which stays in place. That removes the custom substitution logic and schema-only manifest path while covering more queries. The hint now works for a `Filter` directly above `ReadFiles` anywhere in the plan, including queries followed by `map_batches` or `count()`. Wrapped filesystems and S3 are supported too, and unrelated binary or timestamp columns no longer disable pruning. Your point about NaN and signed zero was helpful. I reproduced both cases with ordinary data-filter pushdown and removed the residual-specific workarounds. This version uses the existing Arrow conversion and statistics-pruning behavior, including its current limitations. Empty-result schema handling is also left for a shared fix. I retained the checks for reader casts, block UDFs and scanner limits, where moving pruning earlier can change the result. For example, a timestamp stored as noon can match a date predicate after the reader casts it to a date; pruning against the original timestamp would lose that row. Likewise, a UDF may change a filtered column, and pruning before a limit can select different rows. These checks keep the hint consistent with what the reader actually evaluates. After rebasing onto master, filters on enabled synthesized columns also stay out of the listing hint, preserving [ray-project#67079](ray-project#67079). A generated `path` must not be pruned using a same-named stored column or Hive partition value. The block-UDF check is shared with the existing predicate pushdown path. Subsystems: **Logical planning / query optimization** and **IO**. ```mermaid flowchart LR subgraph Before B1["IO: FooterReader lists candidate groups"] --> B2["ReadFiles: pushed filter"] B2 --> B3["Residual Filter"] end subgraph After A1["Logical planning: pass residual hint"] -.-> A2["IO: FooterReader<br/>partition facts + Arrow pruning"] A2 --> A3["ReadFiles: pushed filter"] A3 --> A4["Residual Filter"] end classDef changed fill:#fff9c4,stroke:#f9a825,color:#000 class A1,A2 changed ``` **Testing** Thanks also @Jadiouo for checking actual reads and catching the gap in the original e2e test. The tests now check both the returned rows and the row groups read by workers. In the earlier validation of the original issue, the result stayed the same 5,000 rows and schema, while the scan dropped from 160 row groups to 25, across five files instead of 32. Disabling the hint preserved the row comparison but failed the pruning assertion. That earlier validation also covered all 21 queries using the data generator and query definitions from [your reproducer](https://gist.github.com/Jadiouo/35fcb95a0aee582953449dc44c65648f). With the hint enabled and disabled, the complete row results and schemas matched. Worker-level observations confirmed pruning for nested conditions, NOT, IN, `!=`, three-way ORs and queries with an existing pushed filter. This check measured row groups read by workers. The focused tests, related footer/planning/expression regressions and pre-commit checks passed on the rebased revision. The tests check returned rows and worker row-group reads, including null and missing partition values, downstream operations, reader casts, block UDFs and limits. They also cover synthesized `path` collisions with stored and partition columns, and confirm that an unrelated synthesized column does not disable pruning. The duplicate filesystem cases have been removed. <details> <summary>Test selections</summary> The following selections passed. Commands show the pytest selections; wheel-overlay runner setup and log paths are omitted. ```bash python -m pytest -q \ python/ray/data/tests/datasource_v2/test_parquet_residual_pruning.py python -m pytest -q \ python/ray/data/tests/datasource_v2/test_footer_reader.py \ python/ray/data/tests/unit/datasource_v2/test_derive_list_files_pushdown.py \ python/ray/data/tests/test_read_files_logical.py \ python/ray/data/tests/unit/expressions \ python/ray/data/tests/unit/test_expression_evaluator.py python -m pytest -q \ python/ray/data/tests/test_predicate_pushdown.py::test_synthesized_path_filters_use_output_values \ python/ray/data/tests/test_predicate_pushdown.py::test_synthesized_path_residual_preserves_read_constraints ``` </details> **Performance** I compared the hint enabled and disabled on a 2.05 GB dataset with 1M rows and 200 row groups, using seven paired runs after warmup and alternating the execution order. When 55% of the groups remained, median query time fell from 1.725 to 1.504 seconds. When 10% remained, it fell from 1.609 to 1.053 seconds. Both cases were faster with pruning in all seven pairs. Keeping every group gave mixed results, with both medians around 1.92 seconds. Timing covers query construction through materialization, with full-result checks and read observations outside the timed runs. These are warm local-storage measurements on a Tencent CVM using the same implementation with the hint toggled; they do not establish cold-cache or S3 performance. AI assistance was used. Signed-off-by: mora <m4ru1@qq.com>
4b50e0c to
0f2f8a3
Compare
|
Thanks @AarryaSaraf for catching these issues and helping improve the PR. I’ve addressed your comments. I also checked filter/projection/limit combinations and plan reuse, and reran the relevant regression tests on PyArrow 25 and 26 to look for similar issues. Could you take another look when you have a chance? |
|
I compared the PR’s master base I wrote a local Parquet file under Then I compared this mixed partition/data predicate: (col("day") == "today") | ~(col("x") >= 30.0)Using each revision’s Ray Data Python code in a separate process, I ran predicate planning and footer row-group selection, read the selected Parquet groups, and applied Ray’s row-level filter:
The NaN is a valid float, not NULL. The mixed OR remains a residual filter on the base, so the matching row is read and returned. With the PR’s residual pruning hint, Arrow statistics pruning drops the entire row group before the row-level filter can evaluate it. I understand that the underlying Arrow pruning behavior also affects existing data-filter pushdown. However, could applying it to this previously unpushed mixed predicate introduce a new result regression? |
|
Another case with Using the same master base ( Each row was in a separate row group. The predicate was: (col("p") == "keep") | ((col("x") + 100) > 120)Running each revision’s Ray Data planning, footer selection, Parquet reader, and row-level filter in separate processes produced:
For the same I also tested a larger synthetic dataset with 3 partitions, 12 files, and 98,304 rows. With continuous windows of values |
[Data] Prune mixed partition/data filters during Parquet listing
Description
This PR lets Datasource V2 use a file’s partition values to prune Parquet row groups during listing, addressing the full scan in #67567.
For
(year == "2023") | (x < 1000), neither side alone is a valid pushdown condition, so Ray keeps the complete filter above the read and can scan every candidate row group. With the partition value available, Arrow can simplify the expression for each file: it is true underyear=2023, and becomesx < 1000elsewhere. Footer statistics can then exclude groups before they become read tasks. Required footer reads still happen, and the complete filter remains above the read.Thanks @AarryaSaraf for suggesting the simpler Arrow-based approach and helping narrow the scope. This revision passes partition values through
make_fragment(partition_expression=...)and letssplit_by_row_grouphandle the residual using the reader schema. The selected groups are intersected with those allowed by any existing pushed data filter, which stays in place.That removes the custom substitution logic and schema-only manifest path while covering more queries. The hint now works for a
Filterdirectly aboveReadFilesanywhere in the plan, including queries followed bymap_batchesorcount(). Wrapped filesystems and S3 are supported too, and unrelated binary or timestamp columns no longer disable pruning.Your point about NaN and signed zero was helpful. I reproduced both cases with ordinary data-filter pushdown and removed the residual-specific workarounds. This version uses the existing Arrow conversion and statistics-pruning behavior, including its current limitations. Empty-result schema handling is also left for a shared fix.
I retained the checks for reader casts, block UDFs and scanner limits, where moving pruning earlier can change the result. For example, a timestamp stored as noon can match a date predicate after the reader casts it to a date; pruning against the original timestamp would lose that row. Likewise, a UDF may change a filtered column, and pruning before a limit can select different rows. These checks keep the hint consistent with what the reader actually evaluates. After rebasing onto master, filters on enabled synthesized columns also stay out of the listing hint, preserving #67079. A generated
pathmust not be pruned using a same-named stored column or Hive partition value. The block-UDF check is shared with the existing predicate pushdown path.Subsystems: Logical planning / query optimization and IO.
flowchart LR subgraph Before B1["IO: FooterReader lists candidate groups"] --> B2["ReadFiles: pushed filter"] B2 --> B3["Residual Filter"] end subgraph After A1["Logical planning: pass residual hint"] -.-> A2["IO: FooterReader<br/>partition facts + Arrow pruning"] A2 --> A3["ReadFiles: pushed filter"] A3 --> A4["Residual Filter"] end classDef changed fill:#fff9c4,stroke:#f9a825,color:#000 class A1,A2 changedTesting
Thanks also @Jadiouo for checking actual reads and catching the gap in the original e2e test. The tests now check both the returned rows and the row groups read by workers. In the earlier validation of the original issue, the result stayed the same 5,000 rows and schema, while the scan dropped from 160 row groups to 25, across five files instead of 32. Disabling the hint preserved the row comparison but failed the pruning assertion.
That earlier validation also covered all 21 queries using the data generator and query definitions from your reproducer. With the hint enabled and disabled, the complete row results and schemas matched. Worker-level observations confirmed pruning for nested conditions, NOT, IN,
!=, three-way ORs and queries with an existing pushed filter. This check measured row groups read by workers.The focused tests, related footer/planning/expression regressions and pre-commit checks passed on the rebased revision. The tests check returned rows and worker row-group reads, including null and missing partition values, downstream operations, reader casts, block UDFs and limits. They also cover synthesized
pathcollisions with stored and partition columns, and confirm that an unrelated synthesized column does not disable pruning. The duplicate filesystem cases have been removed.Test selections
The following selections passed. Commands show the pytest selections; wheel-overlay runner setup and log paths are omitted.
Performance
I compared the hint enabled and disabled on a 2.05 GB dataset with 1M rows and 200 row groups, using seven paired runs after warmup and alternating the execution order.
When 55% of the groups remained, median query time fell from 1.725 to 1.504 seconds. When 10% remained, it fell from 1.609 to 1.053 seconds. Both cases were faster with pruning in all seven pairs. Keeping every group gave mixed results, with both medians around 1.92 seconds.
Timing covers query construction through materialization, with full-result checks and read observations outside the timed runs. These are warm local-storage measurements on a Tencent CVM using the same implementation with the hint toggled; they do not establish cold-cache or S3 performance.
AI assistance was used.