Skip to content

[Data] Prune Parquet row groups for mixed partition/data OR filters - #67855

Open
m4ru1 wants to merge 1 commit into
ray-project:masterfrom
m4ru1:contribution/mixed-or-parquet-pruning
Open

m4ru1 wants to merge 1 commit into
ray-project:masterfrom
m4ru1:contribution/mixed-or-parquet-pruning

Conversation

@m4ru1

@m4ru1 m4ru1 commented Nov 5, 2026 •

Copy link
Copy Markdown

[Data] Prune mixed partition/data filters during Parquet listing

Description

This PR lets Datasource V2 use a file’s partition values to prune Parquet row groups during listing, addressing the full scan in #67567.

For (year == "2023") | (x < 1000), neither side alone is a valid pushdown condition, so Ray keeps the complete filter above the read and can scan every candidate row group. With the partition value available, Arrow can simplify the expression for each file: it is true under year=2023, and becomes x < 1000 elsewhere. Footer statistics can then exclude groups before they become read tasks. Required footer reads still happen, and the complete filter remains above the read.

Thanks @AarryaSaraf for suggesting the simpler Arrow-based approach and helping narrow the scope. This revision passes partition values through make_fragment(partition_expression=...) and lets split_by_row_group handle the residual using the reader schema. The selected groups are intersected with those allowed by any existing pushed data filter, which stays in place.

That removes the custom substitution logic and schema-only manifest path while covering more queries. The hint now works for a Filter directly above ReadFiles anywhere in the plan, including queries followed by map_batches or count(). Wrapped filesystems and S3 are supported too, and unrelated binary or timestamp columns no longer disable pruning.

Your point about NaN and signed zero was helpful. I reproduced both cases with ordinary data-filter pushdown and removed the residual-specific workarounds. This version uses the existing Arrow conversion and statistics-pruning behavior, including its current limitations. Empty-result schema handling is also left for a shared fix.

I retained the checks for reader casts, block UDFs and scanner limits, where moving pruning earlier can change the result. For example, a timestamp stored as noon can match a date predicate after the reader casts it to a date; pruning against the original timestamp would lose that row. Likewise, a UDF may change a filtered column, and pruning before a limit can select different rows. These checks keep the hint consistent with what the reader actually evaluates. After rebasing onto master, filters on enabled synthesized columns also stay out of the listing hint, preserving #67079. A generated path must not be pruned using a same-named stored column or Hive partition value. The block-UDF check is shared with the existing predicate pushdown path.

Subsystems: Logical planning / query optimization and IO.

flowchart LR
  subgraph Before
    B1["IO: FooterReader lists candidate groups"] --> B2["ReadFiles: pushed filter"]
    B2 --> B3["Residual Filter"]
  end
  subgraph After
    A1["Logical planning: pass residual hint"] -.-> A2["IO: FooterReader<br/>partition facts + Arrow pruning"]
    A2 --> A3["ReadFiles: pushed filter"]
    A3 --> A4["Residual Filter"]
  end
  classDef changed fill:#fff9c4,stroke:#f9a825,color:#000
  class A1,A2 changed
Loading

Testing

Thanks also @Jadiouo for checking actual reads and catching the gap in the original e2e test. The tests now check both the returned rows and the row groups read by workers. In the earlier validation of the original issue, the result stayed the same 5,000 rows and schema, while the scan dropped from 160 row groups to 25, across five files instead of 32. Disabling the hint preserved the row comparison but failed the pruning assertion.

That earlier validation also covered all 21 queries using the data generator and query definitions from your reproducer. With the hint enabled and disabled, the complete row results and schemas matched. Worker-level observations confirmed pruning for nested conditions, NOT, IN, !=, three-way ORs and queries with an existing pushed filter. This check measured row groups read by workers.

The focused tests, related footer/planning/expression regressions and pre-commit checks passed on the rebased revision. The tests check returned rows and worker row-group reads, including null and missing partition values, downstream operations, reader casts, block UDFs and limits. They also cover synthesized path collisions with stored and partition columns, and confirm that an unrelated synthesized column does not disable pruning. The duplicate filesystem cases have been removed.

Test selections

The following selections passed. Commands show the pytest selections; wheel-overlay runner setup and log paths are omitted.

python -m pytest -q \
  python/ray/data/tests/datasource_v2/test_parquet_residual_pruning.py

python -m pytest -q \
  python/ray/data/tests/datasource_v2/test_footer_reader.py \
  python/ray/data/tests/unit/datasource_v2/test_derive_list_files_pushdown.py \
  python/ray/data/tests/test_read_files_logical.py \
  python/ray/data/tests/unit/expressions \
  python/ray/data/tests/unit/test_expression_evaluator.py

python -m pytest -q \
  python/ray/data/tests/test_predicate_pushdown.py::test_synthesized_path_filters_use_output_values \
  python/ray/data/tests/test_predicate_pushdown.py::test_synthesized_path_residual_preserves_read_constraints

Performance

I compared the hint enabled and disabled on a 2.05 GB dataset with 1M rows and 200 row groups, using seven paired runs after warmup and alternating the execution order.

When 55% of the groups remained, median query time fell from 1.725 to 1.504 seconds. When 10% remained, it fell from 1.609 to 1.053 seconds. Both cases were faster with pruning in all seven pairs. Keeping every group gave mixed results, with both medians around 1.92 seconds.

Timing covers query construction through materialization, with full-result checks and read observations outside the timed runs. These are warm local-storage measurements on a Tencent CVM using the same implementation with the hint toggled; they do not establish cold-cache or S3 performance.

AI assistance was used.

@m4ru1
m4ru1 requested a review from a team as a code owner November 5, 2026 02:57
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Nov 5, 2026
@AarryaSaraf

Copy link
Copy Markdown
Contributor

Hi @m4ru1, thank you for your contribution! This is a nice simple implementation, but I think there are a few things to keep in mind:

  • We should be able to do the pruning during listing instead of during the read.
  • We can cover all such cases, instead of only the few comparison-based ones handled here.
  • A pushed data filter shouldn't turn pruning off.
  • The per-file partition substitution doesn't need to be format-specific.

@Jadiouo Jadiouo left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @m4ru1. I compared base f492423 and head b2ffe8c on a small local Hive dataset (V2 reader, 3 files x 5 row groups, with nulls and an all-null row group, year as int and as string), counting the bytes read through a wrapped filesystem.

For 20 OR filters the rows were identical between base and PR. (year==2020)|(x>12000) read 9 of 15 row groups (2.43 MB instead of 3.65 MB), and (year==2099)|(x>12000) read 5. Reversed operands and string partitions prune the same way. Nested AND, NOT, is_null, is_in, != and 3-way ORs fall back to a full scan, still correct.

Tests: 154 + 99 pass on the PR. On base the unit file fails to import and two logical tests fail, but test_parquet_mixed_or.py passes on base too, since it only compares rows.

Two small things:

  1. Could one e2e test assert that fewer row groups or bytes are read? Today only the unit tests would catch a pruning regression.
  2. The two new tests in test_read_files_logical.py sit after the if __name__ == "__main__": block.

@AarryaSaraf's points on the PR (pruning at listing time, a pushed-down filter disabling this, wider coverage) seem worth settling first.

m4ru1 commented Nov 8, 2026

Copy link
Copy Markdown
Author

@AarryaSaraf, in reply to your comment:

Thanks, Aarrya. I have moved the pruning decision into listing. Partition substitution now uses the existing column-substitution visitor in a common helper, with the Parquet adapter checking partition values and schema. An existing pushed data filter no longer turns it off: listing applies that filter first, then intersects the remaining groups with those selected after partition substitution. Both filters stay in place.

Nested AND/OR/NOT, leaf null checks and bounded literal IN are covered now. Narrow-integer IN and floating-point residual predicates still fall back; the description has a coverage table. The helper is format-independent, but I have kept admission in the source adapter: it must supply the values the reader actually returns, match the statistics to the reader’s field types, and preserve the empty-result schema. For example, MCAP keeps a message’s own topic even under a conflicting topic=.../ directory, so a path value is not automatically a valid substitute. Only the built-in local Hive Parquet path has those checks here; support for other sources needs its own adapter and reader tests.

The part I would like your input on is the empty result. Keeping a data group just for its schema still caused an unnecessary read. Instead, listing emits at most one schema-only manifest entry per verified shard; the reader uses it to emit a typed empty table, with normal partition columns and projection, before opening data files. Since #67037 also changes these shared interfaces, would you carry that instruction in the manifest?

Skipping a group also skips the work that could raise an error. A pushed expression such as division by zero can stop raising when the groups that would evaluate it are pruned. A task containing only schema-only entries does not construct the data scanner, so options checked there, such as batch_size, may not be validated either. I have kept that behavior without adding a separate validation pass or disabling the hint for those options. Errors from required footers and reads that actually run still propagate.

m4ru1 commented Nov 8, 2026

Copy link
Copy Markdown
Author

@Jadiouo, in reply to your review:

Thanks for checking the base too, Jadiouo. You caught the gap in the e2e test: it checked the answer but could not tell whether pruning happened. The new tests observe actual read workers and compare against the same code with the listing hint disabled.

For the original issue’s query, Arrow 17.0.0 and 23.0.1 both consumed 25 rather than 160 groups, from five rather than 32 files. The test checks the complete multiset of 5,000 (x, year, month) tuples, including duplicates, and the schema in both arms. Footer reads still happen; these are data-consumption counts, not total opens or bytes.

I moved the planning regressions into the listing rule’s test module, before its main guard. The focused tests were rerun on final 17bbe597 with the matching official aa00fdbb Ray Core. Each Arrow version passed 306 tests, including 21 schema-only tests, with one existing pre-footer empty-input failure marked expected. Full upstream CI has not run.

The performance comparison was rerun on the final integrated commit, using one 59.75 GB dataset with filters keeping 100%, 50% and 12.5% of the row groups. Each case has two pairs with reversed version order. Query time fell by 55.0–62.6% when half the groups remained and 94.2–95.9% when one eighth remained. Keeping every group gave mixed results: +5.48% and −0.42% query time. The description includes both pairs for each large-dataset case. Full output and schema matched in separate checks, but caches were not cleared, so these timings do not establish cold-cache performance or a repeatable gain.

Your exact 20-query set has not been rerun. If you still have the script, could you share it? The cases that fell back in your run would be useful to compare on the same input. Pruning is now in listing; the design question I raised with Aarrya is how the manifest should carry the empty-result schema without retaining a data group just for it.

@m4ru1
m4ru1 requested a review from Jadiouo November 9, 2026 14:30
@AarryaSaraf

Copy link
Copy Markdown
Contributor

Thanks for moving this to listing, but this is now 6.3k lines (5.2k of them tests), and I think a much smaller version could cover more. Right now pruning only kicks in for a bare filter → take_all on a local disk: adding map_batches after the filter, calling count(), reading from S3, or having one binary or timestamp column anywhere in the schema all read every row group again.

Something like this should be enough:

  • In FooterReader._read_and_chunk, build the fragment with the file's folder values (make_fragment(..., partition_expression=...)) and prune with split_by_row_group(residual, schema=...), where the schema includes the partition columns. pyarrow does the substitution itself, including nested AND/OR/NOT, isin, floats and timestamps, and _try_split_by_row_group already keeps every group when it can't.
  • Hand the residual to the indexer for any Filter directly above ReadFiles, not only at the plan root.
  • No other admission checks, and no schema-only manifest entries. When every group is pruned, behave like a pushed data filter does today: read_parquet(d).filter(expr=col("x") < 0) also produces no blocks on master. The empty-result schema can be fixed separately for every pushdown.
  • A few tests: one parametrized test through read_parquet(...).filter(...) that checks both the rows and the row groups read, plus one with a null folder value.
    Lets aim to keep this to a few hundred lines including tests.

@Jadiouo

Jadiouo commented Nov 10, 2026

Copy link
Copy Markdown

Thanks for the update, @m4ru1. I re-ran an equivalent of my earlier check on 17bbe59 against aa00fdb: a local Hive dataset (3 files x 5 row groups, with nulls and an all-null group; int and string year), 21 queries, counting the bytes read from the data files.

Rows and schema match base in all of them. (year==2020)|(x>12000) reads 4.2 MB instead of 6.7 MB, and nested AND, NOT, is_in and != now prune too. With a pushed filter first (filter(x>5000), then the mixed OR), it reads 2.1 MB instead of 4.2 MB, so the pushed filter no longer turns pruning off. A query that matches nothing keeps its schema and reads only the footers.

Both of my earlier points are addressed. You asked about my script: I no longer have the original, so here is the rebuilt one: https://gist.github.com/Jadiouo/35fcb95a0aee582953449dc44c65648f. It counts reads with strace, because my first version went through a wrapped filesystem, where this revision doesn't prune at all (that matches the local-only scope in the description).

@m4ru1
m4ru1 force-pushed the contribution/mixed-or-parquet-pruning branch from 8f0a595 to 17bbe59 Compare November 11, 2026 13:43

m4ru1 commented Nov 11, 2026 •

Copy link
Copy Markdown
Author

Thanks, @AarryaSaraf. I’ve kept the smaller revision local for now. It uses Arrow for listing-time pruning, keeps the existing filters, and leaves the empty-result schema fix separate.

I ran into two cases when passing the residual directly to statistics pruning with PyArrow 23.0.1. Both have one row group under p=a and the residual (p == "keep") | predicate:

  • x = [NaN, 0.0], predicate ~(x == 0.0): filtering the rows keeps NaN, but pruning removes the group.
  • x = [-0.0, +0.0], predicate x.is_in([+0.0]): eager filtering keeps the +0.0 row, but pruning removes the group.

There is related upstream work in Arrow #54915 for signed-zero membership and #50807 for NaN-aware pruning. Both are still open.

For now, the smaller revision skips extra residual pruning when it cannot establish safety, rather than rewriting parts of the expression. This can read more groups for mixed expressions, while preserving the existing pushed filter and final residual evaluation. I’d prefer to keep the underlying fixes in Arrow and avoid expanding this PR.

How would you think about this?

m4ru1 commented Nov 11, 2026

Copy link
Copy Markdown
Author

Hi @Jadiouo, sorry for the review-request noise. During my latest update, the PR temporarily included unrelated upstream changes in its diff and appears to have triggered additional CODEOWNERS requests. I should have checked the PR’s actual comparison before pushing. I’ve reverted the update, but the extra requests remain.

I’m discussing a more suitable approach with Aarrya.

Could you help remove the requests below, or point me to a maintainer who can?

  • People: richardliaw, MengjinYan, edoakes, andrewsykim, ryanaoleary
  • Teams: ray-core, ray-docs, ray-ci, ray-serve, ray-llm, ray-train, ray-tune, ray-docs-reviewers, kuberay

I tried removing them through the API and GitHub UI. The API returns 422 when resolving a team node, and the UI change does not persist after refreshing. Thanks for your help, and sorry for the extra work.

@AarryaSaraf

Copy link
Copy Markdown
Contributor

@m4ru1 The two cases you mentioned are already pruned during listing on master, with a plain data filter and no partition column. For example, read_parquet(...).filter(expr=~(col("x") == 0.0)) on [NaN, 0.0] returns [], and filter(expr=col("x").is_in([0.0])) on [-0.0, 0.0] also returns []. This means the gap is in pyarrow's statistics pruning itself (the Arrow PRs you linked), not in pruning the residual. So the residual doesn't need extra safety checks: it can go through the same pruning as the pushed data filter, and the NaN and signed-zero issue can be fixed separately for every pushdown.

m4ru1 commented Nov 15, 2026

Copy link
Copy Markdown
Author

Thanks @AarryaSaraf and @Jadiouo for taking the time to work through this with me. Your feedback helped make the design much simpler.

I’ve pushed the revision using Arrow’s existing pruning path. It’s much smaller now and also works with S3 and operations after the filter. I’ve kept the tests focused on actual worker reads and rechecked the queries from Jadiouo’s reproducer.

Would you both have time to take another look when you get a chance?

m4ru1 commented Nov 17, 2026

Copy link
Copy Markdown
Author

Thanks @AarryaSaraf for reviewing this again and helping me see what could be simpler. I’ve pushed the changes, and I really appreciate your patience throughout these revisions.

@m4ru1
m4ru1 requested a review from AarryaSaraf November 17, 2026 05:48
@m4ru1
m4ru1 requested a review from a team as a code owner November 17, 2026 22:34

m4ru1 commented Nov 17, 2026

Copy link
Copy Markdown
Author

Thanks @AarryaSaraf for taking another look. I’ve used your suggested comment and removed the duplicate S3 test.

residual is not None
and isinstance(scanner, ParquetScanner)
and scanner.limit is None
and scanner.partitioning is not None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a rebase onto master. Since #67079, filters on synthesized columns like path stay in the residual filter, and this hint then prunes on them using the folder value or the file's stored statistics, though the reader only fills those columns in after reading. On master plus this PR, where root/year=2024/data.parquet stores its own path column:

ds = ray.data.read_parquet(root, include_paths=True)
ds.filter(expr=(col("path") == file) | (col("year") == "1999")).take_all()
[]

Master returns both rows. A folder named path=foo (master's test_synthesized_path_filters_use_output_values[path-hive]) also returns [].

@AarryaSaraf

Copy link
Copy Markdown
Contributor

Could you update the description? The Testing section still says the tests cover wrapped filesystems and that S3 coverage uses Moto, but both of those tests are gone now.

**Description**

This PR lets Datasource V2 use a file’s partition values to prune Parquet row groups during listing, addressing the full scan in [ray-project#67567](ray-project#67567).

For `(year == "2023") | (x < 1000)`, neither side alone is a valid pushdown condition, so Ray keeps the complete filter above the read and can scan every candidate row group. With the partition value available, Arrow can simplify the expression for each file: it is true under `year=2023`, and becomes `x < 1000` elsewhere. Footer statistics can then exclude groups before they become read tasks. Required footer reads still happen, and the complete filter remains above the read.

Thanks @AarryaSaraf for suggesting the simpler Arrow-based approach and helping narrow the scope. This revision passes partition values through `make_fragment(partition_expression=...)` and lets `split_by_row_group` handle the residual using the reader schema. The selected groups are intersected with those allowed by any existing pushed data filter, which stays in place.

That removes the custom substitution logic and schema-only manifest path while covering more queries. The hint now works for a `Filter` directly above `ReadFiles` anywhere in the plan, including queries followed by `map_batches` or `count()`. Wrapped filesystems and S3 are supported too, and unrelated binary or timestamp columns no longer disable pruning.

Your point about NaN and signed zero was helpful. I reproduced both cases with ordinary data-filter pushdown and removed the residual-specific workarounds. This version uses the existing Arrow conversion and statistics-pruning behavior, including its current limitations. Empty-result schema handling is also left for a shared fix.

I retained the checks for reader casts, block UDFs and scanner limits, where moving pruning earlier can change the result. For example, a timestamp stored as noon can match a date predicate after the reader casts it to a date; pruning against the original timestamp would lose that row. Likewise, a UDF may change a filtered column, and pruning before a limit can select different rows. These checks keep the hint consistent with what the reader actually evaluates. After rebasing onto master, filters on enabled synthesized columns also stay out of the listing hint, preserving [ray-project#67079](ray-project#67079). A generated `path` must not be pruned using a same-named stored column or Hive partition value. The block-UDF check is shared with the existing predicate pushdown path.

Subsystems: **Logical planning / query optimization** and **IO**.

```mermaid
flowchart LR
  subgraph Before
    B1["IO: FooterReader lists candidate groups"] --> B2["ReadFiles: pushed filter"]
    B2 --> B3["Residual Filter"]
  end
  subgraph After
    A1["Logical planning: pass residual hint"] -.-> A2["IO: FooterReader<br/>partition facts + Arrow pruning"]
    A2 --> A3["ReadFiles: pushed filter"]
    A3 --> A4["Residual Filter"]
  end
  classDef changed fill:#fff9c4,stroke:#f9a825,color:#000
  class A1,A2 changed
```

**Testing**

Thanks also @Jadiouo for checking actual reads and catching the gap in the original e2e test. The tests now check both the returned rows and the row groups read by workers. In the earlier validation of the original issue, the result stayed the same 5,000 rows and schema, while the scan dropped from 160 row groups to 25, across five files instead of 32. Disabling the hint preserved the row comparison but failed the pruning assertion.

That earlier validation also covered all 21 queries using the data generator and query definitions from [your reproducer](https://gist.github.com/Jadiouo/35fcb95a0aee582953449dc44c65648f). With the hint enabled and disabled, the complete row results and schemas matched. Worker-level observations confirmed pruning for nested conditions, NOT, IN, `!=`, three-way ORs and queries with an existing pushed filter. This check measured row groups read by workers.

The focused tests, related footer/planning/expression regressions and pre-commit checks passed on the rebased revision. The tests check returned rows and worker row-group reads, including null and missing partition values, downstream operations, reader casts, block UDFs and limits. They also cover synthesized `path` collisions with stored and partition columns, and confirm that an unrelated synthesized column does not disable pruning. The duplicate filesystem cases have been removed.

<details>
<summary>Test selections</summary>

The following selections passed. Commands show the pytest selections; wheel-overlay runner setup and log paths are omitted.

```bash
python -m pytest -q \
  python/ray/data/tests/datasource_v2/test_parquet_residual_pruning.py

python -m pytest -q \
  python/ray/data/tests/datasource_v2/test_footer_reader.py \
  python/ray/data/tests/unit/datasource_v2/test_derive_list_files_pushdown.py \
  python/ray/data/tests/test_read_files_logical.py \
  python/ray/data/tests/unit/expressions \
  python/ray/data/tests/unit/test_expression_evaluator.py

python -m pytest -q \
  python/ray/data/tests/test_predicate_pushdown.py::test_synthesized_path_filters_use_output_values \
  python/ray/data/tests/test_predicate_pushdown.py::test_synthesized_path_residual_preserves_read_constraints
```

</details>

**Performance**

I compared the hint enabled and disabled on a 2.05 GB dataset with 1M rows and 200 row groups, using seven paired runs after warmup and alternating the execution order.

When 55% of the groups remained, median query time fell from 1.725 to 1.504 seconds. When 10% remained, it fell from 1.609 to 1.053 seconds. Both cases were faster with pruning in all seven pairs. Keeping every group gave mixed results, with both medians around 1.92 seconds.

Timing covers query construction through materialization, with full-result checks and read observations outside the timed runs. These are warm local-storage measurements on a Tencent CVM using the same implementation with the hint toggled; they do not establish cold-cache or S3 performance.

AI assistance was used.

Signed-off-by: mora <m4ru1@qq.com>
@m4ru1
m4ru1 force-pushed the contribution/mixed-or-parquet-pruning branch from 4b50e0c to 0f2f8a3 Compare November 18, 2026 03:13

m4ru1 commented Nov 18, 2026

Copy link
Copy Markdown
Author

Thanks @AarryaSaraf for catching these issues and helping improve the PR. I’ve addressed your comments. I also checked filter/projection/limit combinations and plan reuse, and reran the relevant regression tests on PyArrow 25 and 26 to look for similar issues. Could you take another look when you have a chance?

Copy link
Copy Markdown
Contributor

I compared the PR’s master base 67f94a1c with PR head 0f2f8a34, using PyArrow 23.0.1.

I wrote a local Parquet file under day=old/ with one row group:

id:  0     1     2
x:   35.0  40.0  NaN

Then I compared this mixed partition/data predicate:

(col("day") == "today") | ~(col("x") >= 30.0)

Using each revision’s Ray Data Python code in a separate process, I ran predicate planning and footer row-group selection, read the selected Parquet groups, and applied Ray’s row-level filter:

Selected row groups Result IDs
Master base [0] [2]
PR head [] []

The NaN is a valid float, not NULL. The mixed OR remains a residual filter on the base, so the matching row is read and returned. With the PR’s residual pruning hint, Arrow statistics pruning drops the entire row group before the row-level filter can evaluate it.

I understand that the underlying Arrow pruning behavior also affects existing data-filter pushdown. However, could applying it to this previously unpushed mixed predicate introduce a new result regression?

Copy link
Copy Markdown
Contributor

Another case with int8 arithmetic:

Using the same master base (67f94a1c), PR head (0f2f8a34), and PyArrow 23.0.1, I tested a Parquet file under p=other/ with x stored and read as int8:

id:  0    1
x:   100  1

Each row was in a separate row group. The predicate was:

(col("p") == "keep") | ((col("x") + 100) > 120)

Running each revision’s Ray Data planning, footer selection, Parquet reader, and row-level filter in separate processes produced:

Selected row groups Result IDs
Master base [0, 1] [0]
PR head [] []

For the same int8 input [100, 1], Ray’s row-level x + 100 evaluates to int64 [200, 101], while the Arrow expression generated by Expr.to_pyarrow() evaluates to int8 [-56, 101]. The field’s physical and logical types both match, but the arithmetic expression’s result differs.

I also tested a larger synthetic dataset with 3 partitions, 12 files, and 98,304 rows. With continuous windows of values 100 and 1, the base returned 65,536 rows and the PR path returned 32,768. A randomly mixed input of the same size returned the same results on both paths.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants