Skip to content

[Data] Prune ORC files using column statistics - #67274

Draft
jiangxt2 wants to merge 9 commits into
ray-project:masterfrom
jiangxt2:feature/orc-statistics-pruning-native
Draft

jiangxt2 wants to merge 9 commits into
ray-project:masterfrom
jiangxt2:feature/orc-statistics-pruning-native

Conversation

@jiangxt2

@jiangxt2 jiangxt2 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Description

Use ORC file statistics to skip files that cannot match a pushed filter, while retaining the complete Arrow row filter and existing read-task boundaries.

Before

ORC DataSourceV2 scans whole-file fragments and applies row filters after reading. Selective predicates cannot use file or stripe statistics to avoid data decoding.

After

  • Reuse Arrow's serialized ORC footer and the existing protobuf descriptor, preserving field presence. Support uncompressed, ZLIB, SNAPPY, LZ4 and ZSTD files through Arrow.
  • Evaluate possible TRUE/FALSE/NULL outcomes for signed integer comparisons and sets, boolean equality, null checks, AND/OR/NOT, and qualified exact STRING bounds. Validate writer identity, column source, types, counts and ranges; contradictory boolean counts invalidate that column's statistics.
  • Default DataContext.orc_statistics_pruning="off" performs no added statistics reads. Explicit "file" skips proven nonmatching files and streams surviving files.
  • Preserve projection dependencies, typed partition overrides, batching, limits, post-filter row identity, retries and pickle-column checks. Listing, task grouping and size sampling remain unchanged.

Subsystems

Subsystems: IO and API configuration. The ORC reader carries the behavior change; DataContext adds the pruning mode.

Data flow

flowchart TD
    subgraph Before["Before: IO / ORC reader"]
        B1["Whole-file Arrow scan"] --> B2["Exact filter and projection"]
    end
    subgraph After["After: IO / ORC reader"]
        A1["Predicate and file schema"] --> A2{"Pruning enabled and eligible?"}
        A2 -->|no| A3["Whole-file Arrow scan"]
        A2 -->|yes| A4["Serialized footer statistics"]
        A4 -->|file cannot match| A5["Skip file"]
        A4 -->|surviving or unknown| A3
        A3 --> A8["Exact filter, projection, batches and limit"]
    end
    classDef added fill:#c8e6c9,stroke:#2e7d32,color:#000
    class A2,A4,A5 added
Loading

Related issues

Addresses #66995. Statistics pruning is implemented within existing file read tasks, retaining whole-file manifests and the default streaming path without requiring stripe task splitting. Cross-task stripe splitting is tracked separately in #66999.

Additional information

PyArrow API and maintenance boundary

PyArrow 23.0.1 exposes no public per-column ORC min/max/null-statistics API. This change reuses the private ORCFile.reader.serialized_file_tail() interface and parses a limited file-footer protobuf subset, preserving field presence and writer/type qualification. PyArrow still performs data decoding and exact row filtering.

A suitable public Python statistics API can replace the metadata adapter; related upstream statistics work is tracked in apache/arrow#49360. Native predicate evaluation can replace Ray's evaluator only if its conservative type/NULL semantics match.

Limits

STRING range pruning is limited to exact bounds from the qualified Java ORC 2.2.1 writer. Missing, truncated, untrusted or unsupported statistics stay unknown for the corresponding predicate leaves. Unavailable statistics fall back to streaming; real filesystem errors propagate through existing retry/error handling.

Enabling file pruning can add metadata requests, and planning samples can still read data from files excluded during execution.

Testing

Linux arm64, Python 3.12, PyArrow 23.0.1; Ray Data sources were used with a separate development Ray Core wheel:

  • Footer parsing, file exclusion and malformed-footer fallback: 10 passed.
  • Selected predicate, metadata and public-pipeline cases: 21 passed before the configuration-test consolidation.
  • Public worker file pruning: 4 passed.
  • Scanner configuration and batch sizing: 5 passed.
  • Moto 5.1.18 S3 file-pruning tests: 3 passed.

After configuration-test consolidation, 6 configuration cases passed on macOS arm64, Python 3.13.7 and PyArrow 23.0.1 with a separate development Ray Core wheel; seven isolated in-process fault injections were detected.

Black, Ruff and Pyrefly checks passed for the changed tests. The targeted regressions below were checked locally in isolated source copies or with isolated in-process fault injection; off parameters also protect streaming behavior and need not fail when file pruning is bypassed. The public worker test uses scanner instrumentation to observe actual data scans; the Moto test exercises the internal scanner with the existing S3 fixture.

  • Bypassing file pruning: test_orc_file_statistics_pruning_public_worker, test_logical_partition_pruning_and_statistics, test_orc_file_statistics_s3, test_extended_predicates_exclude_whole_file and test_serialized_footer_statistics_and_file_exclusion fail for the file cases.
  • Dropping the exact Arrow row filter in file mode: test_orc_statistics_public_pipeline fails; it also protects filtering, projection and limit for retained files.
  • Performing statistics work in off mode or for an unsupported predicate: test_no_eligible_predicate_has_no_extra_statistics fails.
  • Forcing the reader mode to off: test_orc_batch_sizing_config_survives_pushdowns fails; it also checks mode preservation after pushdown and serialization. Changing the default or accepting invalid modes makes test_default_and_rejected_stripe_mode fail.
  • Treating an unknown predicate as certain: test_possible_values_against_arrow_oracle fails; changing the <= boundary makes test_string_comparisons_and_integer_limits_against_arrow fail.
  • Accepting unsafe or over-budget sets: test_unsafe_or_over_budget_integer_sets_are_unknown fails; accepting contradictory null counts makes test_independent_no_null_flag_and_contradictions fail.
  • Trusting CHAR/VARCHAR ranges, enabling float ranges or disabling scalar null-count pruning makes test_char_varchar_null_proof_does_not_enable_ranges, test_nan_is_not_null_and_float_ranges_stay_unknown or test_null_counts_do_not_require_range_support fail, respectively.
  • Trusting unqualified string writer bounds makes test_qualified_java_and_unqualified_cpp_string_statistics fail; trusting a truncated lower bound makes test_presence_counts_bounds_and_writer fail.
  • Accepting contradictory boolean counts makes test_contradictory_boolean_bucket_invalidates_count fail. Dropping the writer check or both row-count consistency checks, accepting contradictory boolean counts, or dropping fallback rows makes test_serialized_footer_unavailable_or_contradictory_retains_rows fail.
  • Dropping fallback rows or swallowing I/O errors makes test_unavailable_metadata_falls_back_but_io_errors_propagate fail; skipping the safe-cast guard makes test_unsafe_output_cast_is_not_hidden_by_pruning fail; leaking the metadata handle makes test_statistics_handle_lifetime fail.

Representative commands:

python -m pytest -q \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_serialized_footer_statistics_and_file_exclusion \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_serialized_footer_unavailable_or_contradictory_retains_rows
python -m pytest -q \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_no_eligible_predicate_has_no_extra_statistics \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_unsafe_or_over_budget_integer_sets_are_unknown \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_independent_no_null_flag_and_contradictions \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_char_varchar_null_proof_does_not_enable_ranges \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_nan_is_not_null_and_float_ranges_stay_unknown \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_string_comparisons_and_integer_limits_against_arrow \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_qualified_java_and_unqualified_cpp_string_statistics \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_presence_counts_bounds_and_writer \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_null_counts_do_not_require_range_support \
  python/ray/data/tests/unit/datasource_v2/test_orc_statistics.py::test_default_and_rejected_stripe_mode \
  python/ray/data/tests/datasource/test_orc.py::test_orc_statistics_public_pipeline
python -m pytest -q python/ray/data/tests/datasource/test_orc.py::test_orc_file_statistics_pruning_public_worker
python -m pytest -q python/ray/data/tests/unit/datasource_v2/test_orc_datasource_v2.py::test_orc_batch_sizing_config_survives_pushdowns
python -m pytest -q python/ray/data/tests/datasource_v2/test_orc_statistics_s3.py

AI assistance: Codex assisted with implementation, tests and review.

Use bounded ORC tail metadata for file pruning and opt-in stripe reads. Preserve exact filtering, batching, limits, and existing read-task boundaries.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Compare off/file HTTP costs for retained files, low pruning rates, and small files with zero or 50 ms request delays. Record repeated timing and request metrics without imposing a speedup threshold.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Preserve adaptive batch sizing alongside ORC statistics pruning, cover their combined configuration and reads, and type dynamic metadata and test accounting for the cumulative Data checks.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Preserve schema-aware partition pruning alongside ORC file and stripe statistics pruning, and cover all pruning modes with Hive and directory layouts.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Nov 12, 2026
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2 jiangxt2 changed the title [Data] Prune ORC files and stripes using column statistics [Data] Prune ORC files using column statistics Nov 13, 2026
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

Copy link
Copy Markdown
Contributor Author

Follow-up: add stripe-level statistics pruning on top of #67278.

@github-actions github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Nov 13, 2026
@abhishekverma-ray

Copy link
Copy Markdown
Contributor

Thanks for these! The team is aligning on some Ray Data test conventions (#67919 / #67928). In short: each test should fail if the change is reverted, extend or parametrize existing tests before adding new ones, test through the public read_* APIs rather than asserting on internal mock calls, and use ray_start_regular_shared instead of starting new clusters. This PR has about 1.6× more test lines than code. Could you trim it to the tests that catch a revert, and note in the description which revert each one catches?

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants