Repository navigation
Conversation
Use bounded ORC tail metadata for file pruning and opt-in stripe reads. Preserve exact filtering, batching, limits, and existing read-task boundaries. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Compare off/file HTTP costs for retained files, low pruning rates, and small files with zero or 50 ms request delays. Record repeated timing and request metrics without imposing a speedup threshold. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Preserve adaptive batch sizing alongside ORC statistics pruning, cover their combined configuration and reads, and type dynamic metadata and test accounting for the cumulative Data checks. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Preserve schema-aware partition pruning alongside ORC file and stripe statistics pruning, and cover all pruning modes with Hive and directory layouts. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
Follow-up: add stripe-level statistics pruning on top of #67278. |
|
Thanks for these! The team is aligning on some Ray Data test conventions (#67919 / #67928). In short: each test should fail if the change is reverted, extend or parametrize existing tests before adding new ones, test through the public read_* APIs rather than asserting on internal mock calls, and use ray_start_regular_shared instead of starting new clusters. This PR has about 1.6× more test lines than code. Could you trim it to the tests that catch a revert, and note in the description which revert each one catches? |
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Description
Use ORC file statistics to skip files that cannot match a pushed filter, while retaining the complete Arrow row filter and existing read-task boundaries.
Before
ORC DataSourceV2 scans whole-file fragments and applies row filters after reading. Selective predicates cannot use file or stripe statistics to avoid data decoding.
After
DataContext.orc_statistics_pruning="off"performs no added statistics reads. Explicit"file"skips proven nonmatching files and streams surviving files.Subsystems
Subsystems: IO and API configuration. The ORC reader carries the behavior change; DataContext adds the pruning mode.
Data flow
flowchart TD subgraph Before["Before: IO / ORC reader"] B1["Whole-file Arrow scan"] --> B2["Exact filter and projection"] end subgraph After["After: IO / ORC reader"] A1["Predicate and file schema"] --> A2{"Pruning enabled and eligible?"} A2 -->|no| A3["Whole-file Arrow scan"] A2 -->|yes| A4["Serialized footer statistics"] A4 -->|file cannot match| A5["Skip file"] A4 -->|surviving or unknown| A3 A3 --> A8["Exact filter, projection, batches and limit"] end classDef added fill:#c8e6c9,stroke:#2e7d32,color:#000 class A2,A4,A5 addedRelated issues
Addresses #66995. Statistics pruning is implemented within existing file read tasks, retaining whole-file manifests and the default streaming path without requiring stripe task splitting. Cross-task stripe splitting is tracked separately in #66999.
Additional information
PyArrow API and maintenance boundary
PyArrow 23.0.1 exposes no public per-column ORC min/max/null-statistics API. This change reuses the private
ORCFile.reader.serialized_file_tail()interface and parses a limited file-footer protobuf subset, preserving field presence and writer/type qualification. PyArrow still performs data decoding and exact row filtering.A suitable public Python statistics API can replace the metadata adapter; related upstream statistics work is tracked in apache/arrow#49360. Native predicate evaluation can replace Ray's evaluator only if its conservative type/NULL semantics match.
Limits
STRING range pruning is limited to exact bounds from the qualified Java ORC 2.2.1 writer. Missing, truncated, untrusted or unsupported statistics stay unknown for the corresponding predicate leaves. Unavailable statistics fall back to streaming; real filesystem errors propagate through existing retry/error handling.
Enabling file pruning can add metadata requests, and planning samples can still read data from files excluded during execution.
Testing
Linux arm64, Python 3.12, PyArrow 23.0.1; Ray Data sources were used with a separate development Ray Core wheel:
After configuration-test consolidation, 6 configuration cases passed on macOS arm64, Python 3.13.7 and PyArrow 23.0.1 with a separate development Ray Core wheel; seven isolated in-process fault injections were detected.
Black, Ruff and Pyrefly checks passed for the changed tests. The targeted regressions below were checked locally in isolated source copies or with isolated in-process fault injection;
offparameters also protect streaming behavior and need not fail when file pruning is bypassed. The public worker test uses scanner instrumentation to observe actual data scans; the Moto test exercises the internal scanner with the existing S3 fixture.test_orc_file_statistics_pruning_public_worker,test_logical_partition_pruning_and_statistics,test_orc_file_statistics_s3,test_extended_predicates_exclude_whole_fileandtest_serialized_footer_statistics_and_file_exclusionfail for thefilecases.filemode:test_orc_statistics_public_pipelinefails; it also protects filtering, projection and limit for retained files.offmode or for an unsupported predicate:test_no_eligible_predicate_has_no_extra_statisticsfails.off:test_orc_batch_sizing_config_survives_pushdownsfails; it also checks mode preservation after pushdown and serialization. Changing the default or accepting invalid modes makestest_default_and_rejected_stripe_modefail.test_possible_values_against_arrow_oraclefails; changing the<=boundary makestest_string_comparisons_and_integer_limits_against_arrowfail.test_unsafe_or_over_budget_integer_sets_are_unknownfails; accepting contradictory null counts makestest_independent_no_null_flag_and_contradictionsfail.test_char_varchar_null_proof_does_not_enable_ranges,test_nan_is_not_null_and_float_ranges_stay_unknownortest_null_counts_do_not_require_range_supportfail, respectively.test_qualified_java_and_unqualified_cpp_string_statisticsfail; trusting a truncated lower bound makestest_presence_counts_bounds_and_writerfail.test_contradictory_boolean_bucket_invalidates_countfail. Dropping the writer check or both row-count consistency checks, accepting contradictory boolean counts, or dropping fallback rows makestest_serialized_footer_unavailable_or_contradictory_retains_rowsfail.test_unavailable_metadata_falls_back_but_io_errors_propagatefail; skipping the safe-cast guard makestest_unsafe_output_cast_is_not_hidden_by_pruningfail; leaking the metadata handle makestest_statistics_handle_lifetimefail.Representative commands:
AI assistance: Codex assisted with implementation, tests and review.