Repository navigation
Conversation
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Move the ORC reader and scanner into the current DataSourceV2 format layout. Handle projected synthesized columns missing from the input schema, align synthesized output types, and call the base batch hook before applying the ORC pickle check. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Read stripe metadata through optional PyORC and partition work across existing DataSourceV2 read units. Decode assigned stripes with PyArrow while preserving whole-file fallback, filtering, ordering and retries. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Reuse schema-time layouts and plan stripe manifests with bounded metadata workers and deterministic online grouping. Preserve exact read-unit positions and existing filtering, projection, retry, and limit behavior. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Retain stripe selection and upstream adaptive batch sizing. Add regression coverage for their interaction, preserve zero-limit early stopping, and align test annotations with the cumulative CI type gate. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
There was a problem hiding this comment.
Code Review
This pull request introduces stripe-level scheduling and reading for ORC files in Ray Data (V2 Datasource) when the optional pyorc library is installed. It adds pyorc to test requirements and lockfiles, implements OrcStripeIndexer and OrcStripePartitioner to discover and group ORC layouts by stripe, updates OrcFileReader to materialize and decode individual stripes sequentially, and extends FileManifest and FileChunk to track physical stripe boundaries. It also includes comprehensive unit and integration tests verifying stripe reconstruction, metadata caching, and lineage recovery. Since there are no review comments provided, I have no additional feedback to offer.
Merge master and simplify stripe planning around Arrow footer metadata and native batch iteration. Preserve whole-file fallback, bounded metadata planning, replay, read-unit positions, and shared output semantics. Add focused public API and safety regressions. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Read assigned stripes with the existing PyArrow decoder and retain shared schema, projection, position, retry, and cleanup semantics. Restore physical columns when schema sampling finds no effective physical schema. Reuse Arrow ORC test fixtures and remove the PyORC requirement and generated lock entries. Write Moto service logs to a file to avoid blocking the S3 test fixture. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Register metadata actors through the existing runtime context API and remove the test-only actor creation wrappers. Preserve production code and all test assertions. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Recognize declared partition fields on unpartitioned paths and retain logical types when adding physical columns. Extend existing parameterized coverage for whole-file and stripe reads, null filling, and explicit empty projections. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Retrigger PR CI after the Data docs job failed fetching Go SDK metadata with HTTP 503. Keep the source tree unchanged. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
Hi @jiangxt2. Thank you for your contribution. |
Thanks for taking a look! @AarryaSaraf The design was inspired by Spark’s approach of enabling parallel reads within a file while keeping stripes as the physical read units. The implementation differs partly because the currently released PyArrow API does not expose an interface that combines selecting specific stripes with incremental batch decoding. I opened apache/arrow#54926 to address that limitation. For now, this PR uses the released read_stripe API, which materializes the selected stripe before processing it in batches. Some of the code is needed to integrate stripe reads with Ray Data’s existing behavior, including projection and filtering, partition columns, retries, and avoiding missing or duplicate rows. The PR also includes metadata discovery and reuse optimizations, which add to the size but could be reviewed separately from the core functionality. A possible split would be:
Each PR would include the tests needed for its own behavior. Does this split sound reasonable, or would you prefer a different boundary? |
|
Thanks for the proposal. I'd prefer a different boundary: a simple implementation first, then the optimizations on top of it, each with a benchmark against the simple version.
|
Thanks, I agree with this boundary. I’ll trim this PR to a simple end-to-end implementation: listing reads each file’s stripe layout and emits stripe chunks, the existing OnlineBinPacker groups them using a fixed encoded-byte target, and the reader uses PyArrow’s read_stripe() to read its assigned stripes. I’ll leave out metadata actors, footer reuse, the prepare_read hook, and the separate ordered path. I’ll keep focused tests for task splitting, row correctness, retries, and whole-file fallback, including schema and partition handling. Once that baseline is in place, I’d evaluate the following optimizations separately:
Each follow-up would include a benchmark against the simple version. If an optimization depends on another, I’d also measure its incremental benefit over that prerequisite. |
Merge upstream master and schedule ORC stripes with a single weighted round-robin bucket. Preserve schema and partition handling, retries, duplicate inputs, and whole-file fallback while keeping manifest typing local to ORC. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 4a7e82a. Configure here.
| execution_idx=0, | ||
| excluded_read_unit_ids=None, | ||
| ): | ||
| files = super().list_files( |
There was a problem hiding this comment.
Listing no longer streams under limit
Medium Severity
list_files now waits on super().list_files(), which batches paths in _process_file_infos_to_manifests and only emits the remainder after discovery finishes. A pushed-down limit therefore cannot stop listing after the first matching file: later missing paths can fail the job, and directories under 1000 files are fully statted before any ORC footer is read.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 4a7e82a. Configure here.
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
I’ve updated the PR to use In testing, stripe packing changed the observed order of rows from the same file, even with |
|
Thanks for trimming this down. Three design-level issues before I can review the code:
|
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Thanks, @AarryaSaraf. I’ve updated the PR to use whole-file streaming for small files and other cases that are not assigned stripe chunks. I’d appreciate your input on two remaining tradeoffs:
|
|
Thanks for these! The team is aligning on some Ray Data test conventions (#67919 / #67928). In short: each test should fail if the change is reverted, extend or parametrize existing tests before adding new ones, test through the public read_* APIs rather than asserting on internal mock calls, and use ray_start_regular_shared instead of starting new clusters. This PR has about 1.9× more test lines than code. Could you trim it to the tests that catch a revert, and note in the description which revert each one catches? |
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>


Summary
Add opt-in stripe-level scheduling for ORC files, allowing one file to produce multiple read tasks. The default remains whole-file streaming.
Closes #66999. Statistics-based stripe pruning is outside the scope of this PR and tracked separately in #66995.
Before
Each ORC file is one scheduling unit, so a single large file cannot supply multiple read tasks.
After
enable_stripe_reads=True, discover stripe layouts and split eligible files larger than 128 MiB with multiple stripes. Small files, safe unfiltered limits, and files with missing or excessive decoded-size estimates use whole-file streaming. Existing stripe exclusions require stripe assignments or an explicit failure.WeightedRoundRobinPartitionerwith one bucket and a 128 MiB encoded-byte target to group consecutive whole-file and stripe entries. Both ordered and unordered reads use this path;preserve_order=Truedelivers results in planned input order.ORCFile.read_stripe()API. If the required APIs are unavailable during planning, retain whole-file scheduling. If they are unavailable after a stripe has been assigned, fail rather than silently reading the whole file.The PR does not introduce metadata actors, schema-sampling footer reuse, a
prepare_readhook, or a separate ordered path. It does not change the sharedFileManifestinterface or add a PyORC dependency.Ordered tests check uneven stripes, mixed files, repeated paths, and limit prefixes. This preserves the listing input order without promising a globally sorted filesystem listing.
Subsystems
IO: ORC datasource, listing, partitioning, and reader. API:
read_orcdocumentation.Data flow
flowchart LR A[ORC paths] -- default --> F[Whole-file streaming] A -- opt-in --> B[Listing and stripe layout] B --> C[Stripe chunks or whole-file entries] C --> D[Consecutive single-bucket grouping] D --> E[Read tasks]Limitations and follow-ups
ORCReader.serialized_file_tail()capability; stripe decoding uses the publicread_stripe()API.read_stripe()materializes selected columns of a stripe before batching. The opt-in guard estimates stripe size using file-level statistics andtarget_max_block_size; an individual stripe may exceed that estimate, so it is not a memory bound.Related work
#67105 also implements stripe-level ORC reads for #66999. It uses
OnlineBinPackerand estimates stripe sizes; this PR reads stripe layout from the footer and groups entries consecutively.Testing
Targeted tests used Python 3.12, PyArrow 17.0.0 / 23.0.1, the worktree Ray Data code, and cached Ray Core
891a5a89rather than source baseline11a0a607. A complete ORC suite was not run on the final commit.PyArrow 17: four targeted public multi-stripe / mixed-file cases passed; these do not cover the final
count()assertion.PyArrow 23: the public contract and uneven-stripe tests passed. The
rowscase also passed withcount()on the unmaterialized dataset. Targeted AA/ABA and Moto-backed S3 tests passed.Regression coverage (fault injection verified ordering, disabled splitting, duplicate paths, and empty projection; other entries describe the assertions):
test_read_orc_multiple_stripes: disabled stripe splitting or missing rows.test_orc_stripe_public_contract: mixed-file rows/count/filter-limit, ordered output, empty/path/partition projections, and disabled splitting.test_read_orc_empty_schema_sample_keeps_late_columnsandtest_read_orc_merge_schema_discovers_and_reads_all_columns: lost physical columns and schema merging.test_orc_uneven_stripes_packing_orderandtest_orc_repeated_multi_stripe_paths: reordered uneven stripes, repeated occurrences, and ordered limit/filter prefixes.test_orc_stripe_remote_filesystem: splitting and complete reads through the Moto S3 fixture.test_optional_zero_footer_fields,test_footer_layout_validation, andtest_bad_footer_closes_source_and_keeps_path: valid zero fields, invalid layouts, and error-path cleanup.test_stripe_listing_and_limitandtest_listing_limit_stops_before_later_invalid_footer: splitting, exclusions, filter/limit, and footer early stopping.test_unsafe_stripe_estimate_falls_back_or_rejects_resume,test_small_file_skips_footer_and_missing_layout_rejects_exclusion, andtest_missing_api_freezes_whole_file_and_rejects_assignments: safe fallback or explicit failure on resume.test_packed_small_files_keep_duplicate_paths: mixed whole-file and excluded-stripe occurrences.test_schema_fallback_keeps_physical_columnsandtest_late_physical_columns_follow_schema: physical columns, types, missing columns, and empty projections.test_prelisted_shuffle_excludes_files_before_shufflingandtest_unordered_listing_replay_is_stable: deterministic listing and exclusion replay.test_stripe_positions_filter_and_retry,test_stripe_input_lifetime, andtest_whole_file_boundary_mixed_limit_and_manifest_compatibility: retry/positions, resource cleanup, and mixed-file limit boundaries.test_resolve_partitioning_and_reuse_generic_file_components,test_read_orc_routes_to_v2_and_forwards_read_options, andtest_read_orc_rejects_full_schema_on_v1: default/opt-in routing and V1 rejection.test_orc_filter_cannot_hide_decoded_pickle_columnandtest_orc_reader_initial_estimate_is_cached: pickle-column checks and reader estimate caching.A delayed-first-task experiment observed overlapping decoder processes and ordered output. The available normal-run measurements do not establish a throughput gain.
Pre-commit and pre-push lint and DCO checks passed. The pre-push Bazel query and unit-test layers were marked manual due to the local environment.
The listing-worker failure and two-node manifest reconstruction tests are deferred; these distributed recovery scenarios are not directly validated by this PR.
AI assistance
Codex assisted with implementation, testing, and review.