Skip to content

[Data] Add stripe-level read units for ORC - #67278

Draft
jiangxt2 wants to merge 33 commits into
ray-project:masterfrom
jiangxt2:feature/orc-stripe-read-units
Draft

jiangxt2 wants to merge 33 commits into
ray-project:masterfrom
jiangxt2:feature/orc-stripe-read-units

Conversation

@jiangxt2

@jiangxt2 jiangxt2 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add opt-in stripe-level scheduling for ORC files, allowing one file to produce multiple read tasks. The default remains whole-file streaming.

Closes #66999. Statistics-based stripe pruning is outside the scope of this PR and tracked separately in #66995.

Before

Each ORC file is one scheduling unit, so a single large file cannot supply multiple read tasks.

After

  • With enable_stripe_reads=True, discover stripe layouts and split eligible files larger than 128 MiB with multiple stripes. Small files, safe unfiltered limits, and files with missing or excessive decoded-size estimates use whole-file streaming. Existing stripe exclusions require stripe assignments or an explicit failure.
  • An ORC-local adapter uses the existing WeightedRoundRobinPartitioner with one bucket and a 128 MiB encoded-byte target to group consecutive whole-file and stripe entries. Both ordered and unordered reads use this path; preserve_order=True delivers results in planned input order.
  • Read assigned stripes using PyArrow’s released ORCFile.read_stripe() API. If the required APIs are unavailable during planning, retain whole-file scheduling. If they are unavailable after a stripe has been assigned, fail rather than silently reading the whole file.
  • Preserve schema alignment, projection, filtering, partition handling, retries, and repeated input paths.

The PR does not introduce metadata actors, schema-sampling footer reuse, a prepare_read hook, or a separate ordered path. It does not change the shared FileManifest interface or add a PyORC dependency.

Ordered tests check uneven stripes, mixed files, repeated paths, and limit prefixes. This preserves the listing input order without promising a globally sorted filesystem listing.

Subsystems

IO: ORC datasource, listing, partitioning, and reader. API: read_orc documentation.

Data flow

flowchart LR
    A[ORC paths] -- default --> F[Whole-file streaming]
    A -- opt-in --> B[Listing and stripe layout]
    B --> C[Stripe chunks or whole-file entries]
    C --> D[Consecutive single-bucket grouping]
    D --> E[Read tasks]
Loading

Limitations and follow-ups

  • Stripe layout discovery uses Arrow's private ORCReader.serialized_file_tail() capability; stripe decoding uses the public read_stripe() API.
  • read_stripe() materializes selected columns of a stripe before batching. The opt-in guard estimates stripe size using file-level statistics and target_max_block_size; an individual stripe may exceed that estimate, so it is not a memory bound.
  • Listing discovers metadata serially, and read tasks open their own files and footers.
  • Without a pushed-down predicate, listing can stop reading footers once enough rows are found for a limit. File discovery may still batch paths ahead.
  • Parallel metadata discovery, reuse of layouts obtained during schema sampling, and adaptive grouping targets can be evaluated separately against this baseline.
  • Incremental stripe decoding, including the potential use of GH-54921: [C++][Python][ORC] Add batch iteration over selected stripes apache/arrow#54926, is future work and is not required by this PR.

Related work

#67105 also implements stripe-level ORC reads for #66999. It uses OnlineBinPacker and estimates stripe sizes; this PR reads stripe layout from the footer and groups entries consecutively.

Testing

Targeted tests used Python 3.12, PyArrow 17.0.0 / 23.0.1, the worktree Ray Data code, and cached Ray Core 891a5a89 rather than source baseline 11a0a607. A complete ORC suite was not run on the final commit.

  • PyArrow 17: four targeted public multi-stripe / mixed-file cases passed; these do not cover the final count() assertion.

  • PyArrow 23: the public contract and uneven-stripe tests passed. The rows case also passed with count() on the unmaterialized dataset. Targeted AA/ABA and Moto-backed S3 tests passed.
    Regression coverage (fault injection verified ordering, disabled splitting, duplicate paths, and empty projection; other entries describe the assertions):

  • test_read_orc_multiple_stripes: disabled stripe splitting or missing rows.

  • test_orc_stripe_public_contract: mixed-file rows/count/filter-limit, ordered output, empty/path/partition projections, and disabled splitting.

  • test_read_orc_empty_schema_sample_keeps_late_columns and test_read_orc_merge_schema_discovers_and_reads_all_columns: lost physical columns and schema merging.

  • test_orc_uneven_stripes_packing_order and test_orc_repeated_multi_stripe_paths: reordered uneven stripes, repeated occurrences, and ordered limit/filter prefixes.

  • test_orc_stripe_remote_filesystem: splitting and complete reads through the Moto S3 fixture.

  • test_optional_zero_footer_fields, test_footer_layout_validation, and test_bad_footer_closes_source_and_keeps_path: valid zero fields, invalid layouts, and error-path cleanup.

  • test_stripe_listing_and_limit and test_listing_limit_stops_before_later_invalid_footer: splitting, exclusions, filter/limit, and footer early stopping.

  • test_unsafe_stripe_estimate_falls_back_or_rejects_resume, test_small_file_skips_footer_and_missing_layout_rejects_exclusion, and test_missing_api_freezes_whole_file_and_rejects_assignments: safe fallback or explicit failure on resume.

  • test_packed_small_files_keep_duplicate_paths: mixed whole-file and excluded-stripe occurrences.

  • test_schema_fallback_keeps_physical_columns and test_late_physical_columns_follow_schema: physical columns, types, missing columns, and empty projections.

  • test_prelisted_shuffle_excludes_files_before_shuffling and test_unordered_listing_replay_is_stable: deterministic listing and exclusion replay.

  • test_stripe_positions_filter_and_retry, test_stripe_input_lifetime, and test_whole_file_boundary_mixed_limit_and_manifest_compatibility: retry/positions, resource cleanup, and mixed-file limit boundaries.

  • test_resolve_partitioning_and_reuse_generic_file_components, test_read_orc_routes_to_v2_and_forwards_read_options, and test_read_orc_rejects_full_schema_on_v1: default/opt-in routing and V1 rejection.

  • test_orc_filter_cannot_hide_decoded_pickle_column and test_orc_reader_initial_estimate_is_cached: pickle-column checks and reader estimate caching.

  • A delayed-first-task experiment observed overlapping decoder processes and ordered output. The available normal-run measurements do not establish a throughput gain.

  • Pre-commit and pre-push lint and DCO checks passed. The pre-push Bazel query and unit-test layers were marked manual due to the local environment.

python -m pytest -q -p no:cacheprovider \
  python/ray/data/tests/datasource/test_orc.py::test_orc_stripe_public_contract -k rows

The listing-worker failure and two-node manifest reconstruction tests are deferred; these distributed recovery scenarios are not directly validated by this PR.

AI assistance

Codex assisted with implementation, testing, and review.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Move the ORC reader and scanner into the current DataSourceV2 format layout. Handle projected synthesized columns missing from the input schema, align synthesized output types, and call the base batch hook before applying the ORC pickle check.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Read stripe metadata through optional PyORC and partition work across
existing DataSourceV2 read units. Decode assigned stripes with PyArrow
while preserving whole-file fallback, filtering, ordering and retries.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Reuse schema-time layouts and plan stripe manifests with bounded metadata
workers and deterministic online grouping. Preserve exact read-unit
positions and existing filtering, projection, retry, and limit behavior.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Retain stripe selection and upstream adaptive batch sizing. Add regression
coverage for their interaction, preserve zero-limit early stopping, and
align test annotations with the cumulative CI type gate.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2
jiangxt2 marked this pull request as ready for review November 2, 2026 09:10
@jiangxt2
jiangxt2 requested a review from a team as a code owner November 2, 2026 09:10

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces stripe-level scheduling and reading for ORC files in Ray Data (V2 Datasource) when the optional pyorc library is installed. It adds pyorc to test requirements and lockfiles, implements OrcStripeIndexer and OrcStripePartitioner to discover and group ORC layouts by stripe, updates OrcFileReader to materialize and decode individual stripes sequentially, and extends FileManifest and FileChunk to track physical stripe boundaries. It also includes comprehensive unit and integration tests verifying stripe reconstruction, metadata caching, and lineage recovery. Since there are no review comments provided, I have no additional feedback to offer.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Nov 2, 2026
Merge master and simplify stripe planning around Arrow footer metadata
and native batch iteration. Preserve whole-file fallback, bounded metadata
planning, replay, read-unit positions, and shared output semantics.
Add focused public API and safety regressions.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2
jiangxt2 marked this pull request as draft November 2, 2026 17:12
Read assigned stripes with the existing PyArrow decoder and retain shared
schema, projection, position, retry, and cleanup semantics. Restore physical
columns when schema sampling finds no effective physical schema. Reuse Arrow
ORC test fixtures and remove the PyORC requirement and generated lock entries.
Write Moto service logs to a file to avoid blocking the S3 test fixture.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Register metadata actors through the existing runtime context API and remove
the test-only actor creation wrappers. Preserve production code and all test
assertions.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2
jiangxt2 marked this pull request as ready for review November 3, 2026 06:39
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Recognize declared partition fields on unpartitioned paths and retain logical
types when adding physical columns. Extend existing parameterized coverage
for whole-file and stripe reads, null filling, and explicit empty projections.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Retrigger PR CI after the Data docs job failed fetching Go SDK metadata
with HTTP 503. Keep the source tree unchanged.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@AarryaSaraf

Copy link
Copy Markdown
Contributor

Hi @jiangxt2. Thank you for your contribution.
This diff seems too large to review. Whats the reason behind the size? I believe strip level reading can be done in a much smaller way. We should simplify the design and code or you could reply with why you think its necessary and we could break this into about 3 PRs.

@jiangxt2

Copy link
Copy Markdown
Contributor Author

Hi @jiangxt2. Thank you for your contribution.

This diff seems too large to review. Whats the reason behind the size? I believe strip level reading can be done in a much smaller way. We should simplify the design and code or you could reply with why you think its necessary and we could break this into about 3 PRs.

Thanks for taking a look! @AarryaSaraf

The design was inspired by Spark’s approach of enabling parallel reads within a file while keeping stripes as the physical read units. The implementation differs partly because the currently released PyArrow API does not expose an interface that combines selecting specific stripes with incremental batch decoding. I opened apache/arrow#54926 to address that limitation. For now, this PR uses the released read_stripe API, which materializes the selected stripe before processing it in batches.

Some of the code is needed to integrate stripe reads with Ray Data’s existing behavior, including projection and filtering, partition columns, retries, and avoiding missing or duplicate rows. The PR also includes metadata discovery and reuse optimizations, which add to the size but could be reviewed separately from the core functionality.

A possible split would be:

  1. Add the stripe metadata and reader foundations, with focused tests.
  2. Connect stripe-level planning and scheduling, with tests for correctness, fallback, and retries.
  3. Add metadata discovery and reuse optimizations separately.

Each PR would include the tests needed for its own behavior. Does this split sound reasonable, or would you prefer a different boundary?

@AarryaSaraf

Copy link
Copy Markdown
Contributor

Thanks for the proposal. I'd prefer a different boundary: a simple implementation first, then the optimizations on top of it, each with a benchmark against the simple version.

  1. Simple: listing reads each file's stripe layout and emits stripe chunks, the existing OnlineBinPacker packs them, and the reader reads its assigned stripes. No metadata actors, footer reuse, prepare_read hook, or separate ordered path.
  2. Follow-ups: each optimization (metadata actors, footer reuse, the small-read budget, ...) with numbers showing what it saves over (1).

Copy link
Copy Markdown
Contributor Author

a simple implementation first, then the optimizations on top of it, each with a benchmark against the simple version.

Thanks, I agree with this boundary. I’ll trim this PR to a simple end-to-end implementation: listing reads each file’s stripe layout and emits stripe chunks, the existing OnlineBinPacker groups them using a fixed encoded-byte target, and the reader uses PyArrow’s read_stripe() to read its assigned stripes. I’ll leave out metadata actors, footer reuse, the prepare_read hook, and the separate ordered path. I’ll keep focused tests for task splitting, row correctness, retries, and whole-file fallback, including schema and partition handling.

Once that baseline is in place, I’d evaluate the following optimizations separately:

  1. Parallel metadata discovery: use bounded metadata workers and measure listing time and time to first output, especially for remote files.
  2. Footer reuse: reuse stripe layouts obtained during schema sampling to avoid rereading those footers during listing. Measure the metadata requests and bytes saved, and justify any shared interface changes in that PR.
  3. Adaptive packing for one or a few files: adjust the encoded-byte target and compare task count, parallelism, and end-to-end read time with the fixed-target baseline.

Each follow-up would include a benchmark against the simple version. If an optimization depends on another, I’d also measure its incremental benefit over that prerequisite.

Merge upstream master and schedule ORC stripes with a single weighted
round-robin bucket. Preserve schema and partition handling, retries, duplicate
inputs, and whole-file fallback while keeping manifest typing local to ORC.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 4a7e82a. Configure here.

execution_idx=0,
excluded_read_unit_ids=None,
):
files = super().list_files(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Listing no longer streams under limit

Medium Severity

list_files now waits on super().list_files(), which batches paths in _process_file_infos_to_manifests and only emits the remainder after discovery finishes. A pushed-down limit therefore cannot stop listing after the first matching file: later missing paths can fail the job, and directories under 1000 files are fully statted before any ORC footer is read.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 4a7e82a. Configure here.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

Copy link
Copy Markdown
Contributor Author

the existing OnlineBinPacker packs them

I’ve updated the PR to use OnlineBinPacker, with a small ORC-local adapter to normalize manifest types.

In testing, stripe packing changed the observed order of rows from the same file, even with preserve_order=True. Full reads still returned all rows.

@AarryaSaraf

AarryaSaraf commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for trimming this down. Three design-level issues before I can review the code:

  1. Row order. With preserve_order=True, rows now come back out of order, even within one file.
  2. Memory on files that aren't split. A multi-stripe file smaller than the 128 MiB target stays in one read task, but now goes through read_stripe, which decodes a whole stripe at once instead of streaming batches. A 5-stripe, 18 MiB file: one read task either way, peak memory 74 MiB on master vs 1,067 MiB here.
  3. Size.: Can you explain why this needs so many LoC? Intuitively I would believe the PR has a lot of redundancies.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2

Copy link
Copy Markdown
Contributor Author

Thanks for trimming this down. Three design-level issues before I can review the code:

Thanks, @AarryaSaraf. I’ve updated the PR to use whole-file streaming for small files and other cases that are not assigned stripe chunks. I’d appreciate your input on two remaining tradeoffs:

  1. Order: I’m considering passing one ordered, coalesced stripe run per file to OnlineBinPacker, using one open bin and flushing at file boundaries. Ordered and unordered reads would share the same path, and tasks could still run in parallel. This could preserve input order, but would give up cross-file packing and may increase task counts for small files.

  2. Large-stripe memory: read_stripe() materializes the selected columns of a whole stripe before batching. In a reader-level PyArrow 23 stress test with a predicate and limit(1), the stripe path was OOM-killed under a 640 MiB container limit, while the master whole-file reader returned the row. Choosing whole-file streaming before assigning stripe tasks would avoid this materialization, but encoded size does not reliably bound decoded memory, and a broad fallback could sacrifice stripe parallelism.

@abhishekverma-ray

Copy link
Copy Markdown
Contributor

Thanks for these! The team is aligning on some Ray Data test conventions (#67919 / #67928). In short: each test should fail if the change is reverted, extend or parametrize existing tests before adding new ones, test through the public read_* APIs rather than asserting on internal mock calls, and use ray_start_regular_shared instead of starting new clusters. This PR has about 1.9× more test lines than code. Could you trim it to the tests that catch a revert, and note in the description which revert each one catches?

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2
jiangxt2 requested a review from a team as a code owner November 18, 2026 03:54
@jiangxt2
jiangxt2 marked this pull request as draft November 18, 2026 06:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Data] Split ORC DataSourceV2 read tasks by stripe

3 participants