Skip to content

GH-54921: [C++][Python][ORC] Add batch iteration over selected stripes - #54926

Open
jiangxt2 wants to merge 8 commits into
apache:mainfrom
jiangxt2:orc-stripe-batch-reader
Open

jiangxt2 wants to merge 8 commits into
apache:mainfrom
jiangxt2:orc-stripe-batch-reader

Conversation

@jiangxt2

@jiangxt2 jiangxt2 commented Oct 22, 2026 •

Copy link
Copy Markdown

Rationale for this change

ORCFile.read_stripe() materializes an entire stripe as one record batch. Dataset already scans ORC in batches; this adds native batching with explicit stripe selection to ORCFile, supporting incremental consumption and early termination.

Closes #54921.

What changes are included in this PR?

  • Add ORCFile.iter_batches(batch_size=65536, stripes=None, columns=None) with column projection.
  • Add a single stateless C++ GetStripeReader API with explicit optional column selection that decodes directly into batches without changing the reader's Seek position.
  • Handle zero-column row counts and iterator lifetimes, with API documentation and regression tests.

Are these changes tested?

Validated locally on macOS arm64 with matching Release C++/Python builds, Python 3.13.7 and ORC 2.2.1.

cpp/build/dev/release/arrow-orc-adapter-test

44 passed; one existing timezone-database test restricted to ORC < 2.0 was skipped.

From python/:

python -m pytest -q -ra --enable-orc --enable-dataset --enable-pandas pyarrow/tests/test_orc.py pyarrow/tests/test_dataset.py::test_orc_format pyarrow/tests/test_dataset.py::test_orc_scan_options pyarrow/tests/test_dataset.py::test_orc_writer_not_implemented_for_dataset

140 passed with NumPy 2.5.4. In an isolated environment without NumPy, the iter_batches tests and test_without_numpy.py had 122 passes; only the marked NumPy integer-input test was skipped. Coverage includes five codecs, stripe order and duplicates, batch limits, nested/zero-column projections, invalid arguments, interleaved iterators and read failures. Both environments cover owner lifetime, early close and resource release for whole-file and selected-stripe iteration.

The C++ memory regression uses four int64 columns and batches of 256 rows. For single stripes of 262144 and 1048576 rows, whole-stripe Arrow allocation peaks were 16.125 and 64.5 MiB; streaming peaked at 16.25 KiB for both sizes. NONE/ZLIB range tests verify that consuming the first batch and closing early does not read unselected stripe data.

Changed-file format/lint checks and git diff --check passed.

Are there any user-facing changes?

Yes. The new API yields RecordBatch objects. stripes=None reads all stripes; explicit sequences preserve order and duplicates, and empty sequences yield no batches. columns=[] preserves row counts. Closing the iterator releases its reader without closing the input; returned batches remain valid.

batch_size limits output rows, not total process memory. ORC buffers and dictionaries require additional memory, and small batches can reduce throughput. Reading the same stripes does not inherently reduce I/O.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

The human contributor directed the design and reviewed the proposal and PR text. OpenAI Codex assisted with code, tests, documentation and static review.

…stripes

Expose streaming batches from complete ORC files or selected stripes.
Reuse native decoding through a stateless stripe reader factory and
preserve row counts for zero-column projections.
Add validation, lifecycle, range-reading and memory regression coverage.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #54921 has been automatically assigned in GitHub to PR creator.

Generate shared fixture data with Python integers and isolate NumPy
integer arguments in a marked test. Cover whole-file and selected-stripe
iterator lifetimes across all five codecs.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Merge upstream main at d24791f.
Retain batch iteration and the removal of legacy ORC writer
argument-order compatibility.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Remove a redundant column-list copy and reduce duplicate test
combinations while preserving argument snapshots, codec coverage,
and both iterator lifecycle paths.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

@wgtmac wgtmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, @jiangxt2!

The C++ part looks good to me. I've left some comments with its API design and test pattern.

For the python code, perhaps @rok could help?

Comment thread cpp/src/arrow/adapters/orc/adapter.h Resolved
batches.reset(); // Early stop must not read the rest or another stripe.
ASSERT_EQ(reads, tracked->num_reads());
}
for (const auto& range : tracked->get_read_ranges()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ResetStats() is outside the loop and the final check only excludes stripes 1 and 3. As a result, reading stripe 0 while selecting stripe 2, or vice versa, would still pass. Could we reset inside each iteration and compare every read range against all stripes except the one currently selected?

@jiangxt2 jiangxt2 Nov 1, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ResetStats() now runs inside each iteration, and every read range is checked against all stripes except the selected one. This catches unintended reads between stripes 0 and 2 as well. The updated test passes for NONE and ZLIB, including the early-close assertion.

Comment thread python/pyarrow/orc.py Resolved
Comment thread python/pyarrow/_orc.pyx Resolved
@jiangxt2

Copy link
Copy Markdown
Author

Thanks for working on this, @jiangxt2!

The C++ part looks good to me. I've left some comments with its API design and test pattern.

For the python code, perhaps @rok could help?

@wgtmac Thanks for the detailed review! I will go through all the comments.

Use optional column selection, strengthen per-stripe read-range
assertions, and explicitly reject mixed Python column selectors.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

@dmitry-chirkov-dremio dmitry-chirkov-dremio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Selection and reader lifetime are covered well.
Memory numbers hold up in local tests too.

Comment thread python/pyarrow/orc.py Resolved
Comment thread python/pyarrow/orc.py Resolved
Comment thread python/pyarrow/_orc.pyx Resolved
Comment thread python/pyarrow/_orc.pyx Resolved
…tion

Validate mixed column selectors in the shared ORC helper, remove
redundant checks and test diagnostics, and cover all three read APIs.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Trigger a new CI run after unrelated infrastructure failures.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2

jiangxt2 commented Nov 3, 2026

Copy link
Copy Markdown
Author

The three CI failures appear unrelated to this PR:

  • macOS Python: a timeout in the shared MinIO test setup, also seen on other PRs.
  • Windows GLib: HTTP 503 while downloading vcpkg, before Arrow compilation.
  • Windows RTools 40: HTTP 404 from an outdated download URL in setup-r, being addressed in #55378.

Once the RTools issue is resolved on main, I'll update this branch and check CI again.

@raulcd

raulcd commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

The R failure is currently tracked here:

The other two seem to be just flaky transient errors.

Merge upstream main to include the RTools 40 CI fix while preserving
the ORC batch reader changes.

Generated-by: OpenAI Codex
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2

jiangxt2 commented Nov 4, 2026

Copy link
Copy Markdown
Author

I’ve merged main into this branch to pick up the Windows RTools 40 download fix in #55378. I also verified the corrected archive URL in a local Windows 11 ARM VM: a 1 KiB range request returned HTTP 206 with the expected MZ signature.

@jiangxt2

jiangxt2 commented Nov 4, 2026

Copy link
Copy Markdown
Author

These two seem to be just flaky transient errors, too.
Could you rerun just these two failed jobs when convenient? Thanks! @raulcd

@jiangxt2

jiangxt2 commented Nov 5, 2026

Copy link
Copy Markdown
Author

The macOS C++ CI jobs are timing out in arrow-s3fs-test, and the ARM job also times out in arrow-s3fs-module-test. I found similar failures on main and in #55197.

Is there anything I can do to help resolve these CI failures?

@dmitry-chirkov-dremio

Copy link
Copy Markdown
Contributor

My approval still stands.

@wgtmac

wgtmac commented Nov 18, 2026

Copy link
Copy Markdown
Member

@raulcd @rok Does the python side look good to you?

@raulcd raulcd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concerns were solved with the latest changes. I am not an expert on this area but I don't have more concerns

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Python][ORC] Support batch iteration over selected stripes

4 participants