Repository navigation
Conversation
…stripes Expose streaming batches from complete ORC files or selected stripes. Reuse native decoding through a stateless stripe reader factory and preserve row counts for zero-column projections. Add validation, lifecycle, range-reading and memory regression coverage. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
|
Generate shared fixture data with Python integers and isolate NumPy integer arguments in a marked test. Cover whole-file and selected-stripe iterator lifetimes across all five codecs. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Merge upstream main at d24791f. Retain batch iteration and the removal of legacy ORC writer argument-order compatibility. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Remove a redundant column-list copy and reduce duplicate test combinations while preserving argument snapshots, codec coverage, and both iterator lifecycle paths. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
| batches.reset(); // Early stop must not read the rest or another stripe. | ||
| ASSERT_EQ(reads, tracked->num_reads()); | ||
| } | ||
| for (const auto& range : tracked->get_read_ranges()) { |
There was a problem hiding this comment.
ResetStats() is outside the loop and the final check only excludes stripes 1 and 3. As a result, reading stripe 0 while selecting stripe 2, or vice versa, would still pass. Could we reset inside each iteration and compare every read range against all stripes except the one currently selected?
There was a problem hiding this comment.
ResetStats() now runs inside each iteration, and every read range is checked against all stripes except the selected one. This catches unintended reads between stripes 0 and 2 as well. The updated test passes for NONE and ZLIB, including the early-close assertion.
Use optional column selection, strengthen per-stripe read-range assertions, and explicitly reject mixed Python column selectors. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
dmitry-chirkov-dremio
left a comment
There was a problem hiding this comment.
LGTM.
Selection and reader lifetime are covered well.
Memory numbers hold up in local tests too.
…tion Validate mixed column selectors in the shared ORC helper, remove redundant checks and test diagnostics, and cover all three read APIs. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Trigger a new CI run after unrelated infrastructure failures. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
The three CI failures appear unrelated to this PR:
Once the RTools issue is resolved on main, I'll update this branch and check CI again. |
|
The R failure is currently tracked here: The other two seem to be just flaky transient errors. |
Merge upstream main to include the RTools 40 CI fix while preserving the ORC batch reader changes. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
I’ve merged main into this branch to pick up the Windows RTools 40 download fix in #55378. I also verified the corrected archive URL in a local Windows 11 ARM VM: a 1 KiB range request returned HTTP 206 with the expected |
|
These two seem to be just flaky transient errors, too. |
|
My approval still stands. |
raulcd
left a comment
There was a problem hiding this comment.
My concerns were solved with the latest changes. I am not an expert on this area but I don't have more concerns
Rationale for this change
ORCFile.read_stripe()materializes an entire stripe as one record batch. Dataset already scans ORC in batches; this adds native batching with explicit stripe selection toORCFile, supporting incremental consumption and early termination.Closes #54921.
What changes are included in this PR?
ORCFile.iter_batches(batch_size=65536, stripes=None, columns=None)with column projection.GetStripeReaderAPI with explicit optional column selection that decodes directly into batches without changing the reader's Seek position.Are these changes tested?
Validated locally on macOS arm64 with matching Release C++/Python builds, Python 3.13.7 and ORC 2.2.1.
44 passed; one existing timezone-database test restricted to ORC < 2.0 was skipped.
From
python/:140 passed with NumPy 2.5.4. In an isolated environment without NumPy, the
iter_batchestests andtest_without_numpy.pyhad 122 passes; only the marked NumPy integer-input test was skipped. Coverage includes five codecs, stripe order and duplicates, batch limits, nested/zero-column projections, invalid arguments, interleaved iterators and read failures. Both environments cover owner lifetime, early close and resource release for whole-file and selected-stripe iteration.The C++ memory regression uses four int64 columns and batches of 256 rows. For single stripes of 262144 and 1048576 rows, whole-stripe Arrow allocation peaks were 16.125 and 64.5 MiB; streaming peaked at 16.25 KiB for both sizes. NONE/ZLIB range tests verify that consuming the first batch and closing early does not read unselected stripe data.
Changed-file format/lint checks and
git diff --checkpassed.Are there any user-facing changes?
Yes. The new API yields
RecordBatchobjects.stripes=Nonereads all stripes; explicit sequences preserve order and duplicates, and empty sequences yield no batches.columns=[]preserves row counts. Closing the iterator releases its reader without closing the input; returned batches remain valid.batch_sizelimits output rows, not total process memory. ORC buffers and dictionaries require additional memory, and small batches can reduce throughput. Reading the same stripes does not inherently reduce I/O.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by:
The human contributor directed the design and reviewed the proposal and PR text. OpenAI Codex assisted with code, tests, documentation and static review.