Repository navigation
[Data] Split ORC DataSourceV2 read tasks by stripe - #67105
Asthenia0412 wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces stripe-level read-unit splitting for ORC files in Ray Data by implementing OrcFileIndexer and updating OrcFileReader. This allows multi-stripe ORC files to be bin-packed and read in parallel across multiple Ray tasks. The review feedback highlights three important issues: a premature file closure in _open_orc that can break metadata reads, a potential file descriptor leak if an exception occurs before the try...finally block in OrcFileReader, and a failure risk during schema evolution if read_stripe is called with columns missing from the file schema.
| if filesystem is None: | ||
| return orc.ORCFile(path) | ||
| with filesystem.open_input_file(path) as source: | ||
| return orc.ORCFile(source) |
There was a problem hiding this comment.
The file source is opened in a with block and returned as orc.ORCFile(source). When the function returns, the with block exits and closes source. PyArrow's ORCFile wrapper holds a reference to the underlying C++ stream; closing the Python file object prematurely can cause subsequent metadata reads (like of.nstripes or of.nrows in _chunk_orc_file) to fail, especially on remote filesystems like S3. Avoid closing the file handle inside _open_orc before the caller is done with the ORCFile object.
| if filesystem is None: | |
| return orc.ORCFile(path) | |
| with filesystem.open_input_file(path) as source: | |
| return orc.ORCFile(source) | |
| if filesystem is None: | |
| return orc.ORCFile(path) | |
| source = filesystem.open_input_file(path) | |
| return orc.ORCFile(source) |
| physical_required = list( | ||
| dict.fromkeys( | ||
| name | ||
| for name in (output_columns + list(filter_columns)) | ||
| if name not in partitions and name not in synthesized | ||
| ) | ||
| ) | ||
| if not physical_required: | ||
| physical_required = None |
There was a problem hiding this comment.
If physical_required contains columns that do not exist in file_schema (e.g., due to schema evolution), of.read_stripe will raise an error. Unlike the PyArrow dataset scanner, read_stripe is not lenient and does not automatically null-fill missing columns. Filter physical_required to only include columns present in file_schema. Additionally, if physical_required becomes empty, ensure we pass [] instead of None if columns was specified, to avoid reading all columns.
| physical_required = list( | |
| dict.fromkeys( | |
| name | |
| for name in (output_columns + list(filter_columns)) | |
| if name not in partitions and name not in synthesized | |
| ) | |
| ) | |
| if not physical_required: | |
| physical_required = None | |
| physical_required = list( | |
| dict.fromkeys( | |
| name | |
| for name in (output_columns + list(filter_columns)) | |
| if name not in partitions and name not in synthesized | |
| ) | |
| ) | |
| physical_required = [c for c in physical_required if file_schema.get_field_index(c) != -1] | |
| if not physical_required: | |
| physical_required = None if columns is None else [] |
| if filesystem is None or isinstance(filesystem, LocalFileSystem): | ||
| source = path | ||
| else: | ||
| source = filesystem.open_input_file(path) | ||
| of = orc.ORCFile(source) | ||
| file_schema = of.schema | ||
| # Close the file handle on remote filesystems when we're done with | ||
| # this file; LocalFileSystem strings need no close. | ||
| close_source = ( | ||
| filesystem is not None | ||
| and not isinstance(filesystem, LocalFileSystem) | ||
| and hasattr(source, "close") | ||
| ) | ||
| try: | ||
| for stripe_idx in stripe_ids: | ||
| for table in self._read_one_stripe( | ||
| of=of, | ||
| path=path, | ||
| stripe_idx=int(stripe_idx), | ||
| file_schema=file_schema, | ||
| columns=columns, | ||
| predicate=predicate, | ||
| partition_parser=partition_parser, | ||
| synthesized=synthesized, | ||
| schema=schema, | ||
| ): | ||
| if table.num_rows == 0: | ||
| continue | ||
|
|
||
| if self._limit is not None: | ||
| if rows_read >= self._limit: | ||
| return | ||
| if len(table) > self._limit - rows_read: | ||
| table = table.slice(0, self._limit - rows_read) | ||
|
|
||
| # Project / reorder. | ||
| if columns is not None: | ||
| produced = set(table.column_names) | ||
| projected = [c for c in columns if c in produced] | ||
| table = table.select(projected) | ||
|
|
||
| if table.num_columns == 0 and table.num_rows > 0: | ||
| from ray.data._internal.arrow_block import ( | ||
| _BATCH_SIZE_PRESERVING_STUB_COL_NAME, | ||
| ) | ||
|
|
||
| table = table.append_column( | ||
| _BATCH_SIZE_PRESERVING_STUB_COL_NAME, | ||
| pa.nulls(table.num_rows), | ||
| ) | ||
|
|
||
| self._on_batch_read(table) | ||
| rows_read += len(table) | ||
| yield table | ||
| finally: | ||
| if close_source: | ||
| source.close() |
There was a problem hiding this comment.
If orc.ORCFile(source) or of.schema raises an exception, the try...finally block is never entered, and source.close() is never called. This leads to a file descriptor leak on remote filesystems. Wrap the file opening and initialization in the try...finally block to ensure source is always closed if successfully opened.
source = None
close_source = False
try:
if filesystem is None or isinstance(filesystem, LocalFileSystem):
source = path
else:
source = filesystem.open_input_file(path)
close_source = True
of = orc.ORCFile(source)
file_schema = of.schema
for stripe_idx in stripe_ids:
for table in self._read_one_stripe(
of=of,
path=path,
stripe_idx=int(stripe_idx),
file_schema=file_schema,
columns=columns,
predicate=predicate,
partition_parser=partition_parser,
synthesized=synthesized,
schema=schema,
):
if table.num_rows == 0:
continue
if self._limit is not None:
if rows_read >= self._limit:
return
if len(table) > self._limit - rows_read:
table = table.slice(0, self._limit - rows_read)
# Project / reorder.
if columns is not None:
produced = set(table.column_names)
projected = [c for c in columns if c in produced]
table = table.select(projected)
if table.num_columns == 0 and table.num_rows > 0:
from ray.data._internal.arrow_block import (
_BATCH_SIZE_PRESERVING_STUB_COL_NAME,
)
table = table.append_column(
_BATCH_SIZE_PRESERVING_STUB_COL_NAME,
pa.nulls(table.num_rows),
)
self._on_batch_read(table)
rows_read += len(table)
yield table
finally:
if close_source and source is not None:
source.close()cd3f4d3 to
0e47e4f
Compare
8001c76 to
eb62dce
Compare
Nine inline comments from gemini-code-assist and cursor bugbot flagged six distinct issues in the ORC stripe-splitting changes. This commit addresses all of them with minimal, targeted fixes: orc_file_indexer.py: - Close the remote filesystem handle from _open_orc after the ORC footer has been read. The original code opened the file inside a 'with' block and returned an ORCFile backed by the source -- the 'with' exited before the caller consumed nstripes/nrows, so the C++ stream could be closed prematurely (High severity; gemini, line 65). orc_file_reader.py: - Wrap ORCFile construction in a try...except that also closes the source handle on failure. Previously only the stripe-read loop was wrapped, so any exception from filesystem.open_input_file or orc.ORCFile(source) leaked the descriptor (Medium severity; gemini, line 210). - Filter physical_required against file_schema.names before calling read_stripe. The stripe-direct reader now honors schema-evolved files by intersecting the unified output schema with each file's physical columns and null-filling anything missing afterward (High severity; cursor, line 302). - Preserve an empty physical_required list instead of converting it to None. An empty list is the correct signal for "decode nothing"; None tells pyarrow to decode every column (Medium severity; cursor, line 262). - Convert residual_predicate with to_pyarrow() before passing it to table.filter. The stripe-direct reader was forwarding a Ray Expr, which pyarrow.Table.filter cannot accept (High severity; cursor, line 337). - Append synthesized columns (e.g. path from include_paths=True) after partition synthesis. The stripe-direct reader bypassed FileReader's outer iterator where whole-file rows get these columns attached, so multi-stripe files were silently dropping the path column (High severity; cursor, line 207). - Apply the scanner push-down limit once, at OrcFileReader.read's outer boundary, across both the whole-file and stripe-direct paths. Mixed splits produced by OnlineBinPacker previously honored limit twice (one counter per path) and could emit up to 2x the requested rows (Medium severity; cursor, line 187). Tests run (local): python3 -m compileall on the two modified files passes cleanly; full Ray test suite requires the Cython extension and is not available in this sandbox environment. The fixes were designed to preserve behavior on the single-stripe/whole-file path unchanged. Issue: ray-project#67105 Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
| if filesystem is None: | ||
| return orc.ORCFile(path), None | ||
| source = filesystem.open_input_file(path) | ||
| return orc.ORCFile(source), source |
There was a problem hiding this comment.
File handle leaked on footer open
Medium Severity
_open_orc opens the filesystem handle and then constructs ORCFile without protecting that window. If the constructor fails, the handle is never returned, so _chunk_orc_file's finally cannot close it. Footer-parse failures already fall back to a whole-file row, so a directory of unreadable ORC files can leak descriptors on the listing worker.
Reviewed by Cursor Bugbot for commit a741f30. Configure here.
7b25f6a to
57d80bc
Compare
| if batch.num_rows == 0: | ||
| return | ||
|
|
||
| table = pa.Table.from_batches([batch]) |
There was a problem hiding this comment.
Stripe reads ignore batch size
Medium Severity
The stripe-direct reader yields each read_stripe result as one table and never slices it by self._batch_size. Multi-stripe files — the files this change is meant to split — therefore materialize an entire stripe in memory, unlike the pyarrow dataset path which batches at the configured size.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 57d80bc. Configure here.
57d80bc to
a8296ab
Compare
| ) | ||
| finally: | ||
| if close_source: | ||
| source.close() |
There was a problem hiding this comment.
Stripe path skips IO retry
Medium Severity
The new stripe-direct reader opens the file and calls read_stripe without iterate_with_retry. Whole-file rows still go through FileReader, which retries transient filesystem errors. Multi-stripe files on S3 (the files this PR splits) now fail the task on the first timeout instead of retrying.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a8296ab. Configure here.
| @override | ||
| def as_whole_file_indexer(self) -> None: | ||
| # Footer reads are per-file IO; cannot pretend we don't do them. | ||
| return None |
There was a problem hiding this comment.
Whole-file indexer override returns None
Low Severity
OrcFileIndexer.as_whole_file_indexer returns None, disabling the whole-file listing the parent already provides. NonSamplingFileIndexer (and Parquet's FooterFileIndexer) return a base indexer so metadata-only consumers can list each file once without footer IO. Callers invoke this hook; they do not construct NonSamplingFileIndexer themselves.
Reviewed by Cursor Bugbot for commit a8296ab. Configure here.
a8296ab to
97873ee
Compare
Nine inline comments from gemini-code-assist and cursor bugbot flagged six distinct issues in the ORC stripe-splitting changes. This commit addresses all of them with minimal, targeted fixes: orc_file_indexer.py: - Close the remote filesystem handle from _open_orc after the ORC footer has been read. The original code opened the file inside a 'with' block and returned an ORCFile backed by the source -- the 'with' exited before the caller consumed nstripes/nrows, so the C++ stream could be closed prematurely (High severity; gemini, line 65). orc_file_reader.py: - Wrap ORCFile construction in a try...except that also closes the source handle on failure. Previously only the stripe-read loop was wrapped, so any exception from filesystem.open_input_file or orc.ORCFile(source) leaked the descriptor (Medium severity; gemini, line 210). - Filter physical_required against file_schema.names before calling read_stripe. The stripe-direct reader now honors schema-evolved files by intersecting the unified output schema with each file's physical columns and null-filling anything missing afterward (High severity; cursor, line 302). - Preserve an empty physical_required list instead of converting it to None. An empty list is the correct signal for "decode nothing"; None tells pyarrow to decode every column (Medium severity; cursor, line 262). - Convert residual_predicate with to_pyarrow() before passing it to table.filter. The stripe-direct reader was forwarding a Ray Expr, which pyarrow.Table.filter cannot accept (High severity; cursor, line 337). - Append synthesized columns (e.g. path from include_paths=True) after partition synthesis. The stripe-direct reader bypassed FileReader's outer iterator where whole-file rows get these columns attached, so multi-stripe files were silently dropping the path column (High severity; cursor, line 207). - Apply the scanner push-down limit once, at OrcFileReader.read's outer boundary, across both the whole-file and stripe-direct paths. Mixed splits produced by OnlineBinPacker previously honored limit twice (one counter per path) and could emit up to 2x the requested rows (Medium severity; cursor, line 187). Tests run (local): python3 -m compileall on the two modified files passes cleanly; full Ray test suite requires the Cython extension and is not available in this sandbox environment. The fixes were designed to preserve behavior on the single-stripe/whole-file path unchanged. Issue: ray-project#67105 Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
| if name not in partitions and name not in synthesized | ||
| ) | ||
| ) | ||
| # Intersect with the file's own physical columns so that schema- |
There was a problem hiding this comment.
Edge case: reads that need no column from the file (count(), or selecting only a folder or path column) return 0 rows for multi-stripe files. read_stripe(i, columns=[]) returns a 0-row batch rather than "don't decode anything", so line 324 drops the stripe. For eg. this is the file the existing test_read_orc_multiple_stripes writes:
table = pa.table({"id": list(range(10000))})
orc.write_table(table, "multi.orc", stripe_size=64 * 1024) # 10 stripes
print(ray.data.read_orc("multi.orc").count())
print(len(ray.data.read_orc("multi.orc").take_all()))master: 10000, 10000
this PR: 0, 10000
|
Hi @Asthenia0412. Thank you for your contribution! Ill leave a few comments and we can get this once they are addressed. |
| if batch.num_rows == 0: | ||
| return | ||
|
|
||
| table = pa.Table.from_batches([batch]) |
There was a problem hiding this comment.
Edge case: the stripe path keeps the file's own column types instead of the dataset schema, so when files store a column as different types, blocks don't match ds.schema(). The whole-file path casts through the pyarrow scanner. For eg. consider:
os.makedirs("data")
v = np.random.default_rng(0).integers(-2**31, 2**31 - 1, 40_000_000, dtype=np.int32)
orc.write_table(pa.table({"v": v}), "data/big.orc") # ~153 MiB, 3 stripes, int32
orc.write_table(pa.table({"v": pa.array([1, 2, 3], pa.int64())}), "data/small.orc")
ds = ray.data.read_orc("data")
print(ds.schema().base_schema.field("v").type)
print({str(b.schema.field("v").type) for b in ds.iter_batches(batch_size=None, batch_format="pyarrow")})master: int64, {'int64'}
this PR: int64, {'int64', 'int32'}
| return None | ||
| reader = OrcFileReader(format=FileFormat.ORC, filesystem=self._filesystem) | ||
| return RoundRobinPartitioner(SamplingInMemorySizeEstimator(reader), hints=hints) | ||
| def get_file_partitioner(self, **kwargs): |
There was a problem hiding this comment.
OnlineBinPacker expects each listing row's decoded size (interfaces/datasource_v2.py:179), but here a stripe's size is its even share of the file's bytes on disk, and read_stripe then decodes the whole stripe at once (the whole-file path streams 131k-row batches). For eg. consider:
ids = np.arange(40_000_000)
names = pa.array([f"category_value_{i:06d}" for i in range(1000)])
s = pa.DictionaryArray.from_arrays(pa.array(ids % 1000, pa.int32()), names).cast(pa.string())
orc.write_table(pa.table({"id": ids, "s": s}), "c.orc", compression="zstd", stripe_size=4 * 2**20) # 19 MiB, 5 stripes
f = orc.ORCFile("c.orc")
print(f"{f.file_length / f.nstripes / 2**20:.1f} MiB") # size this PR packs per stripe
print(f"{f.read_stripe(0).nbytes / 2**20:.0f} MiB") # memory read_stripe holds3.7 MiB
273 MiB
This will likely cause OOMs.
| if close_source: | ||
| source.close() | ||
|
|
||
| def _read_one_stripe( |
There was a problem hiding this comment.
Why a separate read path for stripes rather than FileReader's hooks (_get_fragments_to_read / _iter_fragment_tables)? _read_one_stripe copies what FileReader.read and _iter_fragment_tables already do, and the copy has drifted.
| of, source_to_close = _open_orc(file_info.path, filesystem) | ||
| nstripes = int(of.nstripes) | ||
| nrows = int(of.nrows) | ||
| except (OSError, pa.ArrowError) as exc: |
There was a problem hiding this comment.
Edge case: one unreadable footer crashes listing with an AttributeError, because pa.ArrowError doesn't exist, so the whole-file fallback below never runs. For eg. a folder where one file was cut off mid-write:
os.makedirs("data")
for i in range(16):
orc.write_table(pa.table({"id": range(i * 1000, (i + 1) * 1000)}), f"data/{i:02d}.orc")
good = open("data/00.orc", "rb").read()
open("data/z_bad.orc", "wb").write(good[: len(good) // 2]) # 17th file: schema inference reads only 16
ray.data.read_orc("data").count()master: ray::ReadFilesOrcV2() OSError: Could not open ORC input source 'data/z_bad.orc': Invalid ORC postscript length: 252, file length = 110
this PR: ray::ListFiles() AttributeError: module 'pyarrow' has no attribute 'ArrowError'
Importantly, the new message does not say which file is broken and it loses every file not just the bad ones.
| filesystem=filesystem, | ||
| ) | ||
|
|
||
| def _process_file_infos( |
There was a problem hiding this comment.
Why collect every footer before yielding? _process_file_infos waits for the full folder walk (list(file_infos)) and every footer read (list(pool.map(...))) before the first row leaves, so no read task starts until the last footer is read. Master's indexer yields every 1,000 files, and Parquet's footer reader yields as footers arrive. Also, the files sampled for schema inference have their footers read on the driver and then again here.
| # the file, seek to the footer, parse a small protobuf. 32 workers | ||
| # keeps the listing responsive on S3 directories with thousands of | ||
| # files without oversubscribing the driver. | ||
| max_workers = min(len(infos), 32) |
There was a problem hiding this comment.
All footer reads run inside one listing task on at most 32 threads (OnlineBinPacker needs global input, so listing is a single task), and on S3 each footer read is a network round trip. Parquet sends its footer reads to an actor pool spread across the cluster (8 actors x 128 concurrent reads, footer_file_indexer.py:45-46). Could ORC's footer reads be spread out the same way?
| pruners: list[FilePruner] | None = None, | ||
| preserve_order: bool = False, | ||
| predicate: Expr | None = None, | ||
| limit: int | None = None, |
There was a problem hiding this comment.
limit is accepted here but never used, so read_orc(folder).limit(10) reads the footer of every file in the folder before the first read task starts (master reads none). The planner passes the pushed limit so metadata-aware indexers can stop early (plan_list_files_op.py:86-89), as Parquet's footer reader does.
| max_shared_open_bins, | ||
| split_coalesced, | ||
| ) | ||
| return OnlineBinPacker( |
There was a problem hiding this comment.
The PR adds no tests, and the existing suites the description relies on fail on this branch:
pytest python/ray/data/tests/unit/datasource_v2/test_orc_datasource_v2.py
FAILED test_resolve_partitioning_and_reuse_generic_file_components
assert False where False = isinstance(<...OnlineBinPacker...>, RoundRobinPartitioner)
test_orc.py::test_read_orc_multiple_stripes also asserts ds.count() == 10000, which returns 0 here (see the count() comment). Could you add tests through ray.data.read_orc for the stripe path? Each bug in the other comments would have been caught by one.
| max_shared_open_bins = env_integer( | ||
| "RAY_DATA_ORC_BIN_PACKING_MAX_SHARED_OPEN_BINS", 16 | ||
| ) | ||
| split_coalesced = env_bool("RAY_DATA_ORC_FOOTER_SPLIT_COALESCED", False) |
There was a problem hiding this comment.
Does RAY_DATA_ORC_FOOTER_SPLIT_COALESCEDhave any effect?split_coalesced only lets the packer cut rows that hold several units (OnlineBinPacker._units), and OrcFileIndexer emits one stripe per row (unit_ids=(i,)`).
|
|
||
| assert sample is not None | ||
| sample_paths = sample.paths.tolist() | ||
| # After stripe splitting, the same path can appear once per stripe |
There was a problem hiding this comment.
Can the same path appear more than once here? The sample comes from list_file_infos (one row per file, listing_utils.py:179-181), not from the stripe indexer.
jiangxt2
left a comment
There was a problem hiding this comment.
My main concerns with the current design are:
- The stripe path bypasses the shared
FileReaderpipeline and duplicates its output processing. Projections requiring no physical columns can lose rows, existing physical columns are not cast to the unified schema, and stripe reads bypass the existing reader-level I/O retry handling. - Each listing invocation collects its complete file listing and all footer results before emitting any manifests. This delays downstream reads for that listing, and the pushed-down limit does not reduce this metadata-discovery work.
An additional limitation is that per-stripe row counts are estimated by evenly dividing the file total rather than obtained from the actual layout. They therefore cannot serve as exact stripe cardinalities or be used to derive accurate physical row offsets, although the current reader uses stripe IDs rather than these estimates to locate data.
BTW, this overlaps with my draft PR #67278, which also addresses #66999.
| # before every footer has been parsed. | ||
| for manifest in pool.map( | ||
| lambda info: _chunk_orc_file(info, filesystem), infos | ||
| ): |
There was a problem hiding this comment.
Limit still reads every footer
Medium Severity
pool.map submits a footer read for every file before any result is consumed, and exiting the ThreadPoolExecutor context waits for those jobs. An early return after limit therefore still opens every footer, so listing IO is not reduced on large directories.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit b327ed4. Configure here.
When Ray reads a multi-stripe ORC file via DataSourceV2, each file was
treated as a single indivisible read unit -- a large multi-stripe ORC file
could therefore be read by at most one Ray task. This change introduces
stripe-level read units so that several tasks can scan disjoint stripes of
the same file in parallel.
Summary of changes:
- Add :class:`OrcFileIndexer` (new file, ~260 lines) which opens each ORC
file's footer (no data IO, ~0.2 ms per file) and emits one manifest row
per stripe. When a file has a single stripe it falls back to a whole-file
row (chunk metadata = None) so single-stripe files pay no extra overhead.
- Switch :class:`OrcDatasourceV2` to use the new indexer and return
:class:`OnlineBinPacker` as its partitioner (matching the Parquet
row-group path). The bin packer groups stripe-level units across files
into read-task-sized buckets.
- Extend :class:`OrcFileReader` with a stripe-direct read path: when a
manifest row names stripe indices in its chunk metadata, the reader
bypasses pyarrow.dataset and calls `orc.ORCFile.read_stripe` directly.
The existing whole-file path is unchanged.
Estimated stripe sizes use the on-disk file size and `orc.ORCFile.nrows`
evenly across stripes; pyarrow does not expose per-stripe encoded byte
length or row counts from the footer. This is the same estimator-driven
approach used by the whole-file size estimator, and :class:`OnlineBinPacker`
already handles estimator inaccuracy.
Tests run (local, end-to-end smoke):
- Multi-stripe ORC → chunked manifest: correct number of stripe rows,
each carrying the right stripe index in chunk metadata.
- Single-stripe ORC → whole-file manifest fallback, chunk metadata = None.
- OrcFileReader stripe-direct path reads correct rows/columns, applies
pushdown predicates and partition synthesis.
- Full index → bin-pack → stripe-read pipeline on 3 files produces
the correct total row count and per-file content.
- Mixed manifest routing: reader correctly dispatches chunked and
whole-file rows through their respective paths.
Issue: ray-project#66999
Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
ruff's isort hook (--select I) flagged import order violations in: - orc_datasource_v2.py: typing_extensions must come before ray._common.utils (third-party before first-party per ray's ruff config) - orc_file_indexer.py: stdlib groups must be alphabetically ordered (collections.abc before concurrent.futures; also TYPE_CHECKING block) This fixes the pre-commit lint step that failed in microcheck CI. Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
Nine inline comments from gemini-code-assist and cursor bugbot flagged six distinct issues in the ORC stripe-splitting changes. This commit addresses all of them with minimal, targeted fixes: orc_file_indexer.py: - Close the remote filesystem handle from _open_orc after the ORC footer has been read. The original code opened the file inside a 'with' block and returned an ORCFile backed by the source -- the 'with' exited before the caller consumed nstripes/nrows, so the C++ stream could be closed prematurely (High severity; gemini, line 65). orc_file_reader.py: - Wrap ORCFile construction in a try...except that also closes the source handle on failure. Previously only the stripe-read loop was wrapped, so any exception from filesystem.open_input_file or orc.ORCFile(source) leaked the descriptor (Medium severity; gemini, line 210). - Filter physical_required against file_schema.names before calling read_stripe. The stripe-direct reader now honors schema-evolved files by intersecting the unified output schema with each file's physical columns and null-filling anything missing afterward (High severity; cursor, line 302). - Preserve an empty physical_required list instead of converting it to None. An empty list is the correct signal for "decode nothing"; None tells pyarrow to decode every column (Medium severity; cursor, line 262). - Convert residual_predicate with to_pyarrow() before passing it to table.filter. The stripe-direct reader was forwarding a Ray Expr, which pyarrow.Table.filter cannot accept (High severity; cursor, line 337). - Append synthesized columns (e.g. path from include_paths=True) after partition synthesis. The stripe-direct reader bypassed FileReader's outer iterator where whole-file rows get these columns attached, so multi-stripe files were silently dropping the path column (High severity; cursor, line 207). - Apply the scanner push-down limit once, at OrcFileReader.read's outer boundary, across both the whole-file and stripe-direct paths. Mixed splits produced by OnlineBinPacker previously honored limit twice (one counter per path) and could emit up to 2x the requested rows (Medium severity; cursor, line 187). Tests run (local): python3 -m compileall on the two modified files passes cleanly; full Ray test suite requires the Cython extension and is not available in this sandbox environment. The fixes were designed to preserve behavior on the single-stripe/whole-file path unchanged. Issue: ray-project#67105 Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
…eader Ruff microcheck failed on the newly added ORC files due to: - Missing blank line after module docstring (two files) - Multi-line parenthesized expressions that ruff wanted collapsed Pure formatting changes; no functional difference. Signed-off-by: Asthenia0412 <asthenia0412@users.noreply.github.com>
The lint-pre-commit hook in buildkite/microcheck failed due to multi-line parenthesized expressions in orc_datasource_v2.py that ruff format wanted collapsed into single lines: - env_integer/env_bool calls with single long argument - resolve_partitioning method signature Pure formatting changes; no functional difference. Signed-off-by: Asthenia0412 <asthenia0412@users.noreply.github.com>
Fixes across orc_file_indexer.py, orc_file_reader.py, and orc_datasource_v2.py: 1. _open_orc fd leak (orc_file_indexer.py): wrap ORCFile constructor in try/except so a failed constructor closes the source handle it opened. 2. Schema-evolved column filter crash (orc_file_reader.py): when data_predicate references columns missing from a schema-evolved file, null-fill them before calling table.filter so PyArrow evaluates the filter against nulls (matching the whole-file pyarrow.dataset path). 3. Stripe-path schema reorder condition (orc_file_reader.py): mirrored the whole-file path's reorder logic -- reorder to schema names when no projection is pushed but a schema is available; select only projected columns when a projection is pushed. 4. infer_schema path deduplication (orc_datasource_v2.py): after stripe splitting the same path can appear once per stripe in the manifest; deduplicate before launching the schema-read thread pool. Signed-off-by: Asthenia0412 <Asthenia0412@users.noreply.github.com>
Fixes from AarryaSaraf, jiangxt2 and bugbots: orc_file_indexer.py: - Replace non-existent pa.ArrowError with broad Exception catch so corrupt ORC files (mid-write footer) fall back to whole-file read instead of crashing listing with AttributeError - Yield manifests incrementally as footers finish parsing, so the first read task starts before every footer is read - Respect limit: stop listing once enough manifest rows are emitted - Remove unused pyarrow import orc_file_reader.py: - Handle empty physical_required (count() / path-only projections): columns=[] makes read_stripe return 0 rows, so pass None instead and let projection drop extras - Cast physical columns from file types to unified schema types so schema-evolved files all match ds.schema() orc_datasource_v2.py: - Remove ineffective RAY_DATA_ORC_FOOTER_SPLIT_COALESCED env var; each indexer row has exactly one stripe so coalesced splitting never runs - Remove now-unused env_bool import test_orc_datasource_v2.py: - Update assertions for OrcFileIndexer / OnlineBinPacker to match new stripe-splitting implementation Signed-off-by: trae-solo-agent [origin=scheduled_task] <solo-agent@trae.ai>
b327ed4 to
0b6e54d
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 7 total unresolved issues (including 5 from previous reviews).
Reviewed by Cursor Bugbot for commit 0b6e54d. Configure here.
| if batch.num_rows == 0: | ||
| return | ||
|
|
||
| table = pa.Table.from_batches([batch]) |
There was a problem hiding this comment.
Stripe packing underestimates decoded memory
High Severity
OnlineBinPacker budgets on decoded size, but stripe size_bytes is an even share of on-disk file length, and _read_one_stripe then materializes each stripe in one read_stripe call. The whole-file path streams ~131k-row batches. Highly compressed or wide stripes packed into a 128 MiB bin can therefore decode far more than the cap in a single block and OOM the worker.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 0b6e54d. Configure here.
| # the extras we don't need. | ||
| batch = of.read_stripe(stripe_idx, columns=None) | ||
| else: | ||
| batch = of.read_stripe(stripe_idx, columns=physical_required) |
There was a problem hiding this comment.
Pickle check misses full-stripe decode
High Severity
ray.data.arrow_pickled_object columns, which unpickle on access and execute arbitrary code. Call raise_on_pickle_object_columns(table) right after the read and add a reject test proving a planted pickle payload never runs. When physical_required is empty (count or partition-only projection), read_stripe(..., columns=None) decodes every column, then later projection drops them before _on_batch_read's check, and the earlier schema check used an empty schema.
Additional Locations (1)
Triggered by project rule: Bugbot Rules
Reviewed by Cursor Bugbot for commit 0b6e54d. Configure here.
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |


Summary
Implements stripe-level read-unit splitting for Ray's DataSourceV2 ORC path
(issue #66999). Previously every ORC file was one indivisible read unit — a
large multi-stripe file could be scanned by at most one Ray task. This change
lets several tasks scan disjoint stripes of the same file in parallel.
Changes
New
OrcFileIndexer(263 lines) — opens each file's ORC footer(~0.2 ms, no data IO) via
pyarrow.orc.ORCFileand emits oneFileManifestrow per stripe. Single-stripe files fall back to a whole-file row so they
pay no overhead.
OrcDatasourceV2— switches to the new indexer and returnsOnlineBinPackeras its partitioner (matching Parquet's row-group path).The bin packer groups stripe-level units across files into
read-task-sized buckets.
OrcFileReader— gains a stripe-direct read path: when a manifest rownames stripe indices in its chunk metadata the reader bypasses
pyarrow.datasetand callsorc.ORCFile.read_stripedirectly, replayingthe same projection / filter / partition-synthesis logic the pyarrow path
applies. Whole-file rows are untouched.
Stripe size estimation
PyArrow does not expose per-stripe encoded byte length or row counts from the
ORC footer, so we apportion
file_size / nstripesandtotal_nrows / nstripesevenly. This matches the estimator-driven approach of the whole-file size
estimator, and
OnlineBinPackeralready handles estimator inaccuracy. WhenPyArrow exposes per-stripe stats we can refine this; the follow-up
stripe-level pruning change (#66995) will also want the same hook.
Testing (local smoke)
End-to-end through
OrcDatasourceV2._get_file_indexer()→OnlineBinPacker→OrcFileReader.read():CI coverage will come from the existing
test_orc_datasource_v2.py/test_orc_file_reader.pysuites plus theParquet row-group splitting integration patterns they already exercise.
Motivation
When an ORC dataset contains only a few large multi-stripe files, whole-file
read units limit parallelism to the file count and can leave Ray workers
idle. Users want those files read by multiple tasks while retaining the same
results and read semantics.
Non-goals
AI assistance
This PR was implemented with AI assistance. The submitting human reviewed
every changed line and ran the local smoke tests above.
Commands run locally
Duplicate-work check
66999 in:body: noneorc stripe,orc datasource_v2: [Data] Add ORC support to DataSourceV2 #66524 (merged, adds ORCto DataSourceV2 without stripe splitting; our change extends it)
builds on the read-unit foundation we add here.
Files changed (3)
+640 / -60 net insertions.