Skip to content

[Data] Split ORC DataSourceV2 read tasks by stripe - #67105

Open
Asthenia0412 wants to merge 7 commits into
ray-project:masterfrom
Asthenia0412:feat/orc-stripe-splitting
Open

Asthenia0412 wants to merge 7 commits into
ray-project:masterfrom
Asthenia0412:feat/orc-stripe-splitting

Conversation

@Asthenia0412

Copy link
Copy Markdown

Summary

Implements stripe-level read-unit splitting for Ray's DataSourceV2 ORC path
(issue #66999). Previously every ORC file was one indivisible read unit — a
large multi-stripe file could be scanned by at most one Ray task. This change
lets several tasks scan disjoint stripes of the same file in parallel.

Changes

  1. New OrcFileIndexer (263 lines) — opens each file's ORC footer
    (~0.2 ms, no data IO) via pyarrow.orc.ORCFile and emits one FileManifest
    row per stripe. Single-stripe files fall back to a whole-file row so they
    pay no overhead.

  2. OrcDatasourceV2 — switches to the new indexer and returns
    OnlineBinPacker as its partitioner (matching Parquet's row-group path).
    The bin packer groups stripe-level units across files into
    read-task-sized buckets.

  3. OrcFileReader — gains a stripe-direct read path: when a manifest row
    names stripe indices in its chunk metadata the reader bypasses
    pyarrow.dataset and calls orc.ORCFile.read_stripe directly, replaying
    the same projection / filter / partition-synthesis logic the pyarrow path
    applies. Whole-file rows are untouched.

Stripe size estimation

PyArrow does not expose per-stripe encoded byte length or row counts from the
ORC footer, so we apportion file_size / nstripes and total_nrows / nstripes
evenly. This matches the estimator-driven approach of the whole-file size
estimator, and OnlineBinPacker already handles estimator inaccuracy. When
PyArrow exposes per-stripe stats we can refine this; the follow-up
stripe-level pruning change (#66995) will also want the same hook.

Testing (local smoke)

End-to-end through OrcDatasourceV2._get_file_indexer() →
OnlineBinPacker → OrcFileReader.read():

  • Multi-stripe ORC → chunked manifest with correct stripe indices
  • Single-stripe ORC → whole-file manifest fallback
  • Stripe-direct reader returns correct rows/columns, applies predicate pushdown
  • 3 files x 10 stripes each = 30,000 rows total
  • Mixed manifest (chunked + whole-file rows) routes correctly

CI coverage will come from the existing
test_orc_datasource_v2.py / test_orc_file_reader.py suites plus the
Parquet row-group splitting integration patterns they already exercise.

Motivation

When an ORC dataset contains only a few large multi-stripe files, whole-file
read units limit parallelism to the file count and can leave Ray workers
idle. Users want those files read by multiple tasks while retaining the same
results and read semantics.

Non-goals

AI assistance

This PR was implemented with AI assistance. The submitting human reviewed
every changed line and ran the local smoke tests above.

Commands run locally

ruff check python/ray/data/_internal/datasource_v2/formats/orc/
# → only SIM118 (.keys()) remaining, which matches upstream Parquet style

# End-to-end smoke test (standalone script)
# → OrcDatasourceV2 → OrcFileIndexer → OnlineBinPacker → OrcFileReader
# → 3 files x 10 stripes = 30,000 rows read correctly ✅

Duplicate-work check

Files changed (3)

python/ray/data/_internal/datasource_v2/formats/orc/orc_file_indexer.py  NEW (263 lines)
python/ray/data/_internal/datasource_v2/formats/orc/orc_datasource_v2.py MODIFIED (+57/-39)
python/ray/data/_internal/datasource_v2/formats/orc/orc_file_reader.py  MODIFIED (+320/-21)

+640 / -60 net insertions.

@Asthenia0412
Asthenia0412 requested a review from a team as a code owner October 16, 2026 05:36

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces stripe-level read-unit splitting for ORC files in Ray Data by implementing OrcFileIndexer and updating OrcFileReader. This allows multi-stripe ORC files to be bin-packed and read in parallel across multiple Ray tasks. The review feedback highlights three important issues: a premature file closure in _open_orc that can break metadata reads, a potential file descriptor leak if an exception occurs before the try...finally block in OrcFileReader, and a failure risk during schema evolution if read_stripe is called with columns missing from the file schema.

Comment on lines +62 to +65
if filesystem is None:
return orc.ORCFile(path)
with filesystem.open_input_file(path) as source:
return orc.ORCFile(source)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The file source is opened in a with block and returned as orc.ORCFile(source). When the function returns, the with block exits and closes source. PyArrow's ORCFile wrapper holds a reference to the underlying C++ stream; closing the Python file object prematurely can cause subsequent metadata reads (like of.nstripes or of.nrows in _chunk_orc_file) to fail, especially on remote filesystems like S3. Avoid closing the file handle inside _open_orc before the caller is done with the ORCFile object.

Suggested change
if filesystem is None:
return orc.ORCFile(path)
with filesystem.open_input_file(path) as source:
return orc.ORCFile(source)
if filesystem is None:
return orc.ORCFile(path)
source = filesystem.open_input_file(path)
return orc.ORCFile(source)

Comment on lines +254 to +262
physical_required = list(
dict.fromkeys(
name
for name in (output_columns + list(filter_columns))
if name not in partitions and name not in synthesized
)
)
if not physical_required:
physical_required = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If physical_required contains columns that do not exist in file_schema (e.g., due to schema evolution), of.read_stripe will raise an error. Unlike the PyArrow dataset scanner, read_stripe is not lenient and does not automatically null-fill missing columns. Filter physical_required to only include columns present in file_schema. Additionally, if physical_required becomes empty, ensure we pass [] instead of None if columns was specified, to avoid reading all columns.

Suggested change
physical_required = list(
dict.fromkeys(
name
for name in (output_columns + list(filter_columns))
if name not in partitions and name not in synthesized
)
)
if not physical_required:
physical_required = None
physical_required = list(
dict.fromkeys(
name
for name in (output_columns + list(filter_columns))
if name not in partitions and name not in synthesized
)
)
physical_required = [c for c in physical_required if file_schema.get_field_index(c) != -1]
if not physical_required:
physical_required = None if columns is None else []

Comment on lines +154 to +210
if filesystem is None or isinstance(filesystem, LocalFileSystem):
source = path
else:
source = filesystem.open_input_file(path)
of = orc.ORCFile(source)
file_schema = of.schema
# Close the file handle on remote filesystems when we're done with
# this file; LocalFileSystem strings need no close.
close_source = (
filesystem is not None
and not isinstance(filesystem, LocalFileSystem)
and hasattr(source, "close")
)
try:
for stripe_idx in stripe_ids:
for table in self._read_one_stripe(
of=of,
path=path,
stripe_idx=int(stripe_idx),
file_schema=file_schema,
columns=columns,
predicate=predicate,
partition_parser=partition_parser,
synthesized=synthesized,
schema=schema,
):
if table.num_rows == 0:
continue

if self._limit is not None:
if rows_read >= self._limit:
return
if len(table) > self._limit - rows_read:
table = table.slice(0, self._limit - rows_read)

# Project / reorder.
if columns is not None:
produced = set(table.column_names)
projected = [c for c in columns if c in produced]
table = table.select(projected)

if table.num_columns == 0 and table.num_rows > 0:
from ray.data._internal.arrow_block import (
_BATCH_SIZE_PRESERVING_STUB_COL_NAME,
)

table = table.append_column(
_BATCH_SIZE_PRESERVING_STUB_COL_NAME,
pa.nulls(table.num_rows),
)

self._on_batch_read(table)
rows_read += len(table)
yield table
finally:
if close_source:
source.close()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If orc.ORCFile(source) or of.schema raises an exception, the try...finally block is never entered, and source.close() is never called. This leads to a file descriptor leak on remote filesystems. Wrap the file opening and initialization in the try...finally block to ensure source is always closed if successfully opened.

            source = None
            close_source = False
            try:
                if filesystem is None or isinstance(filesystem, LocalFileSystem):
                    source = path
                else:
                    source = filesystem.open_input_file(path)
                    close_source = True
                of = orc.ORCFile(source)
                file_schema = of.schema
                for stripe_idx in stripe_ids:
                    for table in self._read_one_stripe(
                        of=of,
                        path=path,
                        stripe_idx=int(stripe_idx),
                        file_schema=file_schema,
                        columns=columns,
                        predicate=predicate,
                        partition_parser=partition_parser,
                        synthesized=synthesized,
                        schema=schema,
                    ):
                        if table.num_rows == 0:
                            continue

                        if self._limit is not None:
                            if rows_read >= self._limit:
                                return
                            if len(table) > self._limit - rows_read:
                                table = table.slice(0, self._limit - rows_read)

                        # Project / reorder.
                        if columns is not None:
                            produced = set(table.column_names)
                            projected = [c for c in columns if c in produced]
                            table = table.select(projected)

                        if table.num_columns == 0 and table.num_rows > 0:
                            from ray.data._internal.arrow_block import (
                                _BATCH_SIZE_PRESERVING_STUB_COL_NAME,
                            )

                            table = table.append_column(
                                _BATCH_SIZE_PRESERVING_STUB_COL_NAME,
                                pa.nulls(table.num_rows),
                            )

                        self._on_batch_read(table)
                        rows_read += len(table)
                        yield table
            finally:
                if close_source and source is not None:
                    source.close()

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Oct 16, 2026
@Asthenia0412
Asthenia0412 force-pushed the feat/orc-stripe-splitting branch from cd3f4d3 to 0e47e4f Compare October 16, 2026 10:18

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@Asthenia0412
Asthenia0412 force-pushed the feat/orc-stripe-splitting branch from 8001c76 to eb62dce Compare October 16, 2026 23:16
Asthenia0412 pushed a commit to Asthenia0412/ray that referenced this pull request Oct 17, 2026
Nine inline comments from gemini-code-assist and cursor bugbot flagged six
distinct issues in the ORC stripe-splitting changes. This commit addresses
all of them with minimal, targeted fixes:

orc_file_indexer.py:
- Close the remote filesystem handle from _open_orc after the ORC footer
  has been read. The original code opened the file inside a 'with' block
  and returned an ORCFile backed by the source -- the 'with' exited before
  the caller consumed nstripes/nrows, so the C++ stream could be closed
  prematurely (High severity; gemini, line 65).

orc_file_reader.py:
- Wrap ORCFile construction in a try...except that also closes the source
  handle on failure. Previously only the stripe-read loop was wrapped, so
  any exception from filesystem.open_input_file or orc.ORCFile(source)
  leaked the descriptor (Medium severity; gemini, line 210).
- Filter physical_required against file_schema.names before calling
  read_stripe. The stripe-direct reader now honors schema-evolved files
  by intersecting the unified output schema with each file's physical
  columns and null-filling anything missing afterward (High severity;
  cursor, line 302).
- Preserve an empty physical_required list instead of converting it to
  None. An empty list is the correct signal for "decode nothing"; None
  tells pyarrow to decode every column (Medium severity; cursor, line 262).
- Convert residual_predicate with to_pyarrow() before passing it to
  table.filter. The stripe-direct reader was forwarding a Ray Expr, which
  pyarrow.Table.filter cannot accept (High severity; cursor, line 337).
- Append synthesized columns (e.g. path from include_paths=True) after
  partition synthesis. The stripe-direct reader bypassed FileReader's
  outer iterator where whole-file rows get these columns attached, so
  multi-stripe files were silently dropping the path column (High severity;
  cursor, line 207).
- Apply the scanner push-down limit once, at OrcFileReader.read's outer
  boundary, across both the whole-file and stripe-direct paths. Mixed
  splits produced by OnlineBinPacker previously honored limit twice (one
  counter per path) and could emit up to 2x the requested rows (Medium
  severity; cursor, line 187).

Tests run (local): python3 -m compileall on the two modified files passes
cleanly; full Ray test suite requires the Cython extension and is not
available in this sandbox environment. The fixes were designed to preserve
behavior on the single-stripe/whole-file path unchanged.

Issue: ray-project#67105
Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

if filesystem is None:
return orc.ORCFile(path), None
source = filesystem.open_input_file(path)
return orc.ORCFile(source), source

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File handle leaked on footer open

Medium Severity

_open_orc opens the filesystem handle and then constructs ORCFile without protecting that window. If the constructor fails, the handle is never returned, so _chunk_orc_file's finally cannot close it. Footer-parse failures already fall back to a whole-file row, so a directory of unreadable ORC files can leak descriptors on the listing worker.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a741f30. Configure here.

@Asthenia0412
Asthenia0412 force-pushed the feat/orc-stripe-splitting branch from 7b25f6a to 57d80bc Compare October 17, 2026 13:16

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

if batch.num_rows == 0:
return

table = pa.Table.from_batches([batch])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stripe reads ignore batch size

Medium Severity

The stripe-direct reader yields each read_stripe result as one table and never slices it by self._batch_size. Multi-stripe files — the files this change is meant to split — therefore materialize an entire stripe in memory, unlike the pyarrow dataset path which batches at the configured size.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 57d80bc. Configure here.

@Asthenia0412
Asthenia0412 force-pushed the feat/orc-stripe-splitting branch from 57d80bc to a8296ab Compare October 17, 2026 15:06

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

)
finally:
if close_source:
source.close()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stripe path skips IO retry

Medium Severity

The new stripe-direct reader opens the file and calls read_stripe without iterate_with_retry. Whole-file rows still go through FileReader, which retries transient filesystem errors. Multi-stripe files on S3 (the files this PR splits) now fail the task on the first timeout instead of retrying.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a8296ab. Configure here.

@override
def as_whole_file_indexer(self) -> None:
# Footer reads are per-file IO; cannot pretend we don't do them.
return None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whole-file indexer override returns None

Low Severity

OrcFileIndexer.as_whole_file_indexer returns None, disabling the whole-file listing the parent already provides. NonSamplingFileIndexer (and Parquet's FooterFileIndexer) return a base indexer so metadata-only consumers can list each file once without footer IO. Callers invoke this hook; they do not construct NonSamplingFileIndexer themselves.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a8296ab. Configure here.

@Asthenia0412
Asthenia0412 force-pushed the feat/orc-stripe-splitting branch from a8296ab to 97873ee Compare October 18, 2026 04:20
Asthenia0412 pushed a commit to Asthenia0412/ray that referenced this pull request Oct 18, 2026
Nine inline comments from gemini-code-assist and cursor bugbot flagged six
distinct issues in the ORC stripe-splitting changes. This commit addresses
all of them with minimal, targeted fixes:

orc_file_indexer.py:
- Close the remote filesystem handle from _open_orc after the ORC footer
  has been read. The original code opened the file inside a 'with' block
  and returned an ORCFile backed by the source -- the 'with' exited before
  the caller consumed nstripes/nrows, so the C++ stream could be closed
  prematurely (High severity; gemini, line 65).

orc_file_reader.py:
- Wrap ORCFile construction in a try...except that also closes the source
  handle on failure. Previously only the stripe-read loop was wrapped, so
  any exception from filesystem.open_input_file or orc.ORCFile(source)
  leaked the descriptor (Medium severity; gemini, line 210).
- Filter physical_required against file_schema.names before calling
  read_stripe. The stripe-direct reader now honors schema-evolved files
  by intersecting the unified output schema with each file's physical
  columns and null-filling anything missing afterward (High severity;
  cursor, line 302).
- Preserve an empty physical_required list instead of converting it to
  None. An empty list is the correct signal for "decode nothing"; None
  tells pyarrow to decode every column (Medium severity; cursor, line 262).
- Convert residual_predicate with to_pyarrow() before passing it to
  table.filter. The stripe-direct reader was forwarding a Ray Expr, which
  pyarrow.Table.filter cannot accept (High severity; cursor, line 337).
- Append synthesized columns (e.g. path from include_paths=True) after
  partition synthesis. The stripe-direct reader bypassed FileReader's
  outer iterator where whole-file rows get these columns attached, so
  multi-stripe files were silently dropping the path column (High severity;
  cursor, line 207).
- Apply the scanner push-down limit once, at OrcFileReader.read's outer
  boundary, across both the whole-file and stripe-direct paths. Mixed
  splits produced by OnlineBinPacker previously honored limit twice (one
  counter per path) and could emit up to 2x the requested rows (Medium
  severity; cursor, line 187).

Tests run (local): python3 -m compileall on the two modified files passes
cleanly; full Ray test suite requires the Cython extension and is not
available in this sandbox environment. The fixes were designed to preserve
behavior on the single-stripe/whole-file path unchanged.

Issue: ray-project#67105
Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

if name not in partitions and name not in synthesized
)
)
# Intersect with the file's own physical columns so that schema-

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge case: reads that need no column from the file (count(), or selecting only a folder or path column) return 0 rows for multi-stripe files. read_stripe(i, columns=[]) returns a 0-row batch rather than "don't decode anything", so line 324 drops the stripe. For eg. this is the file the existing test_read_orc_multiple_stripes writes:

table = pa.table({"id": list(range(10000))})
orc.write_table(table, "multi.orc", stripe_size=64 * 1024)  # 10 stripes
print(ray.data.read_orc("multi.orc").count())
print(len(ray.data.read_orc("multi.orc").take_all()))
master:  10000, 10000
this PR: 0, 10000

@AarryaSaraf

Copy link
Copy Markdown
Contributor

Hi @Asthenia0412. Thank you for your contribution! Ill leave a few comments and we can get this once they are addressed.

if batch.num_rows == 0:
return

table = pa.Table.from_batches([batch])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge case: the stripe path keeps the file's own column types instead of the dataset schema, so when files store a column as different types, blocks don't match ds.schema(). The whole-file path casts through the pyarrow scanner. For eg. consider:

os.makedirs("data")
v = np.random.default_rng(0).integers(-2**31, 2**31 - 1, 40_000_000, dtype=np.int32)
orc.write_table(pa.table({"v": v}), "data/big.orc")  # ~153 MiB, 3 stripes, int32
orc.write_table(pa.table({"v": pa.array([1, 2, 3], pa.int64())}), "data/small.orc")

ds = ray.data.read_orc("data")
print(ds.schema().base_schema.field("v").type)
print({str(b.schema.field("v").type) for b in ds.iter_batches(batch_size=None, batch_format="pyarrow")})
master:  int64, {'int64'}
this PR: int64, {'int64', 'int32'}

return None
reader = OrcFileReader(format=FileFormat.ORC, filesystem=self._filesystem)
return RoundRobinPartitioner(SamplingInMemorySizeEstimator(reader), hints=hints)
def get_file_partitioner(self, **kwargs):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OnlineBinPacker expects each listing row's decoded size (interfaces/datasource_v2.py:179), but here a stripe's size is its even share of the file's bytes on disk, and read_stripe then decodes the whole stripe at once (the whole-file path streams 131k-row batches). For eg. consider:

ids = np.arange(40_000_000)
names = pa.array([f"category_value_{i:06d}" for i in range(1000)])
s = pa.DictionaryArray.from_arrays(pa.array(ids % 1000, pa.int32()), names).cast(pa.string())
orc.write_table(pa.table({"id": ids, "s": s}), "c.orc", compression="zstd", stripe_size=4 * 2**20)  # 19 MiB, 5 stripes

f = orc.ORCFile("c.orc")
print(f"{f.file_length / f.nstripes / 2**20:.1f} MiB")  # size this PR packs per stripe
print(f"{f.read_stripe(0).nbytes / 2**20:.0f} MiB")     # memory read_stripe holds
3.7 MiB
273 MiB

This will likely cause OOMs.

if close_source:
source.close()

def _read_one_stripe(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why a separate read path for stripes rather than FileReader's hooks (_get_fragments_to_read / _iter_fragment_tables)? _read_one_stripe copies what FileReader.read and _iter_fragment_tables already do, and the copy has drifted.

of, source_to_close = _open_orc(file_info.path, filesystem)
nstripes = int(of.nstripes)
nrows = int(of.nrows)
except (OSError, pa.ArrowError) as exc:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge case: one unreadable footer crashes listing with an AttributeError, because pa.ArrowError doesn't exist, so the whole-file fallback below never runs. For eg. a folder where one file was cut off mid-write:

os.makedirs("data")
for i in range(16):
    orc.write_table(pa.table({"id": range(i * 1000, (i + 1) * 1000)}), f"data/{i:02d}.orc")
good = open("data/00.orc", "rb").read()
open("data/z_bad.orc", "wb").write(good[: len(good) // 2])  # 17th file: schema inference reads only 16

ray.data.read_orc("data").count()
master:  ray::ReadFilesOrcV2() OSError: Could not open ORC input source 'data/z_bad.orc': Invalid ORC postscript length: 252, file length = 110
this PR: ray::ListFiles() AttributeError: module 'pyarrow' has no attribute 'ArrowError'

Importantly, the new message does not say which file is broken and it loses every file not just the bad ones.

filesystem=filesystem,
)

def _process_file_infos(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why collect every footer before yielding? _process_file_infos waits for the full folder walk (list(file_infos)) and every footer read (list(pool.map(...))) before the first row leaves, so no read task starts until the last footer is read. Master's indexer yields every 1,000 files, and Parquet's footer reader yields as footers arrive. Also, the files sampled for schema inference have their footers read on the driver and then again here.

# the file, seek to the footer, parse a small protobuf. 32 workers
# keeps the listing responsive on S3 directories with thousands of
# files without oversubscribing the driver.
max_workers = min(len(infos), 32)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All footer reads run inside one listing task on at most 32 threads (OnlineBinPacker needs global input, so listing is a single task), and on S3 each footer read is a network round trip. Parquet sends its footer reads to an actor pool spread across the cluster (8 actors x 128 concurrent reads, footer_file_indexer.py:45-46). Could ORC's footer reads be spread out the same way?

pruners: list[FilePruner] | None = None,
preserve_order: bool = False,
predicate: Expr | None = None,
limit: int | None = None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

limit is accepted here but never used, so read_orc(folder).limit(10) reads the footer of every file in the folder before the first read task starts (master reads none). The planner passes the pushed limit so metadata-aware indexers can stop early (plan_list_files_op.py:86-89), as Parquet's footer reader does.

max_shared_open_bins,
split_coalesced,
)
return OnlineBinPacker(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR adds no tests, and the existing suites the description relies on fail on this branch:

pytest python/ray/data/tests/unit/datasource_v2/test_orc_datasource_v2.py
FAILED test_resolve_partitioning_and_reuse_generic_file_components
assert False where False = isinstance(<...OnlineBinPacker...>, RoundRobinPartitioner)

test_orc.py::test_read_orc_multiple_stripes also asserts ds.count() == 10000, which returns 0 here (see the count() comment). Could you add tests through ray.data.read_orc for the stripe path? Each bug in the other comments would have been caught by one.

max_shared_open_bins = env_integer(
"RAY_DATA_ORC_BIN_PACKING_MAX_SHARED_OPEN_BINS", 16
)
split_coalesced = env_bool("RAY_DATA_ORC_FOOTER_SPLIT_COALESCED", False)

@AarryaSaraf AarryaSaraf Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does RAY_DATA_ORC_FOOTER_SPLIT_COALESCEDhave any effect?split_coalesced only lets the packer cut rows that hold several units (OnlineBinPacker._units), and OrcFileIndexer emits one stripe per row (unit_ids=(i,)`).


assert sample is not None
sample_paths = sample.paths.tolist()
# After stripe splitting, the same path can appear once per stripe

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the same path appear more than once here? The sample comes from list_file_infos (one row per file, listing_utils.py:179-181), not from the stripe indexer.

@jiangxt2 jiangxt2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concerns with the current design are:

  • The stripe path bypasses the shared FileReader pipeline and duplicates its output processing. Projections requiring no physical columns can lose rows, existing physical columns are not cast to the unified schema, and stripe reads bypass the existing reader-level I/O retry handling.
  • Each listing invocation collects its complete file listing and all footer results before emitting any manifests. This delays downstream reads for that listing, and the pushed-down limit does not reduce this metadata-discovery work.

An additional limitation is that per-stripe row counts are estimated by evenly dividing the file total rather than obtained from the actual layout. They therefore cannot serve as exact stripe cardinalities or be used to derive accurate physical row offsets, although the current reader uses stripe IDs rather than these estimates to locate data.

BTW, this overlaps with my draft PR #67278, which also addresses #66999.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

# before every footer has been parsed.
for manifest in pool.map(
lambda info: _chunk_orc_file(info, filesystem), infos
):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Limit still reads every footer

Medium Severity

pool.map submits a footer read for every file before any result is consumed, and exiting the ThreadPoolExecutor context waits for those jobs. An early return after limit therefore still opens every footer, so listing IO is not reduced on large directories.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b327ed4. Configure here.

Asthenia0412 and others added 7 commits November 3, 2026 03:39
When Ray reads a multi-stripe ORC file via DataSourceV2, each file was
treated as a single indivisible read unit -- a large multi-stripe ORC file
could therefore be read by at most one Ray task. This change introduces
stripe-level read units so that several tasks can scan disjoint stripes of
the same file in parallel.

Summary of changes:

- Add :class:`OrcFileIndexer` (new file, ~260 lines) which opens each ORC
  file's footer (no data IO, ~0.2 ms per file) and emits one manifest row
  per stripe. When a file has a single stripe it falls back to a whole-file
  row (chunk metadata = None) so single-stripe files pay no extra overhead.

- Switch :class:`OrcDatasourceV2` to use the new indexer and return
  :class:`OnlineBinPacker` as its partitioner (matching the Parquet
  row-group path). The bin packer groups stripe-level units across files
  into read-task-sized buckets.

- Extend :class:`OrcFileReader` with a stripe-direct read path: when a
  manifest row names stripe indices in its chunk metadata, the reader
  bypasses pyarrow.dataset and calls `orc.ORCFile.read_stripe` directly.
  The existing whole-file path is unchanged.

Estimated stripe sizes use the on-disk file size and `orc.ORCFile.nrows`
evenly across stripes; pyarrow does not expose per-stripe encoded byte
length or row counts from the footer. This is the same estimator-driven
approach used by the whole-file size estimator, and :class:`OnlineBinPacker`
already handles estimator inaccuracy.

Tests run (local, end-to-end smoke):

  - Multi-stripe ORC → chunked manifest: correct number of stripe rows,
    each carrying the right stripe index in chunk metadata.
  - Single-stripe ORC → whole-file manifest fallback, chunk metadata = None.
  - OrcFileReader stripe-direct path reads correct rows/columns, applies
    pushdown predicates and partition synthesis.
  - Full index → bin-pack → stripe-read pipeline on 3 files produces
    the correct total row count and per-file content.
  - Mixed manifest routing: reader correctly dispatches chunked and
    whole-file rows through their respective paths.

Issue: ray-project#66999
Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
ruff's isort hook (--select I) flagged import order violations in:
- orc_datasource_v2.py: typing_extensions must come before ray._common.utils
  (third-party before first-party per ray's ruff config)
- orc_file_indexer.py: stdlib groups must be alphabetically ordered
  (collections.abc before concurrent.futures; also TYPE_CHECKING block)

This fixes the pre-commit lint step that failed in microcheck CI.

Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
Nine inline comments from gemini-code-assist and cursor bugbot flagged six
distinct issues in the ORC stripe-splitting changes. This commit addresses
all of them with minimal, targeted fixes:

orc_file_indexer.py:
- Close the remote filesystem handle from _open_orc after the ORC footer
  has been read. The original code opened the file inside a 'with' block
  and returned an ORCFile backed by the source -- the 'with' exited before
  the caller consumed nstripes/nrows, so the C++ stream could be closed
  prematurely (High severity; gemini, line 65).

orc_file_reader.py:
- Wrap ORCFile construction in a try...except that also closes the source
  handle on failure. Previously only the stripe-read loop was wrapped, so
  any exception from filesystem.open_input_file or orc.ORCFile(source)
  leaked the descriptor (Medium severity; gemini, line 210).
- Filter physical_required against file_schema.names before calling
  read_stripe. The stripe-direct reader now honors schema-evolved files
  by intersecting the unified output schema with each file's physical
  columns and null-filling anything missing afterward (High severity;
  cursor, line 302).
- Preserve an empty physical_required list instead of converting it to
  None. An empty list is the correct signal for "decode nothing"; None
  tells pyarrow to decode every column (Medium severity; cursor, line 262).
- Convert residual_predicate with to_pyarrow() before passing it to
  table.filter. The stripe-direct reader was forwarding a Ray Expr, which
  pyarrow.Table.filter cannot accept (High severity; cursor, line 337).
- Append synthesized columns (e.g. path from include_paths=True) after
  partition synthesis. The stripe-direct reader bypassed FileReader's
  outer iterator where whole-file rows get these columns attached, so
  multi-stripe files were silently dropping the path column (High severity;
  cursor, line 207).
- Apply the scanner push-down limit once, at OrcFileReader.read's outer
  boundary, across both the whole-file and stripe-direct paths. Mixed
  splits produced by OnlineBinPacker previously honored limit twice (one
  counter per path) and could emit up to 2x the requested rows (Medium
  severity; cursor, line 187).

Tests run (local): python3 -m compileall on the two modified files passes
cleanly; full Ray test suite requires the Cython extension and is not
available in this sandbox environment. The fixes were designed to preserve
behavior on the single-stripe/whole-file path unchanged.

Issue: ray-project#67105
Signed-off-by: Asthenia0412 <dev@asthenia0412.github.io>
…eader

Ruff microcheck failed on the newly added ORC files due to:
- Missing blank line after module docstring (two files)
- Multi-line parenthesized expressions that ruff wanted collapsed

Pure formatting changes; no functional difference.

Signed-off-by: Asthenia0412 <asthenia0412@users.noreply.github.com>
The lint-pre-commit hook in buildkite/microcheck failed due to multi-line
parenthesized expressions in orc_datasource_v2.py that ruff format wanted
collapsed into single lines:
- env_integer/env_bool calls with single long argument
- resolve_partitioning method signature

Pure formatting changes; no functional difference.

Signed-off-by: Asthenia0412 <asthenia0412@users.noreply.github.com>
Fixes across orc_file_indexer.py, orc_file_reader.py, and orc_datasource_v2.py:

1. _open_orc fd leak (orc_file_indexer.py): wrap ORCFile constructor in try/except
   so a failed constructor closes the source handle it opened.

2. Schema-evolved column filter crash (orc_file_reader.py): when data_predicate
   references columns missing from a schema-evolved file, null-fill them
   before calling table.filter so PyArrow evaluates the filter against nulls
   (matching the whole-file pyarrow.dataset path).

3. Stripe-path schema reorder condition (orc_file_reader.py): mirrored the
   whole-file path's reorder logic -- reorder to schema names when no
   projection is pushed but a schema is available; select only projected
   columns when a projection is pushed.

4. infer_schema path deduplication (orc_datasource_v2.py): after stripe
   splitting the same path can appear once per stripe in the manifest;
   deduplicate before launching the schema-read thread pool.

Signed-off-by: Asthenia0412 <Asthenia0412@users.noreply.github.com>
Fixes from AarryaSaraf, jiangxt2 and bugbots:

orc_file_indexer.py:
- Replace non-existent pa.ArrowError with broad Exception catch so
  corrupt ORC files (mid-write footer) fall back to whole-file read
  instead of crashing listing with AttributeError
- Yield manifests incrementally as footers finish parsing, so the first
  read task starts before every footer is read
- Respect limit: stop listing once enough manifest rows are emitted
- Remove unused pyarrow import

orc_file_reader.py:
- Handle empty physical_required (count() / path-only projections):
  columns=[] makes read_stripe return 0 rows, so pass None instead
  and let projection drop extras
- Cast physical columns from file types to unified schema types so
  schema-evolved files all match ds.schema()

orc_datasource_v2.py:
- Remove ineffective RAY_DATA_ORC_FOOTER_SPLIT_COALESCED env var; each
  indexer row has exactly one stripe so coalesced splitting never runs
- Remove now-unused env_bool import

test_orc_datasource_v2.py:
- Update assertions for OrcFileIndexer / OnlineBinPacker to match new
  stripe-splitting implementation

Signed-off-by: trae-solo-agent [origin=scheduled_task] <solo-agent@trae.ai>
@Asthenia0412
Asthenia0412 force-pushed the feat/orc-stripe-splitting branch from b327ed4 to 0b6e54d Compare November 2, 2026 19:41

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 7 total unresolved issues (including 5 from previous reviews).

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 0b6e54d. Configure here.

if batch.num_rows == 0:
return

table = pa.Table.from_batches([batch])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stripe packing underestimates decoded memory

High Severity

OnlineBinPacker budgets on decoded size, but stripe size_bytes is an even share of on-disk file length, and _read_one_stripe then materializes each stripe in one read_stripe call. The whole-file path streams ~131k-row batches. Highly compressed or wide stripes packed into a 128 MiB bin can therefore decode far more than the cap in a single block and OOM the worker.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 0b6e54d. Configure here.

# the extras we don't need.
batch = of.read_stripe(stripe_idx, columns=None)
else:
batch = of.read_stripe(stripe_idx, columns=physical_required)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pickle check misses full-stripe decode

High Severity

⚠️ Externally read Arrow data may carry ray.data.arrow_pickled_object columns, which unpickle on access and execute arbitrary code. Call raise_on_pickle_object_columns(table) right after the read and add a reject test proving a planted pickle payload never runs. When physical_required is empty (count or partition-only projection), read_stripe(..., columns=None) decodes every column, then later projection drops them before _on_batch_read's check, and the earlier schema check used an empty schema.

Additional Locations (1)
Fix in Cursor Fix in Web

Triggered by project rule: Bugbot Rules

Reviewed by Cursor Bugbot for commit 0b6e54d. Configure here.

@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues stale The issue is stale. It will be closed within 7 days unless there are further conversation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants