Skip to content

[Data] read_iceberg on Datasource V2 — end-to-end preview (draft, do not merge) - #66182

Closed
AarryaSaraf wants to merge 5 commits into
ray-project:masterfrom
AarryaSaraf:iceberg-dsv2-preview
Closed

AarryaSaraf wants to merge 5 commits into
ray-project:masterfrom
AarryaSaraf:iceberg-dsv2-preview

Conversation

@AarryaSaraf

@AarryaSaraf AarryaSaraf commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Description

Draft for visibility — do not merge. This is the whole read_iceberg →
Datasource V2 port in one diff so the design can be seen end to end. It lands as
a stack of small reviewable PRs instead; this branch exists so reviewers do not
have to reconstruct the shape from five partial diffs, and it may lag them.

Datasource V2 is itself under active change, so the seams this uses are moving
and the branch is already behind master. It is here for visibility, not review —
the reviewable PRs will be cut against whatever V2 looks like when they land.

Motivation and the measured numbers are in #66181 — please leave design comments
there.

Initially behind an opt-in flag: DataContext.use_iceberg_datasource_v2
(default False, env RAY_DATA_USE_ICEBERG_DATASOURCE_V2). Nothing changes on
the default path and the public read_iceberg signature does not change.

DRIVER Plan Time Task Flow-2026-09-15-181148

The driver keeps only plan-time work: schema and snapshot come from table
metadata, so no file is sampled. plan_files() moves into the listing task with
the pushed-down filter attached, and its scan tasks are encoded as typed manifest
columns — the only thing that crosses from listing to reading. Read units are
bin packed on estimated decoded bytes rather than compressed bytes, and the
reader streams one block per record batch instead of materializing a chunk.

Two shortcuts fall out of the manifest. Where Iceberg's per-file residual is
AlwaysTrue, the recorded record count is the post-filter count, so count()
needs no data — which also required a small shared-code fix, since the count
rule rewrites the plan in a way that was discarding the listing filter and
counting the whole table. Where the snapshot summary is trustworthy, count()
returns on the driver with no tasks at all; both paths fail closed to the
ordinary read.

The existing Iceberg suite runs parametrized over both readers, so every existing
assertion is a parity assertion. New tests cover what has no V1 counterpart:
the manifest round trip including value types, delete files carried and
applied, awkward partition types and an evolved spec, bin-packing granularity,
listing-time file pruning, and the count paths.

Not implemented yet, and what we intend after this

  • Limit early-stop during listing — stop planning once the limit is met.
  • Native fast path for clean files. A file whose residual is AlwaysTrue,
    with no delete files and a schema matching the table's, is just a Parquet
    file — route it through Ray's own Parquet reader and send only the dirty
    minority through PyIceberg. Needs a schema-parity suite between the two
    readers first.
  • Split one file across tasks. One large file is one task today. Iceberg's
    split_offsets gives the split points with no footer reads, but PyIceberg's
    task type has no offset field, so this is gated on the native path above.
  • Delete-file affinity in bin packing — files in a partition share delete
    files and each is read once per task that needs it (the cost in [Data] read_iceberg re-reads a shared delete file once per data file instead of once per read task #66147).

Out of scope, each blocked outside this reader: incremental scans (#64464),
equality deletes (PyIceberg raises at plan time), and plan-time physical
properties (uniqueness / clustering / sortedness / NDV).

Known gaps

  • Merge-on-read is half-covered: a hand-built position-delete file round-trips
    correctly, but nothing yet proves PyIceberg attaches catalog-planned delete
    manifests to the right tasks.
  • Only a local SqlCatalog on file:// so far, so no catalog credential has
    had to survive pickling into a listing task on another machine.
  • The decoded-size estimator is a placeholder ratio.

Related issues

Related to #66181 and #66147. Not a duplicate: no open issue or PR covers moving
read_iceberg to Datasource V2.

Additional information

Tests run locally — pytest on tests/datasource/test_iceberg.py,
tests/unit/datasource_v2/,
_internal/logical/operators/tests/test_read_files_logical.py and
tests/datasource/test_read_parquet_v2.py: 284 passed. Lint clean under Ray's
pinned ruff 0.8.4 / black 22.10.0; pyrefly clean on the changed files.

AI assistance (Claude Code) was used. I have reviewed every changed line, can
defend the design end to end, and ran the tests above locally.

Signed-off-by: Aarrya aarrya.saraf@anyscale.com

🤖 Generated with Claude Code

AarryaSaraf and others added 5 commits September 9, 2026 17:34
First of three PRs that port read_iceberg to Datasource V2. This one is
the data contract the other two are written against, and nothing calls it
yet: the encoder in [2b/6], the read path in [2c/6]. It is split out
because it is the piece with the most detail per line and the least
context needed to review it -- pure encode/decode over table metadata,
no Ray operators involved.

V2 splits a read into a ListFiles task that decides what to read and
ReadFiles tasks that read it. For Iceberg that boundary is awkward:
PyIceberg's unit of work is a FileScanTask, which holds a DataFile, its
delete files and a residual expression, and neither a Table nor a FileIO
can cross a task boundary (issue ray-project#49107 is the V1 version of the same
problem). So the listing task encodes each FileScanTask into manifest
columns and the read task decodes it back.

Ten columns, all prefixed __ib_:

- path and file size, which the base FileManifest already defines;
- __ib_file_format and __ib_record_count, off the DataFile;
- __ib_spec_id and __ib_partition, which together let the reader rebuild
  the partition tuple. The partition rides as a typed Arrow struct built
  from the union of every partition spec the table has ever had, so a
  table that evolved its spec still round-trips, and each field carries
  Iceberg's own internal representation (a date is days-since-epoch, a
  UUID is 16 bytes) rather than a string;
- __ib_delete_paths and __ib_delete_formats, one list per data file;
- __ib_residual_is_true, the one derived column: whether plan_files()
  found the file's partition values already satisfy the whole filter.
  Nothing consumes it in this stack until the metadata-only count() in
  [3/6], where it is exactly the condition that makes a row count exact
  under a filter.

FileManifest allows extra columns (file_manifest.py), which is what makes
this work without a new plumbing type: a manifest block stays a block,
and the partitioner in [2b/6] can slice and concatenate it like any
other.

Tests: six, all round trips against the SQLite-catalog fixture -- every
field the reader will use survives, delete files survive, rebuilt scan
tasks decode to the same rows, and the partition struct is typed and
lossless across an identity partition, an evolved spec and the
unpartitioned case.

Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
…berg

Second of three. The four components that do the work, built against the
manifest from [2a/6]; the datasource that wires them into read_iceberg,
and the flag that turns any of it on, arrive in [2c/6]. Nothing in this
PR changes behaviour -- no existing code path reaches these classes.

Kept together because they are one round trip and each is small: the
indexer produces manifest rows, the partitioner groups them, the scanner
is what a task carries, and the reader turns a group back into blocks.
The scanner also cannot be separated from the indexer -- the indexer
calls combine_row_filter to translate a pushed Ray predicate, and there
is only one such translator.

listing/iceberg_file_indexer.py -- plan_files() inside the listing task,
one manifest row per data file, emitted in blocks of
RAY_DATA_ICEBERG_LISTING_BATCH_FILES files (default 1024) purely to
amortize Arrow array construction. Listing does no data IO at all: an
Iceberg manifest already records each file's path, size, row count and
partition values, so unlike Parquet there is no footer to fetch. It
ignores shuffle_config, since shuffling exists to spread contention over
a directory listing and there is no directory listing here.

partitioners/iceberg_file_partitioner.py -- groups those rows into read
units against a byte budget (RAY_DATA_ICEBERG_BIN_PACKING_BYTES, default
128 MiB). requires_global_input = True, which is also what makes listing
a single task. It only ever slices and concatenates the blocks the
indexer handed it, so every __ib_ column survives to the reader;
OnlineBinPacker cannot be reused because it rebuilds each bin from paths,
sizes and chunk metadata alone, and for Iceberg the other columns are the
contract.

Bins are sized by estimate_decoded_size(), which is compressed bytes
times RAY_DATA_ICEBERG_DECODE_RATIO (default 4). That multiplier is a
deliberate placeholder: Iceberg metadata records compressed bytes and row
counts, nothing anywhere records decoded size, and measured expansion
spans 1x to 26x, so there is no honest constant. The seam matters more
than the number -- packing is written against decoded bytes, which is the
quantity that governs downstream memory, and the sharpener (column_sizes
over value_counts, both spec-optional) can land later without moving it.

scanners/iceberg_scanner.py -- frozen dataclass pickled to every task;
filter, column and limit pushdown. combine_row_filter does the
Ray-to-Iceberg translation and reports what it could not translate, so
the residual is enforced above the read rather than dropped, which is
where V1 raises instead.

readers/iceberg_file_reader.py -- one ArrowScan per file, which is what
makes positional deletes, field-ID schema resolution, type promotion and
partition backfill correct without reimplementing any of them. One file
at a time because ArrowScan materializes every batch of every task it is
handed, so a whole read unit at once would be held at once. Each batch is
cast to the projected schema, because to_record_batches types a batch
from the file it came out of and two files can disagree on string vs
large_string -- the same normalization V1 does.

An empty projection needed a fix no design anticipated: PyIceberg returns
zero-row batches for it, and zero-column Arrow tables lose their length
on concat, so a count() over no columns would report 0 rows on a
101-row table. The reader decodes the narrowest column instead and swaps
in the same __bsp_stub column the Parquet V2 reader uses.

Tests: four, driving the components directly since nothing wires them up
yet -- position deletes actually being applied (built by hand: PyIceberg
0.11.1 only deletes copy-on-write, so no table it writes exercises
merge-on-read), the byte budget deciding how many read units there are,
packing preserving every manifest column, and a pushed predicate pruning
files during listing rather than only filtering rows.

Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
Third of three, and the one that turns [2a/6] and [2b/6] into a working
reader. read_iceberg goes through DataSourceV2 (ListFiles -> ReadFiles)
when DataContext.use_iceberg_datasource_v2 is set -- default False, env
RAY_DATA_USE_ICEBERG_DATASOURCE_V2. V1 is untouched: outside the tests
this PR deletes no line of existing code.

The flag is separate from use_datasource_v2 (already default True) so
that Parquet's default-on state and Iceberg's opt-in state are
independent knobs.

iceberg_datasource_v2.py is the driver side. Two things it does that are
worth review:

- The schema comes from table metadata, not from a file. Iceberg records
  the schema in the table, so schema_to_pyarrow(scan.projection()) is
  exact and free; the V2 sampling path (open up to 16 files at plan time
  and infer) is declined via schema_needs_file_sample = False.
- The snapshot is resolved once on the driver and latched. Listing now
  runs in a task at execution time, so without this two executions of one
  Dataset could resolve different "current" snapshots, and a task retry
  could straddle a commit. None is a legitimate latched value -- a table
  with no snapshot yet -- so "not resolved" is tracked by the latch, not
  by the id being None.

parallelism and override_num_blocks warn and are ignored, since listing
and packing decide task granularity. That is a deliberate difference
from V1, which bin-packs files into exactly `parallelism` chunks.

Tests. The eleven read-path tests in test_iceberg.py now run against
both readers through the both_iceberg_readers fixture, so every
assertion in them is a parity assertion; seven already existed and four
are new. Two of the four branch on the reader instead of asserting the
same thing, pinning a V1 limitation V2 does not have: a predicate
Iceberg cannot express (V1 raises at read time, V2 filters above the
read) and a declared-but-empty table (V1 reports no schema at all). The
other two are new because V2 needs them and V1 never could fail them:
that two executions of one Dataset see the same snapshot, and that
count() with an empty projection is right.

Three more cover the hooks the shared-seams PR [1/6] added but could not
test there, since Iceberg is their first real implementation: that
planning opens no file at all, not even to sample one (sample_files is
patched to raise); that infer_schema and resolve_partitioning accept a
None sample, which is the whole point of schema_needs_file_sample; and
that a source which declines the metadata-only count() gets an ordinary
read instead.

Deliberately not optimized here, each its own follow-up in this stack:
count() decodes one column per file rather than summing record_count
from metadata; a limit does not stop the plan_files() walk early, only
the per-unit read; every file goes through ArrowScan even when it has no
delete files, an AlwaysTrue residual and no schema drift, so none of the
native Parquet reader's benefits apply; and an oversized file becomes one
oversized task, because an Iceberg scan task names a file and never a
range within one.

Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
The port in [2/6] made count() correct but not cheap: it decodes the
narrowest column of every file, because an empty projection makes
PyIceberg return zero-row batches. This makes it free for the common
table, and answers three hooks [1/6] added.

Iceberg writes each data file's record_count into its own manifests, so
the whole answer is already in the listing task's output -- not even a
footer has to be fetched, which the equivalent Parquet path still does
per file. The manifest has carried __ib_record_count since [2/6] for
exactly this, so nothing about the carrier changes.

- IcebergFileReader implements SupportsMetadata. read_metadata reads
  __ib_record_count off the manifest. That count includes rows a delete
  file removes, so a file carrying delete files is counted by decoding
  it instead -- the same branch PyIceberg's own count() takes, and the
  only one that touches data. Falling back per file rather than
  declining outright means a merge-on-read table still gets the cheap
  path for every clean file in it.
- IcebergScanner.metadata_row_count_is_exact() declines only a limit. A
  row filter is fine, and this is the case the hook was reworded in
  [1/6] to allow: Iceberg plans a residual per file -- the part of the
  filter the file's partition values do not already satisfy -- so where
  the residual is AlwaysTrue the recorded record_count IS the
  post-filter count. A filter on a partition column leaves every listed
  file in that state, which makes the common `WHERE partition = x`
  count free. A filter on a data column leaves none of them and every
  file is decoded, which is the same work the ordinary read path would
  have done. A limit has no such metadata: nothing says which file it
  stops at. Column pruning changes a block's width, not its height.
- IcebergFileIndexer.as_whole_file_indexer() returns itself. The base
  default is None because an indexer may emit one manifest row per
  chunk of a file, which would over-count it: that is the footer-based
  Parquet indexer, where one file's row groups become several rows. An
  Iceberg scan task names a whole file and is never split, so one file
  is always one row here -- grouping those rows into read units happens
  later, in IcebergFilePartitioner. The other half of the default -- no
  per-file IO while listing -- holds too, since planning reads table
  metadata only.

available_metadata() repeats the limit guard on the reader side, so a
caller that reaches read_metadata without going through the rule still
cannot be handed a partial count. It does not refuse a row filter, since
read_metadata accounts for one per file.

One shared-code fix comes with this, because answering the hook under a
filter is what exposes it. PushdownCountFiles deletes the ReadFiles and
puts a MapBatches in its place, and DeriveListFilesPushdown clears a
ListFiles' constraints for any consumer that is not a ReadFiles -- so
the listing ran unfiltered while the count trusted per-file residuals
computed as if it had been filtered, and an unfiltered count came back.
Parquet never saw it: its scanner declines the hook under any predicate.
The rule now pins the predicate onto the listing it rebuilds and marks
it ListFiles.pushdown_is_final, which the derive rule leaves alone. That
is sound rather than an exemption -- count_rows applies the filter
through the same reader the deleted ReadFiles would have used, so the
listing prunes by exactly what the consumer applies.

Left to [4/6]: the snapshot summary's total-records, which answers an
unfiltered whole-table count() on the driver with no tasks at all.

Tests: seven Iceberg ones, none with a V1 counterpart, plus one
unit test for the derive rule's new opt-out. The optimized count plan is
ListFiles -> MapBatches(count_rows) with no ReadFiles and the Iceberg
indexer surviving the rule's rebuild. Three filters keep the rewrite and
still count correctly -- on the partition column, on a data column, and
on a data column the count projection does not include, which is the one
that would break an implementation assuming the filter binds only to
projected columns. Each also asserts the mechanism rather than just the
answer: whether every listed file came back with an AlwaysTrue residual
is exactly whether the count was free, and the rebuilt listing carries
the predicate. Those three tests compared take_all() against a count()
called after it, which a materialized Dataset answers from the block
metadata it just produced rather than by re-optimizing -- so they passed
against the unfiltered count above. They count first now, and fail
against the unfixed rule. A limit declines. read_metadata subtracts
hand-built position deletes while the manifest still carries the
pre-delete number, and available_metadata declines a limited reader but
not a filtered one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
[3/6] made an unfiltered count() free of data reads but not free of
tasks: it still lists the table and sums per-file record_counts. Iceberg
already published that sum. Every snapshot carries a summary, and
total-records in it counts the rows of every data file in the snapshot
-- and the summary sits in table metadata this read has already loaded,
for the schema. So the whole answer is a dict lookup on the driver, and
count() returns without submitting anything.

This also restores parity with V1, which has always answered this
count() from Read.infer_metadata; the V2 port lost it because
ReadFiles.infer_metadata reports nothing.

- Scanner.exact_row_count() is a new hook, defaulting to None ("not
  known"). It reports the rows the scan returns *after* every pushdown
  the scanner holds, so it must decline whenever one of them can reduce
  the count -- the same fail-closed reasoning as
  metadata_row_count_is_exact, since the number is handed to the user as
  the answer and not as an estimate.
- ReadFiles.infer_metadata() reports it as num_rows, which is what
  reaches Dataset.count() -- through _meta_count, before the plan
  count() would otherwise build. Everything else in that BlockMetadata
  stays None as before, and a scanner that answers None leaves the
  operator exactly as it was, so Parquet is unaffected.
- IcebergDatasourceV2._snapshot_row_count() reads the pinned snapshot's
  summary, not the current one's, so a concurrent commit cannot make the
  count disagree with the files the tasks would read. Every summary
  field is optional by spec and writer-populated, so a missing key means
  decline rather than assume. total-records counts rows as the data
  files hold them, which is the table's row count only if no delete file
  removes any -- hence total-position-deletes and total-equality-deletes,
  which must both be present and zero. A table with no snapshot at all
  has no data files, which is a count of exactly zero.
- IcebergScanner.exact_row_count() declines a limit and any filter.
  Unlike metadata_row_count_is_exact, this cannot survive a filter even
  one Iceberg fully satisfies per file: the summary describes the whole
  snapshot and is blind to which files a filter leaves. row_filter is
  the single gate because it is where every filter lands --
  read_iceberg(row_filter=...) sets it directly, and push_filters only
  records a predicate in the same step that ANDs it into row_filter.
  Column pruning changes a block's width, not its height.

Declining costs a slower count(), not a wrong one: [3/6]'s per-file
counts are the fallback, and reading the data is the fallback to that.
So a filtered count() is answered by the rule as before.

Tests: five. The count comes back without a plan, and survives column
pruning. Both filter spellings decline, and they arrive by different
routes -- filter() through push_filters, read_iceberg(row_filter=...)
baked in before the optimizer sees it -- and both still count correctly
via the fallback. A pushed limit makes the scanner decline. Each way the
summary can fail to answer is exercised by mutating the fixture table's
own summary: total-records absent, either delete total absent, either
delete total non-zero, and a non-numeric value. A table with no snapshot
counts zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
remaining = None if self._limit is None else self._limit - rows_read
if remaining is not None and remaining <= 0:
return
for batch in self._scan(remaining).to_record_batches([task]):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This carries over the delete-file behaviour in #66147. to_record_batches deduplicates delete files only
within the tasks it is handed, so one task per call means a delete file shared by k data files in this
input split is fetched k times; _read_deletes has no cache, and the bin-packing partitioner does not
group by delete file.

Measured on your branch rather than inferred: one input split with 8 data files sharing one positional
delete file opens that delete file 8 times, and each data file once. On the existing datasource, which has
the same shape, the cost at a realistic size was 24 data files sharing one 19.7 MiB delete file read from
S3 -- 473 MiB fetched instead of 19.7 MiB, and 74-96s of a 94-113s read spent inside _read_deletes.

#66148 fixes it by wrapping the FileIO so a shared file is fetched once. That keeps one data file per
scan call, so the memory bound in your docstring above is unaffected, and it ports here as a call at the
top of read() since you already keep self._io.

Worth noting for Iceberg format version 3: with deletion vectors each data file gets its own DataFile
entry into one shared Puffin file, and PuffinFile parses every blob on each read, so PyIceberg's
DataFile-level deduplication cannot collapse them even if a call were handed every task. Deduplicating
by path does.

"""Rebuild one PyIceberg scan task per manifest row.

The rebuilt ``DataFile``\\ s carry only the fields the read path reads; a
delete file's ``record_count`` and ``file_size_in_bytes`` are 0 because

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file_size_in_bytes on a delete file does get consulted if the cache in #66148 is ported to this reader:
admission is decided on the declared size, so a file too large for the budget is never wrapped. An earlier
version of that PR wrapped it anyway, read the whole file on every reference and kept none of it; Bugbot
caught that.

Confirmed on your branch: after manifest_from_scan_tasks / scan_tasks_from_manifest, the delete file's
file_size_in_bytes comes back as 0, so nothing would ever be admitted and the cache would be inert.

Could the manifest carry a __ib_delete_sizes list column next to __ib_delete_paths and
__ib_delete_formats? It is one more list_(int64) from f.file_size_in_bytes, and it is the only thing
the port needs from the listing side.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. Please add __ib_delete_sizes in your PR once this is merged.

@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Sep 30, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been automatically closed because there has been no more activity in the 14 days
since being marked stale.

Please feel free to reopen or open a new pull request if you'd still like this to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for your contribution!

@github-actions github-actions Bot closed this Oct 14, 2026
richardliaw pushed a commit that referenced this pull request Nov 6, 2026
…, row_id (#66654)

## Description

Part 1 of 3 of the `read_mcap` move to Datasource V2, based on master.
#67166 is stacked on it, and #67167 on #67166. The rest of the
`read_mcap` stack (#67189 onward) sits on top.

**Problem.** One task reads one whole file:

- Parallelism is capped by the file count: ten 30 GB recordings are ten
tasks.
- Every file is opened, even when its index shows it holds nothing
selected.
- Rows have no stable id, so a checkpointed pipeline cannot resume.

**Change.** `read_mcap` moves to Datasource V2, on by default as for
`read_parquet` (`DataContext.use_datasource_v2`). The legacy datasource
stays as the fallback.

```
files
  │  listing: read each file's index (a few ranged reads)
  │           one row per chunk, weighted by the memory reading it takes
  │           drop chunks that cannot match topics / time_range
  ▼
chunk rows
  │  packing: about 128 MiB per read task
  │           one big file ──► many tasks;  small files ──► one task
  │           with preserve_order: one task per file, in listing order
  ▼
read tasks: seek to their chunks, merge them by log time
```

- New code lives in `datasource_v2/formats/mcap/`. The V2 interfaces do
not change.
- Columns and types match the legacy path, with two changes.
`schema_data` is dictionary-encoded. A new `channel_metadata` column
holds each channel's metadata map as JSON text, dictionary-encoded, so a
block stores it once per channel.
- `data` holds decoded JSON only when every selected channel is JSON.
Otherwise it holds bytes. Planning types it from the selected messages
of the first chunk that holds one, as a read task types the messages it
holds.
- `row_id` is `<path>#c=<chunk offset>:<index>`, stable however the read
is split. It is on with `include_row_id=True`, or automatically under
`CheckpointConfig(id_column="row_id")`.
- New `log_time_order` argument, default `True`. Each task returns each
of its files in `log_time` order. With `preserve_order`, each file is
read by one task, so it comes out in order from start to end, and files
come out in the order they were listed. #67166 splits a large file there
too, only where its order holds.
- A file whose summary cannot be read gets a read task of its own, so
`max_errored_blocks` can skip it alone.
- The task size is `RAY_DATA_MCAP_BIN_PACKING_BYTES` (128 MiB).

**Changed after the first review round.** Each is its own commit:

- Summaries and chunks are read in a few ranged requests, not a few
bytes per GET.
- An unreadable file gets a read task of its own.
- Tables are built column by column, cut counting every column, and
chunks are weighted by the memory reading them takes.
- The read stops at the row limit.
- A file without a chunk index is sorted by log time.
- `channel_metadata` is JSON text, stored once per block.
- With `preserve_order`, each file is kept in one read task, and files
keep their listing order.
- Planning skips a chunk it cannot decode, and types `data` from the
first chunk's messages.

**Measured** on an 845 MB recording with 1,052 chunks: reading its 125
KB summary took about 20,000 GETs and now takes 3.

**Decisions to confirm:**

- Splitting is on by default. Block boundaries change; rows do not.
- With `preserve_order`, this PR reads each file in one task, as the
legacy reader does. #67166 lets it split a large file where its log-time
order holds.
- `ds.schema()` comes from the first chunk that holds a selected
message. A field that first appears later is missing from it; the
docstring says so.

## Related issues

- Related to #65640 and #65641.
- No open PR moves MCAP to Datasource V2. #66524 (ORC) and #66182
(Iceberg) are other formats moving to V2.
- The generated-ID stack (#65886 to #65888) is orthogonal.
- `log_time_order` comes from my #66653, now closed. My legacy-path PRs
#65912, #65913 and #65916 are closed in favour of this one.

## Additional information

```
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py
```

178 passed, 3 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed
every line and ran the tests above locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale The issue is stale. It will be closed within 7 days unless there are further conversation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants