Skip to content

[Data] MCAP on Datasource V2: summary indexer, chunk-level read tasks, row_id - #66654

Merged
richardliaw merged 18 commits into
masterfrom
marwan/mcap-dsv2
Nov 6, 2026
Merged

richardliaw merged 18 commits into
masterfrom
marwan/mcap-dsv2

Conversation

@marwan116

@marwan116 marwan116 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 1 of 3 of the read_mcap move to Datasource V2, based on master. #67166 is stacked on it, and #67167 on #67166. The rest of the read_mcap stack (#67189 onward) sits on top.

Problem. One task reads one whole file:

  • Parallelism is capped by the file count: ten 30 GB recordings are ten tasks.
  • Every file is opened, even when its index shows it holds nothing selected.
  • Rows have no stable id, so a checkpointed pipeline cannot resume.

Change. read_mcap moves to Datasource V2, on by default as for read_parquet (DataContext.use_datasource_v2). The legacy datasource stays as the fallback.

files
  │  listing: read each file's index (a few ranged reads)
  │           one row per chunk, weighted by the memory reading it takes
  │           drop chunks that cannot match topics / time_range
  ▼
chunk rows
  │  packing: about 128 MiB per read task
  │           one big file ──► many tasks;  small files ──► one task
  │           with preserve_order: one task per file, in listing order
  ▼
read tasks: seek to their chunks, merge them by log time
  • New code lives in datasource_v2/formats/mcap/. The V2 interfaces do not change.
  • Columns and types match the legacy path, with two changes. schema_data is dictionary-encoded. A new channel_metadata column holds each channel's metadata map as JSON text, dictionary-encoded, so a block stores it once per channel.
  • data holds decoded JSON only when every selected channel is JSON. Otherwise it holds bytes. Planning types it from the selected messages of the first chunk that holds one, as a read task types the messages it holds.
  • row_id is <path>#c=<chunk offset>:<index>, stable however the read is split. It is on with include_row_id=True, or automatically under CheckpointConfig(id_column="row_id").
  • New log_time_order argument, default True. Each task returns each of its files in log_time order. With preserve_order, each file is read by one task, so it comes out in order from start to end, and files come out in the order they were listed. [Data] read_mcap V2: cut large files where their log-time order holds #67166 splits a large file there too, only where its order holds.
  • A file whose summary cannot be read gets a read task of its own, so max_errored_blocks can skip it alone.
  • The task size is RAY_DATA_MCAP_BIN_PACKING_BYTES (128 MiB).

Changed after the first review round. Each is its own commit:

  • Summaries and chunks are read in a few ranged requests, not a few bytes per GET.
  • An unreadable file gets a read task of its own.
  • Tables are built column by column, cut counting every column, and chunks are weighted by the memory reading them takes.
  • The read stops at the row limit.
  • A file without a chunk index is sorted by log time.
  • channel_metadata is JSON text, stored once per block.
  • With preserve_order, each file is kept in one read task, and files keep their listing order.
  • Planning skips a chunk it cannot decode, and types data from the first chunk's messages.

Measured on an 845 MB recording with 1,052 chunks: reading its 125 KB summary took about 20,000 GETs and now takes 3.

Decisions to confirm:

  • Splitting is on by default. Block boundaries change; rows do not.
  • With preserve_order, this PR reads each file in one task, as the legacy reader does. [Data] read_mcap V2: cut large files where their log-time order holds #67166 lets it split a large file where its log-time order holds.
  • ds.schema() comes from the first chunk that holds a selected message. A field that first appears later is missing from it; the docstring says so.

Related issues

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py

178 passed, 3 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116
marwan116 requested a review from a team as a code owner October 2, 2026 04:08
@marwan116 marwan116 added the data Ray Data-related issues label Oct 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new V2 datasource for MCAP files (MCAPDatasourceV2) in Ray Data, enabling summary-driven listing, chunk-level read tasks, and deterministic row IDs. While the overall design is solid, several critical issues were identified in the implementation: pa.repeat does not exist in PyArrow and will cause an AttributeError during path partitioning; returning raw bytes on JSON decoding failure in decode_payload leads to mixed-type columns and pickle crashes; schema inference needs to handle None values from decode_payload to prevent type inference errors; and accessing summary.channels directly by channel ID can raise a KeyError if the ID is missing from the summary.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2 branch 3 times, most recently from 6191ee4 to baf5f6e Compare October 5, 2026 02:16

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 added this pull request to stack #66712 October 5, 2026 02:25
@marwan116 marwan116 self-assigned this Oct 5, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2 branch 2 times, most recently from 88831ee to 67a3255 Compare October 5, 2026 03:37

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0b506e3. Configure here.

@marwan116
marwan116 requested a review from AarryaSaraf October 29, 2026 19:53
@AarryaSaraf AarryaSaraf added the go add ONLY when ready to merge, run all tests label Oct 29, 2026
marwan116 and others added 18 commits November 5, 2026 18:31
…, row_id

Problem:
- read_mcap reads each file in one task, so parallelism is capped by the
  number of files.
- Every file is opened, even when its summary shows it holds nothing the
  read selects.
- Rows have no stable id, so a checkpointed pipeline cannot resume.

Change: read_mcap runs on Datasource V2 when
DataContext.use_datasource_v2 is set (the default). The old reader stays
as the fallback.

  file summaries -> one listing row per chunk -> ~128 MiB read tasks

- Listing reads each file's summary and emits one row per chunk. A file
  that cannot match the selection is dropped before any data is read.
- Chunks are packed into read tasks of about 128 MiB uncompressed
  (RAY_DATA_MCAP_BIN_PACKING_BYTES). A big file spans many tasks; small
  files share one.
- Each task seeks straight to its chunks. The new log_time_order
  argument (default True) merges them by log time.
- row_id names a message by its file, chunk offset and position. It is
  the same however the read is split. Enable it with
  include_row_id=True; a row_id checkpoint turns it on automatically.
- Column pruning and limits are pushed into the reader.
- data holds decoded JSON only when every selected channel is JSON.
  Otherwise it holds bytes. mcap_data_column decides this once, from
  the sampled files' summaries.
- Columns match the old reader, plus channel_metadata (a map).
  schema_data is dictionary-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- mcap's SeekingReader parses a summary a few bytes per read. On S3
  each read of a raw file is one request, and the listing and every
  read task of a file read its summary. One summary read took about
  20,000 requests.
- Chunk records were read field by field too, and a file without a
  chunk index was scanned with no read buffer.

Change:
- New mcap_io module. read_summary_from checks both magic bytes and the
  footer, then parses the summary section in memory, all in one to
  three ranged reads. A file that is not a complete MCAP file raises a
  ValueError naming it.
- read_chunk fetches a chunk record in one ranged read. The listing,
  planning and read tasks use both.
- buffered wraps a linear scan in a read buffer of
  DataContext.streaming_read_buffer_size bytes.
- Tests that patched mcap's readers now count planning scans and strip
  summary records through mcap_testing helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The per-shard packing test defined its pack and units helpers inside
  itself, so no other listing test could pack files into read tasks
  the way a ListFiles task does.

Change:
- pack(datasource, shard) and chunk_units(tasks) move to module level
  in test_mcap_listing.py. Each pack call builds its own partitioner,
  as each ListFiles task does.
- Test code only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A file whose summary cannot be read, such as a recording cut short,
  failed the listing task and every file listed with it.
- Planning reads some summaries on the driver, so a bad sampled file
  failed the whole read there, where max_errored_blocks cannot help.

Change:
- The listing lists such a file as one row that names no chunk and
  weighs a whole read task, so the packer puts no other file in its
  task. The reader takes that row as the whole file.
- Reading it fails that task alone, with an error that names the file.
  max_errored_blocks can then skip just that file, as with the legacy
  reader.
- Planning skips a sampled file whose summary cannot be read. Its own
  read task reports the error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Building a table took several times the table's size. Each row's
  values waited in Python lists until the build, and every row carried
  its own copy of its channel's schema data and metadata.
- schema_data was built as plain binary, a copy per row, and only then
  dictionary-encoded.
- The lists stayed alive while the finished table was yielded.

Change:
- Integers go into typed arrays, and payloads and row ids into one
  buffer each, which the finished arrays wrap without a copy.
- What a row takes from its channel and schema is stored once per
  table, with an int32 per row pointing at it. schema_data is built as
  a dictionary from the start.
- build empties the builder before it returns the table, so a yielded
  table is the only copy of its rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The reader cut a table once its payloads, plus a flat 96 bytes a
  row, reached the target block size. row_id, the channel columns and
  appended columns such as path were not counted.
- So with row_id or long channel metadata on, a table grew far past
  the target: on a file of small IMU messages, to nearly four times it.

Change:
- estimated_bytes counts what each row adds to every column the reader
  emits: the payload, the integers, row_id, the channel strings and
  metadata, and each new schema_data entry once.
- The reader passes the per-row bytes of its partition and synthesized
  columns, such as path, as extra_row_bytes.
- Decoded JSON counts six bytes per payload byte, since its values are
  Python objects until the table is built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The packer filled each read task up to its budget by the chunks'
  uncompressed bytes. A row costs more in a table than in its chunk,
  so a full task built tables many times its budget: up to 30 times.

Change:
- A chunk's listing weight is 1.2 times the bytes its rows take as a
  table: its uncompressed records plus, per message, the columns a row
  adds. Those are the channel's strings and metadata, and row_id and
  path when the read has them (MessageRowLayout).
- A chunk's message count comes from the length of its message indexes,
  16 bytes per message, and falls back to the statistics without them.
- RAY_DATA_MCAP_BIN_PACKING_BYTES budgets this estimate, 128 MiB by
  default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- SeekLog and chunks_read, which record the chunks a read fetches,
  lived in the data column tests, so the reader tests could not use
  them.

Change:
- SeekLog, _SeekLoggingFile and chunks_read move unchanged from
  test_mcap_data_column.py to mcap_testing.py.
- Test code only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A read task built every table of its share of a file, then sliced
  the tables down to the limit. So a small limit still fetched and
  decoded every chunk of the task, 125 on a task of IMU messages.

Change:
- The reader passes the rows still wanted to each file and stops
  pulling messages once it has them. No later message is added to a
  table, and no later chunk is fetched or decoded.
- A task whose limit is met does not open its next file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- log_time_order=True did not sort a file without a chunk index at
  all, though one task reads it whole.
- A file bigger than one read task is split across tasks, and each
  task orders only its own chunks. The docstring did not say so.

Change:
- A file without a chunk index is sorted by log time in its task, as
  the legacy reader does. The sort is stable, so equal log times keep
  file order. With a limit, only the earliest messages are held.
- The log_time_order docstring says the order holds within a read
  task, not across tasks or files, even with preserve_order, and that
  ds.sort("log_time") orders a whole file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- channel_metadata was a map on every row. ROS 2 stores each channel's
  QoS profiles there, so on small, high-rate messages the metadata
  outweighed the message: 70% of the output on an IMU file.
- A dictionary of map values would store it once, but Parquet cannot
  write one.

Change:
- channel_metadata is the channel's metadata map as JSON text with
  sorted keys, dictionary-encoded like schema_data:
  dictionary<int32, string>. A table stores each distinct value once,
  and a row holds an int32 index.
- The listing estimate counts 4 bytes per row for it, and the builder
  counts each text once.
- The read_mcap docstring says to read it with json.loads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The packer places a file's chunks one by one, so a file can be split
  across read tasks, even a small one that lands in a partly full task.
- Those tasks are sent in any order. With preserve_order, a split file
  came out with its first messages last.

Change:
- When the read keeps order (DataContext.execution_options
  .preserve_order), OrderedFileGrouper joins each file's chunk rows
  into one run before the packer sees them. The packer splits no run,
  so one task reads the file and merges its chunks by log time.
- A file heavier than a read task's budget gets a task of its own and
  holds that much memory, as the legacy reader does.
- Without preserve_order, the rows go to the packer as listed.
- The log_time_order docstring says what order a read returns, with
  and without preserve_order.
- New tests: the grouping, and a preserve_order read of three files
  that an unordered read would split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Planning decodes one selected message to type the data column. When
  the chunk it reads was corrupt, planning raised on the driver, so one
  damaged chunk failed the whole read before any task ran.
- A corrupt record in a file without a chunk index did the same.

Change:
- Planning skips a sampled chunk that cannot be decoded and types data
  from the next chunk. The chunk's own read task reports the error.
- A record that does not parse ends the scan of a file without a chunk
  index, so data is typed from the messages before it.
- An error reading the file still raises.
- New tests: a sampled chunk with a corrupt payload or header, and a
  corrupt record in a file without a chunk index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Planning types the data column from the files it samples. Nothing
  said what happens when a file past the sample holds other JSON
  values.

Change:
- No code change. The read_mcap docstring says each read task types
  its own JSON values, as the legacy reader types each file. A field
  only some files have is null in the other rows, and a field whose
  type differs between files fails where their rows are combined.
- New tests pin it. Past the sample, a new field and an int that
  becomes a float unify with the planned blocks, and a clashing type
  fails where blocks combine. A CDR file past the sample fails its read
  task, and mixed types in one task are refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Planning typed data from one sampled message, while a read task
  types it from every message it holds. When the first message lacked
  a field, or had an int where later ones had a float, ds.schema()
  disagreed with the blocks. The legacy reader's schema() reads the
  first file whole, so it agreed.

Change:
- Planning types data from every selected message of the first chunk
  that holds one, up to 1,000, with the function a read task uses. In
  a file without a chunk index it takes the first 1,000 the scan
  meets.
- A payload that does not decode is left out of the type, and its read
  task reports it, so one bad message no longer fails planning.
- The read_mcap docstring says that schema() comes from the first
  chunk, so a field that first appears later is missing from it.
- New tests: a field only some messages have, an int that is a float
  elsewhere, a malformed payload, and the 1,000-message cap. End to
  end, schema() matches the materialized data on both read paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The read-unit comment in MCAPReader._tables said a checkpoint that
  hands a task's unit back skips only that task's rows. Listing
  excludes units chunk by chunk, and a task that owns several chunks
  of a file names only the first, so it would skip only that chunk.

Change:
- The comment says so, and that nothing hands units back yet: a row_id
  checkpoint filters rows by id.
- Comment only, no behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- With preserve_order, the grouper handed whole files to the shared
  packer. The packer sends a heavy file's task at once and holds light
  files in open bins, so a large file listed after small ones came
  out first. A recording split across files came out scrambled, and
  limit could return rows from a later file.

Change:
- With preserve_order, the grouper packs whole files into tasks
  itself, in listing order. A file joins the open task while it fits
  the budget; otherwise the open task leaves first.
- The read_mcap docstring says files come out in listing order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The tests/unit module list depends only on tests/conftest.py, so
  Bazel leaves tests/unit/datasource_v2/conftest.py out of the
  runfiles. Premerge failed the MCAP listing, reader and data-column
  tests with "fixture 'chunked_file' not found".

Change:
- Add a py_library for that conftest and depend on it from the
  tests/unit module list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@richardliaw
richardliaw merged commit f0a9db5 into master Nov 6, 2026
6 checks passed
@richardliaw
richardliaw deleted the marwan/mcap-dsv2 branch November 6, 2026 19:36
goutamvenkat-anyscale pushed a commit that referenced this pull request Nov 9, 2026
… source's own metadata (#67037)

Three changes to shared Datasource V2 code so a source whose listing
already knows
more than a path and a size can use the common machinery instead of
forking it. 1 of
5 in a stack moving `read_iceberg` onto DSv2; this one touches only
shared code.

**A source's own manifest columns survive bin packing** —
`interfaces/file_manifest.py`,
`common/online_bin_packer.py`. `FileManifest` already lets a source add
columns beside
the three required ones, but `OnlineBinPacker` rebuilt each bin from
those three alone,
so the rest were dropped. Adds `FileManifest.extra_columns` and an
`extra_columns`
argument to `construct_manifest`, and carries a zero-copy one-row table
on each
`BinItem`. Packing a manifest with two source columns and reading the
names back out:

```
master:   ['__path', '__file_size', '__file_chunk_metadata']
this PR:  [... , '__ib_record_count', '__ib_delete_paths']
```

```mermaid
flowchart TD
    L["ListFiles task"] -->|"FileManifest: 3 required + 2 source columns"| P["OnlineBinPacker"]
    P -->|"before: rebuilt from the 3 required"| B["ReadFiles sees 3"]
    P -->|"after: BinItem.extra_columns carries each row"| A["ReadFiles sees 5"]
```

**The pushdown mixins carry their own state** —
`interfaces/pushdown.py`,
`common/arrow_file_scanner.py`. `SupportsColumnPruning` and
`SupportsLimitPushdown` were
pure interfaces, so every scanner rewrote the same field and the same
`replace(self, ...)` body. They are now frozen dataclasses with
keyword-only fields and
default implementations: pruning intersects, limit takes the smaller.
`ArrowFileScanner`
drops 46 lines. Intersecting also fixes a bug — the old bodies overwrote
the projection
and tested it for truthiness, so an empty projection (what `count()`
pushes down)
re-widened:

```python
scanner.prune_columns([]).prune_columns(["a"]).pruned_column_names()
# master: ('a',)   this PR: ()
```

**`ListFiles.pushdown_is_final`** —
`logical/operators/read_operator.py`,
`logical/rules/derive_list_files_pushdown.py`,
`logical/rules/pushdown_count_files.py`.
`DeriveListFilesPushdown` clears a listing's constraints when the
consumer is not a
`ReadFiles`, since an arbitrary consumer will not apply them.
`PushdownCountFiles`
rewrites `Count(ReadFiles(ListFiles))` into `MapBatches(count_rows,
ListFiles)`, and its
`count_rows` does apply the scanner's filter, through the same reader
the deleted
`ReadFiles` would have used. The flag lets that rule say so instead of
silently losing
the predicate and returning an unfiltered count.

---

Tests: `pytest
python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py
test_pushdown_mixins.py test_derive_list_files_pushdown.py` — 31 passed.
`test_pushdown_mixins.py` is new; the other two gain cases for extra
columns surviving
packing/splitting/merging and for `pushdown_is_final`. ruff 0.8.4
(default and
`--select I`), black 22.10.0, pydoclint 0.9.1 and pyrefly 0.51.0 clean
on these files.

Not a duplicate: no open PR changes `OnlineBinPacker`, the pushdown
mixins or
`PushdownCountFiles`. #66654 (MCAP) and #66573 (Hive HS2) consume this
shared code
rather than modify it, and the mixin consolidation is what review on
#66654 asked for.

AI assistance (Claude Code) was used. I have reviewed every changed line
and ran the
tests above locally.

Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
richardliaw pushed a commit that referenced this pull request Nov 12, 2026
…#67166)

## How to review

Start with the module docstring of `mcap_file_cuts.py`: it states when a
file is cut and when it is read whole. Then read
`test_mcap_file_cuts.py`, one case per test.

| File | What changes | Lines |
|---|---|---|
| `mcap_file_cuts.py` | Cuts a heavy file at the task budget, and keeps
files in listing order. | 190 |
| `mcap_datasource_v2.py`, `mcap_file_indexer.py`,
`mcap_task_summary.py` | Use the cuts when listing a file. | 64 |
| `read_api.py` | The docstring says when a file keeps its order. | 17 |
| `test_mcap_file_cuts.py`, `test_mcap.py` | The cases, and an
end-to-end ordered read. | 413 |

**Problem.**

- With `preserve_order`, #66654 reads each file in one task, even when
the read does not keep log-time order. A large recording then loses its
parallelism.

**Change.**

- With `preserve_order` and `log_time_order`, each file is read whole by
one task. That task merges the file's chunks by log time and holds one
ranged read, one chunk and one block at a time, whatever the file's
size. Chunks that overlap in time are held together, as before.
- Other reads cut a large file at the task budget. With
`preserve_order`, its tasks are sent in order.
- With `preserve_order`, files also come out in the order they were
listed. Small files are packed in listing order, and a large file's
tasks leave after the files listed before it.
- **Behaviour change:** a file lighter than one task stays in one task,
with or without `preserve_order`. Before, an unordered read split such
files to fill tasks, so tasks now come out less full: 20 files of 0.6
budget make 20 tasks instead of 13. Splitting made video reads slower,
because a split piece decodes back to its keyframe and drops those
frames. Message rows read the same either way.

**Decisions to confirm:**

- With `preserve_order` and `log_time_order`, read each file whole
instead of cutting it where its log-time order holds. The cuts bought
parallelism, not memory, since chunks that overlap in time were never
cut apart. They needed per-chunk end times, a tied-time rule and a
memory warning.
- Keep light files whole in unordered reads. On 128 robot episodes with
two cameras, splitting to fill tasks saved 2 to 15 tasks and made the
read 2% to 16% slower.

## Related issues

- The summary handoff and ranged reads were split out below this PR at a
reviewer's request. No other open PR touches these files.

## Additional information

```
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py
```

250 passed, 4 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed
every line and ran the tests above locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants