Skip to content

[Data] read_mcap V2: cut large files where their log-time order holds - #67166

Merged
richardliaw merged 7 commits into
masterfrom
marwan/mcap-dsv2-io
Nov 12, 2026
Merged

richardliaw merged 7 commits into
masterfrom
marwan/mcap-dsv2-io

Conversation

@marwan116

@marwan116 marwan116 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

How to review

Start with the module docstring of mcap_file_cuts.py: it states when a file is cut and when it is read whole. Then read test_mcap_file_cuts.py, one case per test.

File What changes Lines
mcap_file_cuts.py Cuts a heavy file at the task budget, and keeps files in listing order. 190
mcap_datasource_v2.py, mcap_file_indexer.py, mcap_task_summary.py Use the cuts when listing a file. 64
read_api.py The docstring says when a file keeps its order. 17
test_mcap_file_cuts.py, test_mcap.py The cases, and an end-to-end ordered read. 413

Problem.

Change.

  • With preserve_order and log_time_order, each file is read whole by one task. That task merges the file's chunks by log time and holds one ranged read, one chunk and one block at a time, whatever the file's size. Chunks that overlap in time are held together, as before.
  • Other reads cut a large file at the task budget. With preserve_order, its tasks are sent in order.
  • With preserve_order, files also come out in the order they were listed. Small files are packed in listing order, and a large file's tasks leave after the files listed before it.
  • Behaviour change: a file lighter than one task stays in one task, with or without preserve_order. Before, an unordered read split such files to fill tasks, so tasks now come out less full: 20 files of 0.6 budget make 20 tasks instead of 13. Splitting made video reads slower, because a split piece decodes back to its keyframe and drops those frames. Message rows read the same either way.

Decisions to confirm:

  • With preserve_order and log_time_order, read each file whole instead of cutting it where its log-time order holds. The cuts bought parallelism, not memory, since chunks that overlap in time were never cut apart. They needed per-chunk end times, a tied-time rule and a memory warning.
  • Keep light files whole in unordered reads. On 128 robot episodes with two cameras, splitting to fill tasks saved 2 to 15 tasks and made the read 2% to 16% slower.

Related issues

  • The summary handoff and ranged reads were split out below this PR at a reviewer's request. No other open PR touches these files.

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py

250 passed, 4 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes the MCAP V2 datasource by introducing OrderedFileCutter to split large MCAP files into ordered read tasks and TaskSummaryPartitioner to carry metadata directly in task manifests, avoiding redundant summary reads. It also implements ranged chunk reads to group close chunk records together, alongside updated documentation and comprehensive unit tests. Since there are no review comments, we have no feedback to provide.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@ray-gardener ray-gardener Bot added the data Ray Data-related issues label Oct 19, 2026
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-io branch from 87559ae to 4ebef63 Compare October 20, 2026 13:01

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-io branch from 4ebef63 to f6f6167 Compare October 20, 2026 13:14
@marwan116
marwan116 added this pull request to stack #67196 October 20, 2026 16:31
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-io branch from f6f6167 to d7d7671 Compare October 29, 2026 14:04
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-io branch 2 times, most recently from df03a20 to 43b1a6d Compare October 29, 2026 23:38
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-io branch from 43b1a6d to 408b105 Compare November 6, 2026 02:36
Base automatically changed from marwan/mcap-dsv2 to master November 6, 2026 19:36
richardliaw pushed a commit that referenced this pull request Nov 6, 2026
…, row_id (#66654)

## Description

Part 1 of 3 of the `read_mcap` move to Datasource V2, based on master.
#67166 is stacked on it, and #67167 on #67166. The rest of the
`read_mcap` stack (#67189 onward) sits on top.

**Problem.** One task reads one whole file:

- Parallelism is capped by the file count: ten 30 GB recordings are ten
tasks.
- Every file is opened, even when its index shows it holds nothing
selected.
- Rows have no stable id, so a checkpointed pipeline cannot resume.

**Change.** `read_mcap` moves to Datasource V2, on by default as for
`read_parquet` (`DataContext.use_datasource_v2`). The legacy datasource
stays as the fallback.

```
files
  │  listing: read each file's index (a few ranged reads)
  │           one row per chunk, weighted by the memory reading it takes
  │           drop chunks that cannot match topics / time_range
  ▼
chunk rows
  │  packing: about 128 MiB per read task
  │           one big file ──► many tasks;  small files ──► one task
  │           with preserve_order: one task per file, in listing order
  ▼
read tasks: seek to their chunks, merge them by log time
```

- New code lives in `datasource_v2/formats/mcap/`. The V2 interfaces do
not change.
- Columns and types match the legacy path, with two changes.
`schema_data` is dictionary-encoded. A new `channel_metadata` column
holds each channel's metadata map as JSON text, dictionary-encoded, so a
block stores it once per channel.
- `data` holds decoded JSON only when every selected channel is JSON.
Otherwise it holds bytes. Planning types it from the selected messages
of the first chunk that holds one, as a read task types the messages it
holds.
- `row_id` is `<path>#c=<chunk offset>:<index>`, stable however the read
is split. It is on with `include_row_id=True`, or automatically under
`CheckpointConfig(id_column="row_id")`.
- New `log_time_order` argument, default `True`. Each task returns each
of its files in `log_time` order. With `preserve_order`, each file is
read by one task, so it comes out in order from start to end, and files
come out in the order they were listed. #67166 splits a large file there
too, only where its order holds.
- A file whose summary cannot be read gets a read task of its own, so
`max_errored_blocks` can skip it alone.
- The task size is `RAY_DATA_MCAP_BIN_PACKING_BYTES` (128 MiB).

**Changed after the first review round.** Each is its own commit:

- Summaries and chunks are read in a few ranged requests, not a few
bytes per GET.
- An unreadable file gets a read task of its own.
- Tables are built column by column, cut counting every column, and
chunks are weighted by the memory reading them takes.
- The read stops at the row limit.
- A file without a chunk index is sorted by log time.
- `channel_metadata` is JSON text, stored once per block.
- With `preserve_order`, each file is kept in one read task, and files
keep their listing order.
- Planning skips a chunk it cannot decode, and types `data` from the
first chunk's messages.

**Measured** on an 845 MB recording with 1,052 chunks: reading its 125
KB summary took about 20,000 GETs and now takes 3.

**Decisions to confirm:**

- Splitting is on by default. Block boundaries change; rows do not.
- With `preserve_order`, this PR reads each file in one task, as the
legacy reader does. #67166 lets it split a large file where its log-time
order holds.
- `ds.schema()` comes from the first chunk that holds a selected
message. A field that first appears later is missing from it; the
docstring says so.

## Related issues

- Related to #65640 and #65641.
- No open PR moves MCAP to Datasource V2. #66524 (ORC) and #66182
(Iceberg) are other formats moving to V2.
- The generated-ID stack (#65886 to #65888) is orthogonal.
- `log_time_order` comes from my #66653, now closed. My legacy-path PRs
#65912, #65913 and #65916 are closed in favour of this one.

## Additional information

```
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py
```

178 passed, 3 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed
every line and ran the tests above locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
@marwan116
marwan116 removed this pull request from stack #67196 November 7, 2026 01:26
@marwan116
marwan116 changed the base branch from master to marwan/mcap-dsv2-ranged-reads November 7, 2026 01:26
@marwan116
marwan116 added this pull request to stack #67931 November 7, 2026 01:27
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-io branch 2 times, most recently from 4deaca8 to fafef66 Compare November 9, 2026 05:44
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-io branch from fafef66 to aa4389e Compare November 9, 2026 17:56
@marwan116
marwan116 removed this pull request from stack #67931 November 9, 2026 19:35
@marwan116
marwan116 added this pull request to stack #67983 November 9, 2026 19:35

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116 marwan116 self-assigned this Nov 10, 2026
richardliaw pushed a commit that referenced this pull request Nov 11, 2026
#67929)

## Summary

Read tasks take their files' summary records from the listing's manifest
instead of reading each file's summary again.

## How to review

Start with `mcap_task_summary.py`: it holds the records each read task
needs, and its module docstring shows the flow. Then read how the reader
uses them.

| File | What changes | Lines |
|---|---|---|
| `mcap_task_summary.py` | New. Packs each task's summary records into
the manifest. | 240 |
| `mcap_reader.py` | Reads the records from the manifest instead of the
file. | 120 |
| `mcap_summary.py`, `mcap_file_indexer.py` | The listing keeps the raw
records it already read. | 111 |
| `mcap_io.py`, `mcap_chunks.py`, `mcap_datasource_v2.py` | Small hooks.
| 94 |
| `online_bin_packer.py`, `mcap_file_cuts.py` | Docstrings only. | 21 |
| `test_mcap_task_summary.py` | Rows match reading the summary from the
file. | 427 |

## Before

The listing reads every file's summary to plan the read, then each read
task reads the same summaries again: one to three ranged requests per
file, per task.

## After

The listing hands each read task the summary records its chunks need, in
the manifest (`mcap_task_summary`). A read task opens its files' chunks
directly.

```mermaid
flowchart LR
    L[listing: read summary once] -->|records in manifest| T[read task]
    T --> C[read chunks]
```

- Each manifest row is read on its own, so a path listed twice is read
twice. The packer contract and Parquet read such a path the same way.
- The reader asserts that its task carries the summary-records column.
It reads a file's summary itself only for a file the listing could not
read, and that read fails with the file's own error.
- Also: the `OnlineBinPacker` docstring describes per-shard packing
correctly, and the MCAP tests share one `pack` helper. Neither changes
behaviour.

## Subsystems

IO (`_internal/datasource_v2/formats/mcap`).

## Testing

```
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py
```

206 passed, 3 skipped. pre-commit and pyrefly clean.

Not a duplicate: split out of #67166 at a reviewer's request. AI
assistance (Claude Code) was used to draft this change; I reviewed every
line and ran the tests above locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0acbae7. Configure here.

Base automatically changed from marwan/mcap-dsv2-ranged-reads to master November 12, 2026 19:33
marwan116 and others added 7 commits November 12, 2026 11:33
Problem:
- An ordered read (preserve_order) keeps each file in one read task, so
  a large recording is read by one task and held in memory whole.
- In other reads the packer places a file's chunks one by one, so a
  small file can be split across two shared tasks.

Change:
- OrderedFileCutter replaces OrderedFileGrouper. It cuts each large
  file into read tasks and sends them at once, in order. It cuts only
  where every chunk before the cut ends no later than every chunk after
  it starts, so with preserve_order each file still comes out in
  log-time order from start to end.
- In a recording with one writer every boundary allows a cut, so tasks
  are as full as in an unordered read. Where chunks overlap in time, as
  when cameras and robot state are written to chunks of their own, the
  overlapping stretch stays in one task.
- A small file goes to the packer whole, so it is never split.
- A file whose rows cannot be placed in time goes to the packer as
  listed: one read whole, an unreadable one, or rows without a chunk
  index record.
- With log_time_order=False a file is cut in file order, where every
  boundary keeps the order.
- The log_time_order docstring says where an ordered read cuts a file.
- New tests: the cuts and the tasks' summary records. The preserve_order
  read of three files now checks that each file is split. Tests that
  force small tasks now check that they still split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A file whose chunks overlap in time has few cuts that keep log-time
  order, so it was read by one task on every read. Without
  preserve_order its blocks come out in any order anyway, so that one
  task bought nothing and cost parallelism.
- In an ordered read, that task's merge holds every chunk that overlaps
  the current log time. A file whose chunks all overlap is held whole,
  and nothing told the user.

Change:
- The cutter keeps a file's cuts in log-time order only when the read
  keeps its blocks in order. It reads
  DataContext.execution_options.preserve_order in the listing task,
  which runs under the dataset's own context.
- Otherwise every boundary allows a cut. Chunks are still taken by
  first log time, so each task spans one stretch of time.
- A recording with one writer is cut the same way in either case.
- On a tied log time, an ordered cut is allowed only when the chunks
  before it that end then lie earlier in the file, since the merge
  yields equal log times in file order.
- An ordered task whose overlapping chunks would hold more than four
  read-task budgets at once logs one warning per process. It names the
  file and suggests read_mcap(memory=...) or dropping preserve_order.
- The warning stays quiet when each read task already reserves at
  least that much. read_mcap passes memory, or else
  ray_remote_args["memory"], to the datasource, which hands it to the
  cutter.
- The read_mcap docstring says that an overlapping stretch stays in
  one task only with preserve_order, that the task holds it whole, and
  when the warning fires.
- New tests: an unordered read may cut a file whose chunks overlap in
  time at any boundary; the warning fires once for a file with no clean
  cut, and not for shallow overlap, an unordered read, one big chunk or
  a reservation that covers it; read_mcap's memory reaches the cutter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- With preserve_order, the cutter sent a heavy file's tasks at once
  and left light files in the packer's open bins. A large file listed
  after small ones came out first.

Change:
- With preserve_order, the cutter packs light files and files it
  cannot place in time itself, in listing order. A heavy file's tasks
  leave after the files listed before it.
- Without preserve_order, light files go to the packer as before.
- The read_mcap docstring says files come out in listing order with
  preserve_order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
This is 2a's fix applied to 2c's code. The 2c-specific part is the
docstring of OrderedFileCutter's _rows_by_file.

Problem:
- The reader merged the manifest rows of one path. A path listed twice
  was read once when its copies shared a read task and twice when they
  did not, so the row count depended on the task budget.
- The packer keeps one row per listed copy, and Parquet reads a
  repeated path twice.
- Three fallbacks could turn off the summary handoff unnoticed: a
  missing task summary column, a path named by two rows, and a listing
  without record columns all made read tasks read summaries again.

Change:
- The reader reads each manifest row on its own, in manifest order.
- task_summaries asserts the task summary column, and the partitioner
  asserts the listing record columns. A null record only marks a
  file the listing could not read.
- Tests pack listing rows into a read task (as_read_task) instead of
  reading listing manifests directly.
- OrderedFileCutter's _rows_by_file states why copies stay apart: each
  listing manifest holds one listed file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The cutter checked for a missing chunk index column and for rows
  naming several chunks. The listing never makes such rows, and
  TaskSummaryPartitioner reads the column before the cutter runs.
- Several cutter tests repeated what other tests already assert.
- The memory test mocked MCAPDatasourceV2.__init__ to see one argument.

Change:
- Read the chunk index column directly, and drop the one-chunk check.
- Drop the empty-record and two-chunk rows from unplaceable_files().
- Drop tests whose cases other tests cover, and merge the light-file
  test and the two cut-anywhere tests into parametrized tests.
- Check the memory reservation on the cutter that read_mcap's plan
  holds, with no mock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- An ordered read warned only once its log-time merge held four read
  task budgets. Nothing measured chose four, and a task with eight
  overlapping chunks peaked at 2.5 budgets with no warning.
- The warning named `preserve_order`, but the merge comes from
  `log_time_order`, and it gave no value to reserve.

Change:
- Warn once the merge holds more than the budget a task is sized for.
- Name the reservation to pass, `read_mcap(memory=...)` rounded up to
  a whole MiB, and `log_time_order=False` as the other way out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- An ordered log-time read cut a large file only where its log-time
  order held. That needed per-chunk end times, a tied-time rule and a
  memory warning with a reservation handed down from read_mcap.
- The cuts bought parallelism, not memory. A whole-file task holds one
  ranged read, one chunk and one block at a time, whatever the file's
  size. Chunks that overlap in time were never cut apart, so a task
  held them together either way.

Change:
- With preserve_order and log_time_order, each file is read whole by
  one task, in listing order.
- Other reads still cut a large file at the budget, and files still
  keep their listing order with preserve_order.
- Remove the ordered-task memory warning, read_task_memory on the
  datasource, and their tests.
- chunk_start_time replaces chunk_log_times, since only the first log
  time is still read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
richardliaw pushed a commit that referenced this pull request Nov 12, 2026
…67930)

## Summary

A read task fetches neighbouring chunks in one ranged request instead of
one request per chunk. The listing holds at most one read task's budget
of summaries, and a read task's memory is estimated from what its chunks
hold. Also four small `read_mcap` fixes that fail early or speak up
instead of failing late or quietly.

## How to review

Start with `mcap_io.plan_chunk_reads`: it groups neighbouring chunks
into one read. `read_chunks` then frees each read's bytes after its last
chunk.

```
chunks a task reads   [c3][c4][c5]      [c9]
before                 req  req  req     req
after                 [  one request ]  [req]
```

| File | What changes | Lines |
|---|---|---|
| `mcap_io.py` | Plans and makes the ranged reads. Reads a summary's
tail, then its section. | 202 |
| `mcap_chunks.py` | Reads a task's chunks through them. | 70 |
| `mcap_summary.py`, `mcap_file_indexer.py`, `mcap_data_column.py`,
`mcap_datasource_v2.py` | Bound the summaries a listing holds. Estimate
a task's peak from its chunks. | 417 |
| `read_api.py`, `context.py` | The small `read_mcap` items and the
docstring. | 89 |
| `test_mcap_io.py`, `test_mcap_reader.py`, `test_mcap.py` | Reads match
one request per chunk. | 298 |
| `test_mcap_listing.py`, `test_mcap_read_memory.py` | The listing's
byte bound, and the estimate against measured peaks. | 388 |

## Before

- Each chunk was its own request, even when chunks sat next to each
other in the file.
- The listing read summaries a few at a time.
- Each chunk was weighed as 1.2 times the table it produces. Large
chunks and small messages peak well above that.
- `override_num_blocks` was ignored without a word, and a generated
checkpoint id column failed with a misleading error.

## After

- Chunks that sit next to each other are read in one ranged request
(`mcap_io`).
- The listing reads 32 summary tails at once by default. A tail gives
the summary section's exact length, and sections are fetched in file
order while they fit one read task's budget. Threads only fetch, and the
listing parses one summary at a time.
- A chunk weighs its share of the task's table, scaled so that each task
leaves room for its largest chunk's working set and one ranged read.
Every term is sized from the summary.
- A file whose selected channels are all JSON counts each payload byte
as the Python values it decodes into, which hold several times its
bytes.
- `override_num_blocks` or `parallelism` logs a warning on V2, since
tasks are sized by bytes. The warning names the argument the caller
passed and prints the configured byte budget.
- A generated checkpoint id column raises `InvalidCheckpointingConfig`.
The legacy path refuses a `row_id` checkpoint as it refuses
`include_row_id`.
- The `read_mcap` docstring describes the V2 datasource and how its rows
differ from the legacy reader's.

**Measured** (local disk):

| Case | Before | After |
|---|---|---|
| Listing 100 copies of a 10,000-chunk, 20-channel file: summaries held
| 1.3 GiB | 240 MiB, same time |
| Listing 600 copies of a real recording: memory added | 81 MiB | 37 MiB
|
| Read of 20,000 small messages: estimate ÷ measured peak | 0.25 to 1.08
| 1.45 to 1.50 |

**Decisions to confirm:**

- The estimate errs high on small messages, so they pack into more
tasks: 600 copies of a real recording make 5,405 tasks instead of 4,803.
- The per-message sizes are upper bounds for CPython 3.10, which also
covers 3.12.
- The listing task doesn't reserve its summary budget as memory yet.
That needs a change in the shared `plan_list_files_op.py`.

## Subsystems

IO (`_internal/datasource_v2/formats/mcap`), API (`read_api.py`).

## Testing

```
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py
```

249 passed, 4 skipped. pre-commit and pyrefly clean.

Not a duplicate: split out of #67166 at a reviewer's request. AI
assistance (Claude Code) was used to draft this change; I reviewed every
line and ran the tests above locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@richardliaw
richardliaw enabled stacked auto-merge (squash) November 12, 2026 19:34
@richardliaw
richardliaw merged commit e375ec1 into master Nov 12, 2026
6 checks passed
@richardliaw
richardliaw deleted the marwan/mcap-dsv2-io branch November 12, 2026 20:16
richardliaw pushed a commit that referenced this pull request Nov 14, 2026
…its error; read past corrupt MCAP chunks (#67167)

## How to review

Start with `map_transformer.py`: it is the shared map-task change, and
it needs the most care. The MCAP part only reads past a chunk that
fails.

| File | What changes | Lines |
|---|---|---|
| `map_transformer.py`, `map_operator.py`, `task_pool_map_operator.py`,
`actor_pool_map_operator.py`, `task_context.py`, `output_buffer.py`,
`util.py` | Emit the rows a task buffered before an error the job skips.
| 278 |
| `context.py`, `execution-configurations.md` | The `max_errored_blocks`
docs. | 11 |
| `mcap_reader.py`, `mcap_chunks.py`, `mcap_io.py` | Skip a corrupt
chunk and keep reading. | 344 |
| `test_map_transformer.py`, `test_actor_pool_map_operator.py`,
`test_streaming_executor_errored_blocks.py` | The shared map-task
behaviour. | 394 |
| `test_mcap_reader.py`, `test_mcap_io.py`, `test_mcap.py` | Reading
past corrupt chunks. | 459 |

**Problem.**

- When `max_errored_blocks` lets a job skip a failed task, the job also
loses every row the task had already read but not yet emitted. A map
task buffers rows until a block is full, so a task that fails part-way
drops all of its buffered rows with no error.
- In an MCAP read, one corrupt chunk failed its whole read task. The
task's other files, and the rest of the damaged file, were lost.

**Change.**

- A map task emits its buffered rows before it raises an error that the
job will skip. If the error is one the task will retry, it raises
without emitting, so the retry rebuilds the whole block and no row is
read twice.

```
 rows:  [r1 r2 r3 r4 | r5 r6 ✗]
         block 1       buffered, then an error
 before: block 1 kept, r5 r6 lost
 after:  block 1 kept, r5 r6 emitted, then the error is skipped
```

- A fused downstream batcher flushes its short last batch before a
skipped error, so its rows are kept too.
- When errors can be skipped, the MCAP reader skips a corrupt chunk and
reads the rest of its file. Each chunk is compressed on its own, so one
bad chunk costs only its own rows. The task then raises one error naming
every chunk it skipped, so it counts once against `max_errored_blocks`.
The reader keeps one error for all of a file's corrupt chunks, so
skipped chunks add no memory.
- When errors can be skipped, a message whose payload is not valid JSON
costs only its own row. The file's one error counts such messages and
names the first by row id. A wrong message encoding still fails its
file.
- A skipped message does not count against a pushed-down row limit, so
`limit(n)` still returns n rows when the file has them.
- A file that fails part-way keeps the rows built before the failure.
Before, how many it kept depended on where a block ended.
- Only an error that says a chunk does not decompress or parse marks it
as corrupt. Any other error, such as an `OSError` or a bug, raises at
once, so the task's retries see it.
- A frozen `RerunPolicy` on `TaskContext` decides whether an error can
still be rerun. A task can be rerun by `retried_map_errors` while
`_map_task` has attempts left, or by Ray on its own `retry_exceptions`
(for an actor task, built from `actor_task_retry_on_errors`). A
task-pool task is also rerun on a `RaySystemError`. Ray reruns nothing
when a task's `max_retries` or an actor task's `max_task_retries` is 0.
While a rerun can follow, the task raises without a partial block.
`_map_task` updates the policy for each attempt, so its last attempt
emits its rows.
- The policy's default is "may rerun". A `TaskContext` built without
retry settings, as on the shuffle reduce paths, therefore never emits a
partial block.
- A chunk read back for a declaration that does not decode is skipped,
so the chunk being read is not blamed for it. If a schema the task needs
is declared only in that corrupt chunk, the read fails and names the
chunk, rather than returning rows without a schema. A retried file read
starts with no corrupt chunks recorded.
- The `max_errored_blocks` docstring and the execution-configurations
docs page say that rows a task emitted before a skipped error are kept.
- With `max_errored_blocks=0`, nothing changes: the first failure fails
the task.

**Measured** on a folder of recordings with two corrupt chunks and
`max_errored_blocks=1`: the read used to return 0 rows, and now returns
59. The legacy reader returns 40.

**Decisions to confirm:**

- Emit the partial block only for errors the job skips. Retried errors
still rebuild the whole block.

## Related issues

- Stacked on #67166. No other open PR touches `map_transformer.py`'s
error path.

## Additional information

```
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/test_map_transformer.py \
  python/ray/data/tests/test_streaming_executor_errored_blocks.py
```

283 passed, 6 skipped; the shared map tests (17), errored-blocks tests
(10), map operator tests (78), task pool tests (12), operator tests (8),
util tests (45) and actor pool tests (60) pass. pre-commit and pyrefly
clean.

AI assistance (Claude Code) was used to draft this change; I reviewed
every line and ran the tests above locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
eugenepaniot added a commit to eugenepaniot/ray that referenced this pull request Nov 19, 2026
…floop-input

* upstream/master:
  [Data][3/n] update progress bar on lineage reconstruction (ray-project#67366)
  [Data] Enable the Top-N rewrite by default and expose it on DataContext (ray-project#68329)
  [Data] Adaptively skip map-side partial aggregation for high-cardinality keys (ray-project#68118)
  [Core] Spawn specific servers without preexec_fn to fix intermittent startup crashes (ray-project#67947)
  [Serve] Stop the __wrapped__ walk at non-callables (ray-project#67117)
  [Data] Deflake test_read_parquet_v2 by cutting redundant Ray Data executions (ray-project#68318)
  [Data] Retry FooterReader actor deaths with Ray Core and yield footers in listing order (ray-project#67988)
  [Data] Fix order-dependent schema unification when a list<null> arm is present (ray-project#68259)
  [Serve] Make @serve.batch work when wrapped in another decorator (ray-project#68177)
  [serve] Deflake test_metrics_3 - record the batch queue length once a batch is taken (ray-project#68302)
  [serve] Drop completed requests from the multiplexed model ID index (ray-project#68085)
  [Data] Fix flaky test_hudi: read each table once instead of sorting in Ray Data, keep size small (ray-project#68162)
  [Data] Fix flaky test_context: start 2 Ray clusters instead of 5 and keep size small (ray-project#68161)
  [Data] Fix flaky test_kafka: use recent timestamps in test_read_kafka_datetime_partial_range (ray-project#68248)
  [Data] Iceberg DSv2 [2/5]: The listing-to-reading manifest for Iceberg (ray-project#67951)
  [Train] Report training errors when the post-mortem debugger raises a BaseException (ray-project#68317)
  [RLlib] Skip the PPOConfig docstring example that crashes the rllib doctest target (ray-project#68306)
  [Train] Support post-mortem debugging for training function errors (ray-project#68296)
  [train][1/n] Give each health probe its own timeout (ray-project#68191)
  [Data] Fix unifying a fixed-shape tensor array with a lower-ndim variable-shaped one (ray-project#67266)
  [Train] Add a guide to debugging failures and hangs (ray-project#68299)
  [Data] Persist BlockMetadata.num_rows and batch ray.get in _calculate_blocks_rows (ray-project#67662)
  [Data] Vectorize LeRobot delta segment preparation and validate index contiguity (ray-project#67845)
  [Data] Avoid full-DataFrame apply in _AbstractKBinsDiscretizer._transform_pandas (ray-project#67862)
  [doc] Refresh intersphinx inventory snapshots (ray-project#68303)
  [Data] Fix SimpleImputer on categorical columns in append mode and with existing category values (ray-project#67467)
  [Data] Replace O(N) list.pop(0) with deque.popleft() in PushBasedShuffleTaskScheduler (ray-project#67243)
  [Train] Reject an elastic ScalingConfig with max_workers=0 (ray-project#68218)
  [core][move semantics][3/n]Populate task args with is_move for movable refs (ray-project#66803)
  [core][sandbox] Remove the readonly option (ray-project#68088)
  [Data](1/n) Implement the line-chunk reader for CSV and JSONL on DataSource V2 (ray-project#67907)
  [Data] Set MIMALLOC_PURGE_DELAY=0 on random_shuffle to fix the pyarrow 25 OOM regression (ray-project#68151)
  [core] Quote each `uv run` argument in the uv runtime env hook (ray-project#68156)
  [core] Don't let gRPC cancel calls that wait over 30s in the server queue (ray-project#68210)
  [Data] Fix shuffle and join on native tensor columns with pyarrow 25+ (ray-project#68072)
  [serve][llm] Remove decode load progress (ray-project#68258)
  [ci] Bump sglang to 0.5.23 in release tests (ray-project#68250)
  [Core] Build the test auth token file locally so Windows tests can read it (ray-project#68262)
  [image] Bump EFA installer to 1.51.0 in CPU base-extra images (ray-project#67730)
  [serve] Observe the result of rank reassignment reconfigures (ray-project#67923)
  [Serve][Docs] Explain imperative vs declarative deploys and in-place updates from Python (ray-project#68243)
  [Data] (Fault tolerance) Ray Data lineage reconstruction docs (ray-project#68061)
  [RLlib] Add a multi-node footsies APPO memory release test (ray-project#67595)
  [Train] Remove local mode (ray-project#68239)
  [Data] Fix filtered Lance row-count metadata (ray-project#67228)
  [Serve][2/2] Propagate TracingConfig to proxies at startup (ray-project#65437)
  [Data] Fix synthesized path filtering in DataSourceV2 (ray-project#67079)
  [RLlib] Run the BCConfig Tune example one trial at a time to unblock the rllib doctest target (ray-project#68242)
  [Data][Delta DSv2][2/7] List Delta Lake table files from the log (ray-project#67888)
  [Data] Parallelize Parquet and ORC full-schema reads (ray-project#67542)
  [Data] Random Shuffle on Shuffle V2 Map/Reduce Operators (ray-project#68004)
  [Data] Propagate metadata prefetch thread failures to executor (ray-project#68171)
  [core] Send generator consumed-progress updates from io_service_ (ray-project#67160)
  [Data] Add @ray-data-connectors as codeowner for datasource/sinks files (ray-project#68154)
  [data][llm] Keep the prompt in vLLM error rows (ray-project#67665)
  [data][actor-only][3/n] Actor-only backend: operator fusion (ray-project#67699)
  [Data] Read text files on DataSource v2 (ray-project#68063)
  [Autoscaler][KubeRay] Never patch worker group replicas below minReplicas (ray-project#67332)
  [Serve] Show progress of updating (ray-project#67107)
  [core][test] Trigger usage-stats OOM test without exhausting host memory (ray-project#68009)
  [Serve] Remove unused RAY_SERVE_HAPROXY_INGRESS_RETRY_ON and RAY_SERV… (ray-project#68175)
  [Data] Implement lineage tracker garbage collection upon marking operator durable (ray-project#67788)
  [Core] Fix recovery suppression race in ObjectRecoveryManager (ray-project#64974)
  [serve] Pass grpc.so_reuseport to gRPC as an integer (ray-project#66816)
  [Autoscaler][v2] Preserve retry accounting when trimming status history (ray-project#67360)
  [doc] Refresh intersphinx inventory snapshots (ray-project#68069)
  [Doc] Add item to AGENTS.md to update documentation when modifying API-facing code. (ray-project#68204)
  [Train] Preserve `num_boost_round=0` and avoid mutating `callbacks` in legacy LightGBM/XGBoost trainers (ray-project#68145)
  [Train] Raise in prepare_data_loader instead of dropping a custom batch_sampler (ray-project#68201)
  [air] Return None in skip_exceptions when exc or StartTraceback cause is None (ray-project#68172)
  [Train] Fix `checkpoint_manager` to support metrics only reported in a validation function (ray-project#67909)
  [Train] Count elastic workers in resource units for fractional resources (ray-project#68139)
  [Data] Refactor Disk shuffle map to use map Operator (ray-project#67130)
  [core][move semantics][2/n] Change state to moved on consumption and add enforcements (ray-project#66767)
  [Data] Propagate labels to object-store SHUFFLE_V2 map/reduce tasks (ray-project#67135)
  feat(core): add Cambricon MLU resource support (ray-project#66368)
  [Data] Make `zero_copy_batch=False` copy default batches (ray-project#67154)
  [data] Fix Parquet filename suffix detection (ray-project#67156)
  [data] Fix docstring/behavior mismatches in read_json and Dataset.sort (ray-project#66927)
  [Data] Run video_object_detection.ray on T4s with more memory (ray-project#68149)
  [core][sandbox] Let a node override network="public" sandboxes' default resolvers (ray-project#67691)
  [Data] read_mcap: resolve each sample's steps and lay out its row (ray-project#67925)
  [Data] read_mcap: sample read inputs: TopicSpec, steps, match and samples checks (ray-project#67924)
  [Data] Keep a map task's buffered rows when max_errored_blocks skips its error; read past corrupt MCAP chunks (ray-project#67167)
  [core][move semantics][1/n] Adding consume_once flag in actor methods (ray-project#66620)
  [Data] Size the schema pin and the columns Ray adds in the Parquet decoded-size estimate [3/6] (ray-project#67838)
  [Serve][LLM][1/3] Token based autoscaling: track autoscaling workload in KV router (ray-project#67937)
  [Data] Add native direct reads for nonpartitioned Hive tables (ray-project#67114)
  [Train] Count elastic workers in resource units for fractional resources (ray-project#68139)
  [Data] Add `condition` to `Dataset.join` and extract equi-join keys from it (ray-project#67590)
  [data] Add Top-N execution (ray-project#66931)
  [build] Bazel 8.8.1: bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0 (ray-project#66483)
  [build] Use Arena::Create and pin Java 8 bytecode ahead of protobuf 29 / rules_java 8 (ray-project#67720)
  [core][sandbox] Don't join a sandbox's cgroup on every exec (ray-project#67690)
  [Data] Run `training_ingest_regression_test` on Instances that can be Provisioned (ray-project#68110)
  [Data] Run Sort on Disk-based Shuffle under SHUFFLE_V2 (ray-project#67850)
  [core] Deflake tests that race on timing (ray-project#67717)
  [ci] Bump pypa manylinux2014 base image to 2026.10.22-0 for CPython 3.15 (ray-project#67285)
  [Data] Make Mcap Data Column Test Insensitive to Struct Field Order (ray-project#68109)
  [Data] read_mcap V2: cut large files where their log-time order holds (ray-project#67166)
  [core][sandbox] Commit sandbox as new registry image (ray-project#67972)
  [Data] read_mcap V2: read neighbouring chunks in one ranged request (ray-project#67930)
  [train][4/n] Wire WorkerProbes through the HealthManager (ray-project#67915)
  [train][3/n] Add WorkerProbe results and health reports to the HealthState (ray-project#67879)
  [Data] (Data fault tolerance ray-project#6/6) Add a manual release test for lineage reconstruction (ray-project#67519)
  [train][6/n] Act on health decisions in the Ray Train controller (ray-project#67856)
  [Train] Do not start an elastic worker group with zero workers (ray-project#68101)
  [Train] Keep the last checkpoint after a metrics-only report in local mode (ray-project#68070)
  [Data][2/n] Migrate GPU Shuffle to Unified Phase Metrics (ray-project#67359)
  [Data][Chkpt] Fix `ArrowInvalid` when loading large generated-ID checkpoints (ray-project#68065)
  [core][test] Create auth token in test_redis_wrong_password (ray-project#68014)
  [data][actor-only][2/n] Actor-only backend: promote task pools to actor pools (ray-project#67698)
  [Data] Fix typing mismatches in Dataset public API (ray-project#62077)
  [Data] read_mcap V2: carry read tasks' summary records in the manifest (ray-project#67929)
  [Data] Distinguish generated ID collisions with synthesized columns (ray-project#67164)
  [Data] Fix incorrect scale-up in rate-based autoscaler for custom resources (ray-project#66647)
  [Data][Delta DSv2][1/7] Shared Parquet and partitioning fixes for the Delta reader (ray-project#67887)
  [Data] Eliminate redundant aggregation shuffle (ray-project#67198)
  [Data] Raise minimum supported pyarrow to 23.0.0 and test up to 26.0.1 (ray-project#68005)
  [Train] Use the default validation timeout when a report's ValidationTaskConfig has none (ray-project#68045)
  [doc][train] Use the global batch size in the TensorFlow Train examples (ray-project#68006)
  [TPU] Default multi-slice MASTER_ADDR to slice 0 host 0 in SlicePlacementGroup (ray-project#67878)
  [Data] Estimate a Parquet row group's decoded Arrow bytes from its footer [2/6] (ray-project#67837)
  [Data] Predicate pushdown through joins: split conjuncts, derive one-sided OR weakenings (ray-project#67122)
  [Data] Add partition_filters to read_delta and take file sizes from the log (ray-project#67900)
  [Data] Stop writing Unity Catalog vended credentials to the environment (ray-project#67899)
  [Data] Raise an error for Delta tables with column mapping in write_delta (ray-project#67898)
  [Core] Add initial foundational configurations and flags for fork server (ray-project#67356)
  [core][sandbox] Image layers caching (ray-project#67874)
  [Doc][Train] Add device stream batch prefetching section to accelerator guide (ray-project#67779)
  [Train] Use user-defined train function name in training thread name and error stack traces (ray-project#64292)
  [serve][docs] Remove beta label from custom autoscaling policies (ray-project#68046)
  [core] Don't treat a gRPC client with calls in flight as idle (ray-project#67718)
  [train] Fix checkpoint sorting to support nested metrics (ray-project#59130)
  [train][5/n] Run ControllerProbes on the controller's node monitor (ray-project#67671)
  [Train] Rank a checkpoint without the score attribute as worst in min order (ray-project#67901)
  [Core][RuntimeEnv] Stream runtime environment package downloads (ray-project#67184)
  [core] Deflake `_run_hanging_command` tests (ray-project#67461)
  [Data] Add Gravitino Iceberg REST catalog provider (ray-project#67007)
  [Data] Fix mixed-root partition filtering in Parquet and ORC V2 (ray-project#67067)
  [core] Keep absl logging at ERROR unless GRPC_VERBOSITY is set (ray-project#67719)
  [Data][1/n]Add Shuffle Metrics Unification Instrumentation (ray-project#66968)
  [ci][release] Stop publishing Python 3.10 wheels and images (ray-project#67722)
  [serve] Copy the ASGI lifespan state into each request scope (ray-project#67844)
  [Serve][LLM][4/5] Support scale-to-zero application routing (ray-project#67087)
  [core][sandbox] Let GPU sandboxes request NVIDIA profiling capabilities (ray-project#67974)
  [Serve][LLM] Exclude LLMRouter from dashboard QPS (ray-project#67873)
  [Data] Read the Parquet SizeStatistics PyArrow doesn't expose [1/6] (ray-project#67836)
  Add perf metrics for 2.60.0 (ray-project#67727)
  [Observability / Train] Add Grafana Annotation support with Ray Train usage (ray-project#66958)

Signed-off-by: Eugene Paniot <e.paniot@gmail.com>

# Conflicts:
#	python/ray/data/_internal/planner/plan_udf_map_op.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants