Skip to content

[Data] read_mcap: resume a sample read without reading finished samples - #67191

Closed
marwan116 wants to merge 2 commits into
marwan/mcap-dsv2-samplesfrom
marwan/mcap-dsv2-samples-resume
Closed

marwan116 wants to merge 2 commits into
marwan/mcap-dsv2-samplesfrom
marwan/mcap-dsv2-samples-resume

Conversation

@marwan116

Copy link
Copy Markdown
Contributor

Description

Part of the read_mcap redesign. Stacked on #67189 (#66654 → #67166 → #67167 → #67189 → this PR). It changes the checkpoint planner and ReadFiles, so it is kept apart from the reader.

Problem. A sample read resumed from a checkpoint keyed on a sample column (CheckpointConfig(id_column="sample_id")) went wrong three ways:

  • With every sample already finished, the rerun crashed with AssertionError in FileManifest. The checkpoint filter left an empty block without the manifest columns, and ReadFiles failed on it.
  • With samples derived from another read, such as a flat_map over a Parquet list of episodes, the rerun failed with KeyError: 'sample_id'. Every read on the path got a checkpoint filter, including the episode listing, whose rows have no sample_id.
  • With samples from from_items, every sample was read again, and the finished ones were dropped only after the read.

Change.

  • ReadFiles skips a manifest block with no rows.
  • The planner puts a checkpoint filter only on the read nearest the sink on each path. A read further up feeds another read, and its rows need not carry the id.
  • ReadFiles(checkpoint_on_input=True) filters its input instead of its output. A sample read sets it, because each of its rows comes from one sample and keeps the sample's columns.
before:  samples -> manifest -> ReadFilesMCAP -> CheckpointFilter -> Write
after:   samples -> manifest -> CheckpointFilter -> ReadFilesMCAP -> Write

So finished samples are dropped before their files are opened.

Decisions to confirm:

  • checkpoint_on_input as an explicit flag on ReadFiles, rather than inferring it from a ReadFiles that has no ListFiles upstream.

Related issues

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap_resume.py \
  python/ray/data/tests/test_checkpoint.py python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py

The new resume tests pass (6); 4 of them fail without this change. test_checkpoint.py: 74 passed, 17 skipped. The MCAP suite: 388 passed, 14 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

marwan116 and others added 2 commits October 20, 2026 06:14
Problem:
- Robot learning reads examples, not messages: a clip of a recording,
  or a moment inside one, with each topic read over its own window.
- Message rows need a shuffle or a join to rebuild each example.

Change:
- read_mcap(samples=ds, topics={topic: TopicSpec(...)}) reads one row
  per sample: its own columns, then each topic's data:, times: and
  valid: lists, one entry per step.
- TopicSpec(steps, match=None, pad="edge"). steps.before/after place
  steps 1/fps apart around the anchor; steps.last/next/every take the
  recorded messages themselves. match.nearest() fills each time step.
  The built-ins live in ray.data.datasource.mcap.steps and .match.
- The samples dataset feeds ReadFiles as manifest rows, about 64
  samples per task (RAY_DATA_MCAP_SAMPLES_PER_TASK). A task reads only
  the chunks its steps need, nearest first.
- A checkpoint keyed on row_id is refused for a sample read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A rerun of a sample read whose samples are all finished crashed: the
  checkpoint filter left an empty manifest block without the manifest
  columns, and ReadFiles failed on it.
- Samples derived from another read, such as a flat_map over a list of
  episodes, failed with a KeyError: every read on the path got a
  checkpoint filter, including the listing, whose rows lack the id.
- Samples from from_items were read again and only dropped after the
  read.

Change:
- ReadFiles skips a manifest block with no rows.
- The planner puts a checkpoint filter only on the read nearest the
  sink on each path.
- ReadFiles(checkpoint_on_input=True) filters its input instead of its
  output. A sample read sets it, so finished samples are dropped before
  their files are opened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 234e217. Configure here.

@marwan116

Copy link
Copy Markdown
Contributor Author

Folded into #67189: resuming a sample read is now part of the sample-rows PR, to keep the stack to four PRs.

@marwan116 marwan116 closed this Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant