Skip to content

[Data] read_mcap: read training and labeling samples, one row per sample - #67189

Open
marwan116 wants to merge 2 commits into
masterfrom
marwan/mcap-dsv2-samples
Open

marwan116 wants to merge 2 commits into
masterfrom
marwan/mcap-dsv2-samples

Conversation

@marwan116

@marwan116 marwan116 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

How to review

Start with _read_mcap_samples in read_api.py: it is the whole path from a samples dataset to ReadFiles. The planner and checkpoint changes are small and only matter for resume.

File What changes Lines
read_api.py read_mcap(samples=...): checks, task sizing, the manifest map. 433
mcap_sample_tasks.py Samples per task, from the bytes the first samples read. 130
planner.py, plan_read_files_op.py, checkpoint/plan_read_files_op.py, read_operator.py Resume: filter finished samples before the read. 113
test_mcap.py, test_mcap_resume.py, test_mcap_sample_tasks.py, BUILD.bazel End-to-end reads, resume, and task sizing. 856

Problem.

  • Robot learning reads examples, not messages. An example is a clip of a recording, or a moment inside one, with each topic read over its own window and rate.
  • With message rows, users rebuild each example with a shuffle or a join. A grid of fixed windows cannot express this: clips start anywhere and run any length, each topic needs its own span, and training draws new anchors every epoch.

Change. A table of samples goes in; one row per sample comes out.

from ray.data.datasource.mcap import TopicSpec, match, steps

# Training: inputs look back from an anchor, labels look forward.
rows = ray.data.read_mcap(
    samples=samples,  # files, clip_start, clip_end, anchor, plus any columns to carry
    topics={
        "/cam/wrist": TopicSpec(steps.before(0.5, fps=10)),                     # 5 frames up to the anchor
        "/can/speed": TopicSpec(steps.before(1, fps=5), match.backward()),      # last value at each step
        "/lidar":     TopicSpec(steps.before(0.5, fps=10), match.all_in_step()),# every packet of each step
        "/action":    TopicSpec(steps.after(2, fps=2, within="recording")),     # may read past the clip
    },
)

# Captioning: whole clips of any length, no anchor.
rows = ray.data.read_mcap(samples=clips, topics={"/cam/front": TopicSpec(steps.across(8))})
# Each row keeps the sample's columns and adds data:<topic>, times:<topic>, is_pad:<topic>.
  • TopicSpec, steps and match come from the PRs below; this PR connects them to read_mcap.
  • The samples feed ReadFiles as its manifest; planning lists no files. A task takes as many samples as read about RAY_DATA_MCAP_BIN_PACKING_BYTES of chunks, estimated from the first 256. The estimate uses the reader's reach rule (TopicReach, [Data] read_mcap: read sample rows from a file's chunks #67927). Samples whose windows meet or touch share one coalesced read (see the reader PR below).
  • Samples that name local:// files read on the driver's node, and fail at read_mcap under Ray Client, as a read of local:// paths does.
  • Rows do not keep the samples' order, even with preserve_order. The docstring says so, and says to match rows to samples by joining on a sample column such as sample_id.
  • Resume: a checkpoint keyed on a sample column skips finished samples before they are cut into read tasks, so their files are never opened. ReadFiles(checkpoint_on_input=True) asks for this. The planner puts the filter only on the read nearest the sink, ahead of the StreamingRepartition that cuts the read's input into tasks. ReadFiles skips empty manifest blocks. A resumed run plans:
sample_manifest -> CheckpointFilter -> StreamingRepartition -> ReadFilesMCAP
  • read_mcap refuses a checkpoint whose id_column the samples lack, such as row_id. The error suggests a column that names each sample.

Measured on S3: 1,024 one-minute recordings (115 GB), 2 s back-to-back samples against the same windows read by the earlier WindowSpec design, 128 CPUs, raw payloads:

WindowSpec samples
rows/s 871 785
read CPU-s 472 449
peak worker RSS 455 MB 425 MB

The remaining gap is planning, mostly the probe that sizes the tasks. Planning runs the samples once: their schema comes from their plan when it is known, else from the samples probed for task size.

Decisions to confirm:

  • checkpoint_on_input as an explicit ReadFiles flag (core Ray Data change; reviewers for the checkpoint planner please weigh in).

Related issues

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/datasource/test_mcap_resume.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py \
  python/ray/data/tests/unit/datasource_v2/test_file_partitioners.py

483 passed, 23 skipped; test_checkpoint.py: 97 passed, 17 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from e5b773e to 0cc15fb Compare October 20, 2026 15:56
@marwan116 marwan116 changed the title [Data] read_mcap: read training samples, one row per sample [Data] read_mcap: read training and labeling samples, one row per sample Oct 20, 2026
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread python/ray/data/read_api.py Resolved
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from 0cc15fb to d53ea8b Compare October 20, 2026 16:26
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@marwan116
marwan116 added this pull request to stack #67196 October 20, 2026 16:31
@marwan116 marwan116 added the data Ray Data-related issues label Oct 20, 2026
@marwan116
marwan116 marked this pull request as ready for review October 20, 2026 16:36
@marwan116
marwan116 requested a review from a team as a code owner October 20, 2026 16:36

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the capability to read training samples from MCAP files via read_mcap(samples=...), adding new modules for sample row processing, task sizing, and topic specifications, alongside corresponding unit and integration tests. The code review identified several critical compatibility and runtime issues: using the key argument in bisect.bisect_right will break on Python 3.9; calling files.value_lengths() and anchor.fill_null(0) will raise AttributeErrors in PyArrow; and accessing self._samples.nbytes is incompatible with PyArrow versions older than 14.0.0. Code suggestions were provided to resolve these issues using backward-compatible PyArrow and standard library alternatives.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from d53ea8b to a334086 Compare October 20, 2026 16:51
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from a334086 to dc72fcb Compare October 20, 2026 17:14
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch 2 times, most recently from 11609d7 to 36c32a1 Compare October 29, 2026 14:16

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread python/ray/data/read_api.py Resolved
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from aa1215b to 921ed60 Compare November 9, 2026 05:45

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread python/ray/data/read_api.py Resolved
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from 921ed60 to 58c4eed Compare November 9, 2026 15:18
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from 58c4eed to da721f4 Compare November 9, 2026 17:56
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-samples branch from da721f4 to c38003f Compare November 9, 2026 19:30
@marwan116
marwan116 removed this pull request from stack #67931 November 9, 2026 19:35
@marwan116
marwan116 added this pull request to stack #67983 November 9, 2026 19:35

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c38003f. Configure here.

Comment thread python/ray/data/read_api.py Resolved
)
if not_specs:
raise ValueError(f"topics must map each topic to a TopicSpec: {not_specs}")
probe = _first_samples(samples, PROBE_SAMPLES)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the samples dataset ends in a sort, as the docstring suggests ("sorted by file and anchor"), we sort twice: once at planning to probe the first 256 samples, and again in the read. A sort has to see every sample before it outputs any, so this is expensive.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dfd9423. The docstring now says to sort the samples, then call .materialize(). Planning reads the first samples to size the read tasks, so an unmaterialized sort would run twice.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK for this PR. Since #67991 reworks the sizing anyway, could it also stop running the samples at planning? With only the docstring, anyone who skips .materialize() still runs the samples twice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, added to #67991.

@richardliaw
richardliaw force-pushed the marwan/mcap-dsv2-samples branch 2 times, most recently from 7ce43bd to 470185c Compare November 12, 2026 20:16
marwan116 and others added 2 commits November 13, 2026 18:02
Problem:
- A sample row needs each topic's messages around its anchor, read
  from the recording's chunks. Reading each sample on its own reads
  the same chunks many times when samples lie close together.
- The chunks a task holds at once must stay within its budget.

Change:
- mcap_sample_rows: a read task takes its samples file by file, in
  anchor order, and groups samples whose windows meet into runs. A
  run ends before its chunks would hold more than a read task's
  budget (RAY_DATA_MCAP_BIN_PACKING_BYTES). Each run reads the chunks
  its topics' steps reach once (ChunkTimeline), and forgets chunks
  the next run does not need.
- A file without a chunk index is scanned once (WholeFileTimeline).
  The scan keeps only what steps can take: the runs' spans, the
  nearest messages around each span, and the first and last messages
  inside each sample's bounds.
- A corrupt chunk fails the read task. Ray's max_errored_blocks then
  decides whether the task's block is dropped or the read fails.
- SampleRows builds one row per sample: the sample's columns, then each
  topic's data:, times: and is_pad: lists. A row over
  RAY_DATA_MCAP_MAX_ROW_BYTES fails the read, naming the sample.
- mcap_summary.topic_chunks: the chunks that may hold a topic, one rule
  for the reader and, later, for sizing tasks.
- MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...) sets
  up a sample read: its schema is the sample row's, and its reader
  builds sample rows from manifest rows that name each sample's file.
  read_mcap wires it up in the next PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Robot learning reads examples, not messages: a clip of a recording,
  or a moment inside one, with each topic read over its own window.
- The stack below adds the spec, step resolution, sample checks and
  the sample reader; nothing reaches them from read_mcap yet.

Change:
- read_mcap(samples=ds, topics={topic: TopicSpec(...)}) reads one row
  per sample. The samples feed ReadFiles as its manifest; planning
  lists no files. Options that only a read of paths takes are refused.
- Planning runs the samples once: their schema comes from their plan
  when it is known, else from the samples probed for task size.
- A task takes as many samples as read about
  RAY_DATA_MCAP_BIN_PACKING_BYTES of chunks, estimated from the first
  256 samples (mcap_sample_tasks).
- A checkpoint keyed on a sample column skips finished samples before
  their files are opened: ReadFiles(checkpoint_on_input=True), a
  checkpoint filter only at the read nearest the sink, and ReadFiles
  skipping empty manifest blocks. A checkpoint keyed on a column the
  samples lack, such as row_id, is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants