Skip to content

[Data] read_mcap: read sample rows from a file's chunks - #67927

Merged
richardliaw merged 1 commit into
masterfrom
marwan/mcap-dsv2-sample-reader
Nov 20, 2026
Merged

richardliaw merged 1 commit into
masterfrom
marwan/mcap-dsv2-sample-reader

Conversation

@marwan116

@marwan116 marwan116 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Reads sample rows from a recording's chunks: the datasource and reader side of a sample read. read_mcap wires it up in the next PR.

How to review

Start with SampleRows in mcap_sample_rows.py: it is the read task's loop. Then read ChunkTimeline, which decides which chunks to read and when to let them go.

File What changes Lines
mcap_sample_rows.py New. Groups samples into runs, reads their chunks once, builds the rows. 685
mcap_reader.py, mcap_chunks.py, mcap_scanner.py, mcap_datasource_v2.py Route a sample read to SampleRows. 277
mcap_summary.py topic_chunks: which chunks may hold a topic. 20
test_mcap_sample_rows.py, mcap_testing.py, conftest.py Rows match the step rules on real recordings. 637

Before

The pieces below resolve steps and lay out rows, but no reader reads a sample's messages from a file.

After

MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...) sets up a sample read. Its read tasks take manifest rows that name each sample's file and build one row per sample.

flowchart LR
    M[manifest rows: one per sample] --> G[group by file, anchor order]
    G --> R[runs: samples whose windows meet]
    R --> T[ChunkTimeline: read each chunk in reach once]
    T --> S[resolve_steps per topic]
    S --> B[SampleRows: one row per sample]
Loading
one file's samples, by anchor      a1   a2   a3              a4
their windows, all topics       [-----]
                                     [-----]
                                          [-----]          [-----]
runs                            [   run 1          ]       [run 2]
chunks read                     c3   c4   c5   c6          c9  c10
  • Samples whose windows meet share one read of their chunks, and chunks the next run does not need are let go.
  • A run ends before its chunks would hold more than a read task's budget (RAY_DATA_MCAP_BIN_PACKING_BYTES).
  • One reach rule, TopicReach, estimates from the file's statistics how far last and next steps reach. The reader groups samples into runs by it, so nearby steps.last samples share their chunk reads. The task sizing in [Data] read_mcap: read training and labeling samples, one row per sample #67189 uses the same rule.
  • A file without a chunk index is scanned once (WholeFileTimeline). The scan keeps only what steps can take: the runs' spans, the nearest messages around each span, and the first and last messages inside each sample's bounds.
  • A corrupt chunk fails the read task. With max_errored_blocks set, Ray then drops that task's block instead of failing the read.
  • mcap_summary.topic_chunks is the one rule for which chunks may hold a topic.
  • A row over RAY_DATA_MCAP_MAX_ROW_BYTES fails the read, naming the sample.
  • Inside the module, a named tuple holds each message a row keeps, a _Timeline protocol covers both timelines, and the tuning values are named constants.

Subsystems

IO (_internal/datasource_v2/formats/mcap).

Testing

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/unit/datasource_v2/test_mcap_*.py

444 passed, 6 skipped (the MCAP unit and datasource tests at this level). pre-commit and pyrefly clean.

Not a duplicate: split out of #67189 at a reviewer's request. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116
marwan116 requested a review from a team as a code owner November 7, 2026 01:05

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for sample reads in the MCAP datasource, allowing users to read specific topic steps around sample anchors. It adds a new mcap_sample_rows.py module to manage grouping, timeline reading, and building Arrow tables for sample rows, along with corresponding updates to the MCAP datasource, reader, scanner, and unit tests. The feedback highlights a compatibility issue in mcap_sample_rows.py where the key parameter in bisect.bisect_right is used (unsupported in Python 3.9), and recommends replacing an assert statement in MCAPDatasourceV2 with a standard TypeError for robust public API argument validation.

@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-reader branch from dfff3b3 to 06c6122 Compare November 7, 2026 01:37
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-reader branch from 06c6122 to 02d6e87 Compare November 7, 2026 01:43
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-reader branch from 02d6e87 to 9e0511d Compare November 7, 2026 02:17
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@ray-gardener ray-gardener Bot added the data Ray Data-related issues label Nov 7, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-reader branch from 7fbeff9 to 3268d20 Compare November 9, 2026 19:30
@marwan116
marwan116 removed this pull request from stack #67931 November 9, 2026 19:35
@marwan116
marwan116 added this pull request to stack #67983 November 9, 2026 19:35

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-reader branch from 3268d20 to 2b772e0 Compare November 9, 2026 21:46

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2b772e0. Configure here.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-reader branch from e2f6a58 to d7a150c Compare November 11, 2026 19:43
@richardliaw
richardliaw force-pushed the marwan/mcap-dsv2-sample-reader branch 3 times, most recently from 7ae336f to c1d2a73 Compare November 12, 2026 20:16
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-reader branch from c1d2a73 to d7d4c80 Compare November 13, 2026 22:50
@richardliaw
richardliaw force-pushed the marwan/mcap-dsv2-sample-reader branch 2 times, most recently from c2a805c to ab44a58 Compare November 14, 2026 01:18
Base automatically changed from marwan/mcap-dsv2-sample-steps to master November 14, 2026 02:02
Problem:
- A sample row needs each topic's messages around its anchor, read
  from the recording's chunks. Reading each sample on its own reads
  the same chunks many times when samples lie close together.
- The chunks a task holds at once must stay within its budget.

Change:
- mcap_sample_rows: a read task takes its samples file by file, in
  anchor order, and groups samples whose windows meet into runs. A
  run ends before its chunks would hold more than a read task's
  budget (RAY_DATA_MCAP_BIN_PACKING_BYTES). Each run reads the chunks
  its topics' steps reach once (ChunkTimeline), and forgets chunks
  the next run does not need.
- A file without a chunk index is scanned once (WholeFileTimeline).
  The scan keeps only what steps can take: the runs' spans, the
  nearest messages around each span, and the first and last messages
  inside each sample's bounds.
- A corrupt chunk fails the read task. Ray's max_errored_blocks then
  decides whether the task's block is dropped or the read fails.
- SampleRows builds one row per sample: the sample's columns, then each
  topic's data:, times: and is_pad: lists. A row over
  RAY_DATA_MCAP_MAX_ROW_BYTES fails the read, naming the sample.
- mcap_summary.topic_chunks: the chunks that may hold a topic, one rule
  for the reader and, later, for sizing tasks.
- MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...) sets
  up a sample read: its schema is the sample row's, and its reader
  builds sample rows from manifest rows that name each sample's file.
  read_mcap wires it up in the next PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@richardliaw
richardliaw force-pushed the marwan/mcap-dsv2-sample-reader branch from ab44a58 to 1c268cd Compare November 14, 2026 02:02
@richardliaw
richardliaw merged commit 9f47f22 into master Nov 20, 2026
6 checks passed
@richardliaw
richardliaw deleted the marwan/mcap-dsv2-sample-reader branch November 20, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants