Skip to content

[Data] read_mcap: window, topic and file granularities with video lead-in - #66655

Closed
marwan116 wants to merge 2 commits into
marwan/mcap-dsv2from
marwan/mcap-dsv2-granularity
Closed

marwan116 wants to merge 2 commits into
marwan/mcap-dsv2from
marwan/mcap-dsv2-granularity

Conversation

@marwan116

@marwan116 marwan116 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 2 of 3 of the read_mcap redesign. Stacked on #66654: review the top commit only.

Problem. A message row is the wrong unit for most MCAP work:

  • A VLM labeling job wants decodable 10 s clips.
  • An export wants one camera stream per recording.
  • Episode features want one row per recording.

Users rebuild these with a shuffle today. Blocks cut mid-GOP cannot be decoded on their own.

Change. A new read_granularity option chooses what one row is:

read_granularity one row is
message (default) one message, as today
window all selected messages in one time window of one file
topic all messages of one topic in one file
file all selected messages of one file
ds = ray.data.read_mcap(
    "s3://bucket/recordings/",
    topics=["/cam_front/compressed", "/imu"],
    read_granularity="window",
    window=WindowSpec(length_s=10),  # also stride_s, anchor, drop_partial
)

How a window row is built:

chunks    | c0      | c1      | c2      | c3      |
tasks     |<---- task 1 ----->|<---- task 2 ----->|
frames     K . . . . K . . . . K . . . . K . . . .
window W                 [============)
lead-in              [===)
  • W starts in c1, so task 1 emits it. Task 1 reads on into c2 until W ends.
  • Every window is emitted by exactly one task.
  • The lead-in holds the frames from the keyframe (K) before W. num_lead_in counts them.

Row contents:

  • Coarse rows hold their messages as parallel lists (topic, log_time, data, ...). Entry i of each list is the same message.
  • A channels column describes each channel, so a row decodes on its own. Payloads stay encoded.
  • A topic or file row over 1 GiB fails early and points to window (RAY_DATA_MCAP_MAX_ROW_BYTES).

Video topics:

  • A topic is video when its schema name is a known video schema: Foxglove's CompressedVideo and CompressedImage (ROS 1, ROS 2, protobuf names) and ROS's sensor_msgs CompressedImage.
  • Any other topic is video only when the new video_topics option lists it:
ds = ray.data.read_mcap(
    "s3://bucket/recordings/",
    read_granularity="window",
    window=WindowSpec(length_s=10),
    video_topics=["/cam_front/h264"],  # custom schema name
)
  • A known schema's codec is read from the message's format field. A listed topic, or a missing or unknown format, falls back to the bytes: JPEG, PNG, H.264, H.265, VP9, AV1.
  • Keyframes are always found in the bytes.
  • video_topics needs the V2 datasource. With topics, every entry must be one of them.

Code: window, topic and file rows live in mcap_coarse_rows.py. The lead-in rules are in mcap_lead_in.py and the row layout in mcap_coarse_layout.py.

Decisions to confirm:

  • Only a known schema name or video_topics makes a topic video, never the bytes. A camera under a custom schema name must be listed.
  • The name read_granularity is shared with read_lerobot, where it groups episodes into tasks.
  • The keyframe search looks back at most 10 s (RAY_DATA_MCAP_MAX_LEAD_IN_S).
  • If a video codec cannot be parsed, the row carries the whole 10 s as lead-in and logs a warning. The read does not fail.

Related issues

Additional information

pytest python/ray/data/tests/unit/datasource_v2/test_mcap_*.py
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py

126 unit tests passed. test_mcap.py: 53 passed, 5 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116
marwan116 requested a review from a team as a code owner October 2, 2026 04:26
@marwan116 marwan116 added the data Ray Data-related issues label Oct 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for reading MCAP files at different granularities ('message', 'window', 'topic', and 'file') in the V2 datasource, along with options like WindowSpec and VideoOptions to handle window slicing and video keyframe/lead-in detection. The review feedback highlights critical Out-Of-Memory (OOM) risks in _topic_tables and _file_tables where entire topics or files are loaded into memory before checking the row size limit. It is recommended to incrementally iterate over the messages and check the accumulated payload size to raise a size-limit error early.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2 branch 2 times, most recently from 3cb420d to 6191ee4 Compare October 5, 2026 00:36
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 599f8bd to ddda933 Compare October 5, 2026 00:36

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from ddda933 to 1442166 Compare October 5, 2026 02:19
@marwan116
marwan116 added this pull request to stack #66712 October 5, 2026 02:25
@marwan116 marwan116 self-assigned this Oct 5, 2026
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 1442166 to 64590f5 Compare October 5, 2026 02:32

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 64590f5 to 23ab24f Compare October 5, 2026 02:44

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 3ff5f5f to 7bd7106 Compare October 6, 2026 06:04

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 7bd7106 to 9fac625 Compare October 6, 2026 06:33

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 9fac625 to 6bb6dfc Compare October 6, 2026 06:46
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 6bb6dfc to 0566127 Compare October 6, 2026 16:21

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from 0566127 to ed446a0 Compare October 6, 2026 16:55

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from d2433c0 to c523f7f Compare October 7, 2026 01:01
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-granularity branch from c523f7f to 0d8e578 Compare October 7, 2026 01:31

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4f583e0. Configure here.

marwan116 and others added 2 commits October 7, 2026 15:38
…, row_id

Problem:
- read_mcap reads each file in one task, so parallelism is capped by the
  number of files.
- Every file is opened, even when its summary shows it holds nothing the
  read selects.
- Rows have no stable id, so a checkpointed pipeline cannot resume.

Change: read_mcap runs on Datasource V2 when
DataContext.use_datasource_v2 is set (the default). The old reader stays
as the fallback.

  file summaries -> one listing row per chunk -> ~128 MiB read tasks

- Listing reads each file's summary and emits one row per chunk. A file
  that cannot match the selection is dropped before any data is read.
- Chunks are packed into read tasks of about 128 MiB uncompressed
  (RAY_DATA_MCAP_BIN_PACKING_BYTES). A big file spans many tasks; small
  files share one.
- Each task seeks straight to its chunks. The new log_time_order
  argument (default True) merges them by log time.
- row_id names a message by its file, chunk offset and position. It is
  the same however the read is split. Enable it with
  include_row_id=True; a row_id checkpoint turns it on automatically.
- Column pruning and limits are pushed into the reader.
- data holds decoded JSON only when every selected channel is JSON.
  Otherwise it holds bytes. mcap_data_column decides this once, from
  the sampled files' summaries.
- Columns match the old reader, plus channel_metadata (a map).
  schema_data is dictionary-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…d-in

Problem: one message per row is the wrong unit for most uses. A labeling
job wants clips, an export wants one stream, episode features want one
recording. Building those from message rows needs a shuffle, and the
result often cannot be decoded on its own.

Change: new read_granularity argument (V2 only).
- message (default): one message per row, as before.
- window: one row per time window of a file, set by WindowSpec (length,
  stride, anchor). Each window is emitted once, by the task that owns
  the chunk its start falls in.
- topic: one row per topic of a file.
- file: one row per file.

Window, topic and file rows hold their messages as parallel lists
(log_time, topic, data, ...), plus each channel's schema, so a row
decodes on its own. Payloads stay encoded. A topic or file row over
RAY_DATA_MCAP_MAX_ROW_BYTES (1 GiB) fails with a hint to use windows.

Video lead-in: a window also carries, for each video topic in it, the
frames back to the last keyframe, so the clip decodes on its own. A
topic row of a video topic starts on a keyframe too.

  frames    K p p p K p p | p p K p p |
  lead-in           [-----)
  window                  [-----------)

- A topic is video if its schema is a known video schema (Foxglove
  CompressedVideo or CompressedImage, sensor_msgs CompressedImage), or
  if the new video_topics argument lists it.
- The codec comes from the message's format field, or else its bytes.
- The look-back is capped at 10 s (RAY_DATA_MCAP_MAX_LEAD_IN_S).

Row ids name the file, the window or topic, and a digest of the read
options, so a checkpoint never mixes rows of different reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116

Copy link
Copy Markdown
Contributor Author

Superseded by #67189: window rows were replaced by sample rows (read_mcap(samples=..., topics={topic: TopicSpec(steps, match)})), and topic and file rows left the first release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant