Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for reading MCAP files at different granularities ('message', 'window', 'topic', and 'file') in the V2 datasource, along with options like WindowSpec and VideoOptions to handle window slicing and video keyframe/lead-in detection. The review feedback highlights critical Out-Of-Memory (OOM) risks in _topic_tables and _file_tables where entire topics or files are loaded into memory before checking the row size limit. It is recommended to incrementally iterate over the messages and check the accumulated payload size to raise a size-limit error early.
d943ab6 to
3da2f29
Compare
0414b77 to
ff23fad
Compare
3da2f29 to
f606969
Compare
ff23fad to
599f8bd
Compare
3cb420d to
6191ee4
Compare
599f8bd to
ddda933
Compare
6191ee4 to
baf5f6e
Compare
ddda933 to
1442166
Compare
baf5f6e to
19b765f
Compare
1442166 to
64590f5
Compare
19b765f to
e2dd57a
Compare
64590f5 to
23ab24f
Compare
3ff5f5f to
7bd7106
Compare
7bd7106 to
9fac625
Compare
9fac625 to
6bb6dfc
Compare
cd96b2e to
93ad58e
Compare
6bb6dfc to
0566127
Compare
93ad58e to
7eb3747
Compare
0566127 to
ed446a0
Compare
ed446a0 to
1db9e8f
Compare
7eb3747 to
e64d8a6
Compare
1db9e8f to
1a51115
Compare
e64d8a6 to
8137f9b
Compare
1a51115 to
d2433c0
Compare
d2433c0 to
c523f7f
Compare
8137f9b to
9617b46
Compare
c523f7f to
0d8e578
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4f583e0. Configure here.
…, row_id Problem: - read_mcap reads each file in one task, so parallelism is capped by the number of files. - Every file is opened, even when its summary shows it holds nothing the read selects. - Rows have no stable id, so a checkpointed pipeline cannot resume. Change: read_mcap runs on Datasource V2 when DataContext.use_datasource_v2 is set (the default). The old reader stays as the fallback. file summaries -> one listing row per chunk -> ~128 MiB read tasks - Listing reads each file's summary and emits one row per chunk. A file that cannot match the selection is dropped before any data is read. - Chunks are packed into read tasks of about 128 MiB uncompressed (RAY_DATA_MCAP_BIN_PACKING_BYTES). A big file spans many tasks; small files share one. - Each task seeks straight to its chunks. The new log_time_order argument (default True) merges them by log time. - row_id names a message by its file, chunk offset and position. It is the same however the read is split. Enable it with include_row_id=True; a row_id checkpoint turns it on automatically. - Column pruning and limits are pushed into the reader. - data holds decoded JSON only when every selected channel is JSON. Otherwise it holds bytes. mcap_data_column decides this once, from the sampled files' summaries. - Columns match the old reader, plus channel_metadata (a map). schema_data is dictionary-encoded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…d-in Problem: one message per row is the wrong unit for most uses. A labeling job wants clips, an export wants one stream, episode features want one recording. Building those from message rows needs a shuffle, and the result often cannot be decoded on its own. Change: new read_granularity argument (V2 only). - message (default): one message per row, as before. - window: one row per time window of a file, set by WindowSpec (length, stride, anchor). Each window is emitted once, by the task that owns the chunk its start falls in. - topic: one row per topic of a file. - file: one row per file. Window, topic and file rows hold their messages as parallel lists (log_time, topic, data, ...), plus each channel's schema, so a row decodes on its own. Payloads stay encoded. A topic or file row over RAY_DATA_MCAP_MAX_ROW_BYTES (1 GiB) fails with a hint to use windows. Video lead-in: a window also carries, for each video topic in it, the frames back to the last keyframe, so the clip decodes on its own. A topic row of a video topic starts on a keyframe too. frames K p p p K p p | p p K p p | lead-in [-----) window [-----------) - A topic is video if its schema is a known video schema (Foxglove CompressedVideo or CompressedImage, sensor_msgs CompressedImage), or if the new video_topics argument lists it. - The codec comes from the message's format field, or else its bytes. - The look-back is capped at 10 s (RAY_DATA_MCAP_MAX_LEAD_IN_S). Row ids name the file, the window or topic, and a digest of the read options, so a checkpoint never mixes rows of different reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
|
Superseded by #67189: window rows were replaced by sample rows ( |

Description
Part 2 of 3 of the
read_mcapredesign. Stacked on #66654: review the top commit only.Problem. A message row is the wrong unit for most MCAP work:
Users rebuild these with a shuffle today. Blocks cut mid-GOP cannot be decoded on their own.
Change. A new
read_granularityoption chooses what one row is:read_granularitymessage(default)windowtopicfileHow a window row is built:
num_lead_incounts them.Row contents:
topic,log_time,data, ...). Entry i of each list is the same message.channelscolumn describes each channel, so a row decodes on its own. Payloads stay encoded.topicorfilerow over 1 GiB fails early and points towindow(RAY_DATA_MCAP_MAX_ROW_BYTES).Video topics:
CompressedVideoandCompressedImage(ROS 1, ROS 2, protobuf names) and ROS'ssensor_msgsCompressedImage.video_topicsoption lists it:formatfield. A listed topic, or a missing or unknownformat, falls back to the bytes: JPEG, PNG, H.264, H.265, VP9, AV1.video_topicsneeds the V2 datasource. Withtopics, every entry must be one of them.Code: window, topic and file rows live in
mcap_coarse_rows.py. The lead-in rules are inmcap_lead_in.pyand the row layout inmcap_coarse_layout.py.Decisions to confirm:
video_topicsmakes a topic video, never the bytes. A camera under a custom schema name must be listed.read_granularityis shared withread_lerobot, where it groups episodes into tasks.RAY_DATA_MCAP_MAX_LEAD_IN_S).Related issues
read_granularity="file"provides, and to [Data] MCAPDatasource: batched (streaming) message reads to reduce peak memory and enable intra-file pipelining #65641.read_mcap.Additional information
126 unit tests passed.
test_mcap.py: 53 passed, 5 skipped. pre-commit and pyrefly clean.AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code