Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds support for reading MCAP Attachment and Metadata records as rows using the V2 datasource, introducing the 'attachment' and 'metadata' read granularities. It includes schema definition, record indexing, scanning, and reading capabilities, along with comprehensive unit tests. The review feedback highlights a critical issue in mcap_reader.py where read_metadata reports inflated, file-wide row counts instead of task-specific counts when records are split across multiple tasks. A fix is suggested to sum the exact row counts from the assigned chunk-level metadata when available.
bf84a43 to
00c9095
Compare
68f5c1b to
b0f4a19
Compare
00c9095 to
7a194ff
Compare
b0f4a19 to
b900f19
Compare
7a194ff to
1c0d1c9
Compare
b900f19 to
86fbf60
Compare
1c0d1c9 to
43e2723
Compare
86fbf60 to
ad7da13
Compare
43e2723 to
86a68dc
Compare
ad7da13 to
a787087
Compare
86a68dc to
5717191
Compare
a787087 to
0c572a3
Compare
5717191 to
345de8f
Compare
0c572a3 to
9e7bfcc
Compare
345de8f to
5613406
Compare
9e7bfcc to
2b66629
Compare
5613406 to
789c37b
Compare
2b66629 to
611c6e8
Compare
789c37b to
4577a1c
Compare
611c6e8 to
8346b18
Compare
4577a1c to
a4ef303
Compare
8346b18 to
33ff762
Compare
a4ef303 to
ebc61e3
Compare
33ff762 to
e5493d3
Compare
5b978d3 to
5a1ede2
Compare
108a6d6 to
1ad265e
Compare
5a1ede2 to
dc3b70c
Compare
1ad265e to
489d1f5
Compare
dc3b70c to
b05ded8
Compare
489d1f5 to
9de4f69
Compare
b05ded8 to
435a908
Compare
9de4f69 to
6da4604
Compare
435a908 to
27e604d
Compare
6da4604 to
6ac35c7
Compare
27e604d to
6f23281
Compare
6ac35c7 to
8c038c9
Compare
6f23281 to
f31f376
Compare
8c038c9 to
cc8d75d
Compare
f31f376 to
3e97fe4
Compare
cc8d75d to
8d5657c
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8d5657c. Configure here.
3e97fe4 to
9bd111b
Compare
8d5657c to
ee2c047
Compare
9bd111b to
da9edd1
Compare
…, row_id Problem: - read_mcap reads each file in one task, so parallelism is capped by the number of files. - Every file is opened, even when its summary shows it holds nothing the read selects. - Rows have no stable id, so a checkpointed pipeline cannot resume. Change: read_mcap runs on Datasource V2 when DataContext.use_datasource_v2 is set (the default). The old reader stays as the fallback. file summaries -> one listing row per chunk -> ~128 MiB read tasks - Listing reads each file's summary and emits one row per chunk. A file that cannot match the selection is dropped before any data is read. - Chunks are packed into read tasks of about 128 MiB uncompressed (RAY_DATA_MCAP_BIN_PACKING_BYTES). A big file spans many tasks; small files share one. - Each task seeks straight to its chunks. The new log_time_order argument (default True) merges them by log time. - row_id names a message by its file, chunk offset and position. It is the same however the read is split. Enable it with include_row_id=True; a row_id checkpoint turns it on automatically. - Column pruning and limits are pushed into the reader. - data holds decoded JSON only when every selected channel is JSON. Otherwise it holds bytes. mcap_data_column decides this once, from the sampled files' summaries. - Columns match the old reader, plus channel_metadata (a map). schema_data is dictionary-encoded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…d-in Problem: one message per row is the wrong unit for most uses. A labeling job wants clips, an export wants one stream, episode features want one recording. Building those from message rows needs a shuffle, and the result often cannot be decoded on its own. Change: new read_granularity argument (V2 only). - message (default): one message per row, as before. - window: one row per time window of a file, set by WindowSpec (length, stride, anchor). Each window is emitted once, by the task that owns the chunk its start falls in. - topic: one row per topic of a file. - file: one row per file. Window, topic and file rows hold their messages as parallel lists (log_time, topic, data, ...), plus each channel's schema, so a row decodes on its own. Payloads stay encoded. A topic or file row over RAY_DATA_MCAP_MAX_ROW_BYTES (1 GiB) fails with a hint to use windows. Video lead-in: a window also carries, for each video topic in it, the frames back to the last keyframe, so the clip decodes on its own. A topic row of a video topic starts on a keyframe too. frames K p p p K p p | p p K p p | lead-in [-----) window [-----------) - A topic is video if its schema is a known video schema (Foxglove CompressedVideo or CompressedImage, sensor_msgs CompressedImage), or if the new video_topics argument lists it. - The codec comes from the message's format field, or else its bytes. - The look-back is capped at 10 s (RAY_DATA_MCAP_MAX_LEAD_IN_S). Row ids name the file, the window or topic, and a digest of the read options, so a checkpoint never mixes rows of different reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: a video message cannot be decoded on its own. It needs the frames back to its keyframe. Users had to read windows and decode them again in map_batches. Change: new video=VideoOptions(fps=..., resize=...) argument. - message granularity: one row per decoded RGB frame, in a frame column (uint8, shape (height, width, 3)). Every selected topic must be video. - window granularity: each row gets frames:<topic> and frame_times:<topic> for each video topic. Other topics stay in the lists. - topic and file granularity refuse video. Frames stay correct however the read is split: - A task that starts mid-stream first decodes the frames back to the previous keyframe, then drops them. - fps keeps at most one frame per interval, on a grid aligned to the epoch, so the kept frames do not depend on the split. - Each frame is matched to its message by log_time, even when the decoder releases it late. Decoders: PyAV for H.264, H.265, VP9 and AV1; Pillow for JPEG and PNG. The frame shape comes from resize, or from the first keyframe. A window whose frames exceed RAY_DATA_MCAP_MAX_ROW_BYTES fails with a hint to use fps or resize. With the frame columns pruned, nothing is decoded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
… from statistics Problem: topic and log_time are known from the summary before any data is read. Still, a ds.filter on them ran only after every chunk was read, and count() read every chunk too. Change: - A ds.filter on topic (== or is_in) or on log_time comparisons narrows the read's selection. Listing then skips the files and chunks that cannot match. Other filter terms stay above the read. - At window, topic and file granularity the filter is not pushed, since those rows hold topic and log_time as lists. - count() sums the summaries' per-channel message counts and reads no chunk. It falls back to reading when a time range, a limit, a partition filter or video decoding makes that count inexact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: an MCAP file also holds attachments (named blobs such as calibration files) and metadata records (string maps such as the recorder version). read_mcap could not return either. Change: read_granularity="attachment" and "metadata", one record per row. - The summary indexes both by byte offset, so each task seeks straight to its records. - count() comes from the summary's record counts. - topics, message_types and video do not apply and are refused. time_range is refused for metadata. - Row ids are <path>#a=<offset> and <path>#md=<offset>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
|
Folded into #66656, which now carries the attachment and metadata rows too, to keep the stack to four PRs. |

Description
Optional follow-up to the
read_mcapredesign, kept as a draft. Stacked on #66656, which sits on #66670: review the top commit only.Problem. Besides messages, MCAP files hold two kinds of records that
read_mcapcould not return:Change. Two new
read_granularityvalues, one record per row:read_granularityattachmentpath,name,media_type,log_time,create_time,datametadatapath,name,metadata(map<string, string>)count()comes from the record counts in the index.topics,message_typesandvideodo not apply.time_rangeapplies to attachments only.Code: record rows live in
mcap_records.py.Decision to confirm: records are
read_granularityvalues rather than a separate function, so they share the path, filesystem and partitioning options.Related issues
No open issue or PR reads MCAP attachments or metadata in Ray Data.
Additional information
227 unit tests passed.
test_mcap.py: 63 passed, 5 skipped. pre-commit and pyrefly clean.AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code