Skip to content

[Data] read_mcap: attachment and metadata granularities - #66657

Closed
marwan116 wants to merge 5 commits into
marwan/mcap-dsv2-pushdownfrom
marwan/mcap-dsv2-records
Closed

marwan116 wants to merge 5 commits into
marwan/mcap-dsv2-pushdownfrom
marwan/mcap-dsv2-records

Conversation

@marwan116

@marwan116 marwan116 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

Optional follow-up to the read_mcap redesign, kept as a draft. Stacked on #66656, which sits on #66670: review the top commit only.

Problem. Besides messages, MCAP files hold two kinds of records that read_mcap could not return:

  • Attachments: calibration files, maps, thumbnails.
  • Metadata: recorder version, vehicle id, notes.

Change. Two new read_granularity values, one record per row:

read_granularity columns
attachment path, name, media_type, log_time, create_time, data
metadata path, name, metadata (map<string, string>)
calib = ray.data.read_mcap("s3://bucket/recordings/", read_granularity="attachment")
| chunk | chunk | attachment | chunk | metadata | ... | index |
                     ▲                    ▲               │
                     └── seek ────────────┴── offsets ────┘
  • Records are found through the file's index and read by seeking straight to them.
  • A file without that index is scanned. Message chunks are skipped without being decompressed.
  • count() comes from the record counts in the index.
  • topics, message_types and video do not apply. time_range applies to attachments only.

Code: record rows live in mcap_records.py.

Decision to confirm: records are read_granularity values rather than a separate function, so they share the path, filesystem and partitioning options.

Related issues

No open issue or PR reads MCAP attachments or metadata in Ray Data.

Additional information

pytest python/ray/data/tests/unit/datasource_v2/test_mcap_*.py
PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py

227 unit tests passed. test_mcap.py: 63 passed, 5 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116
marwan116 requested a review from a team as a code owner October 2, 2026 04:41
@marwan116 marwan116 added the data Ray Data-related issues label Oct 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for reading MCAP Attachment and Metadata records as rows using the V2 datasource, introducing the 'attachment' and 'metadata' read granularities. It includes schema definition, record indexing, scanning, and reading capabilities, along with comprehensive unit tests. The review feedback highlights a critical issue in mcap_reader.py where read_metadata reports inflated, file-wide row counts instead of task-specific counts when records are split across multiple tasks. A fix is suggested to sum the exact row counts from the assigned chunk-level metadata when available.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from bf84a43 to 00c9095 Compare October 2, 2026 16:10
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 68f5c1b to b0f4a19 Compare October 2, 2026 16:11
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 00c9095 to 7a194ff Compare October 4, 2026 17:38
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from b0f4a19 to b900f19 Compare October 4, 2026 17:39
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 7a194ff to 1c0d1c9 Compare October 5, 2026 00:22
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from b900f19 to 86fbf60 Compare October 5, 2026 00:23
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 1c0d1c9 to 43e2723 Compare October 5, 2026 00:37
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 86fbf60 to ad7da13 Compare October 5, 2026 00:38
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 43e2723 to 86a68dc Compare October 5, 2026 02:20
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from ad7da13 to a787087 Compare October 5, 2026 02:22
@marwan116
marwan116 added this pull request to stack #66712 October 5, 2026 02:25
@marwan116 marwan116 self-assigned this Oct 5, 2026
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 86a68dc to 5717191 Compare October 5, 2026 02:32
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from a787087 to 0c572a3 Compare October 5, 2026 02:33
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 5717191 to 345de8f Compare October 5, 2026 02:45
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 0c572a3 to 9e7bfcc Compare October 5, 2026 02:46
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 345de8f to 5613406 Compare October 5, 2026 02:56
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 9e7bfcc to 2b66629 Compare October 5, 2026 02:56
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 5613406 to 789c37b Compare October 5, 2026 03:01
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 2b66629 to 611c6e8 Compare October 5, 2026 03:01
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 789c37b to 4577a1c Compare October 5, 2026 03:05
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 611c6e8 to 8346b18 Compare October 5, 2026 03:06
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 4577a1c to a4ef303 Compare October 5, 2026 03:13
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 8346b18 to 33ff762 Compare October 5, 2026 03:14
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from a4ef303 to ebc61e3 Compare October 5, 2026 03:43
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 33ff762 to e5493d3 Compare October 5, 2026 03:44
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 5b978d3 to 5a1ede2 Compare October 5, 2026 21:36
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 108a6d6 to 1ad265e Compare October 5, 2026 21:36
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 5a1ede2 to dc3b70c Compare October 6, 2026 04:51
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 1ad265e to 489d1f5 Compare October 6, 2026 04:51

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from dc3b70c to b05ded8 Compare October 6, 2026 05:14
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 489d1f5 to 9de4f69 Compare October 6, 2026 05:14
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from b05ded8 to 435a908 Compare October 6, 2026 06:04
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 9de4f69 to 6da4604 Compare October 6, 2026 06:04
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 435a908 to 27e604d Compare October 6, 2026 06:34
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 6da4604 to 6ac35c7 Compare October 6, 2026 06:34
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 27e604d to 6f23281 Compare October 6, 2026 06:47
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 6ac35c7 to 8c038c9 Compare October 6, 2026 06:47
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 6f23281 to f31f376 Compare October 6, 2026 16:21
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 8c038c9 to cc8d75d Compare October 6, 2026 16:21

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from f31f376 to 3e97fe4 Compare October 6, 2026 16:55
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from cc8d75d to 8d5657c Compare October 6, 2026 16:55

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8d5657c. Configure here.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 3e97fe4 to 9bd111b Compare October 6, 2026 17:17
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-records branch from 8d5657c to ee2c047 Compare October 6, 2026 17:17
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 9bd111b to da9edd1 Compare October 6, 2026 19:00
marwan116 and others added 5 commits October 7, 2026 15:38
…, row_id

Problem:
- read_mcap reads each file in one task, so parallelism is capped by the
  number of files.
- Every file is opened, even when its summary shows it holds nothing the
  read selects.
- Rows have no stable id, so a checkpointed pipeline cannot resume.

Change: read_mcap runs on Datasource V2 when
DataContext.use_datasource_v2 is set (the default). The old reader stays
as the fallback.

  file summaries -> one listing row per chunk -> ~128 MiB read tasks

- Listing reads each file's summary and emits one row per chunk. A file
  that cannot match the selection is dropped before any data is read.
- Chunks are packed into read tasks of about 128 MiB uncompressed
  (RAY_DATA_MCAP_BIN_PACKING_BYTES). A big file spans many tasks; small
  files share one.
- Each task seeks straight to its chunks. The new log_time_order
  argument (default True) merges them by log time.
- row_id names a message by its file, chunk offset and position. It is
  the same however the read is split. Enable it with
  include_row_id=True; a row_id checkpoint turns it on automatically.
- Column pruning and limits are pushed into the reader.
- data holds decoded JSON only when every selected channel is JSON.
  Otherwise it holds bytes. mcap_data_column decides this once, from
  the sampled files' summaries.
- Columns match the old reader, plus channel_metadata (a map).
  schema_data is dictionary-encoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…d-in

Problem: one message per row is the wrong unit for most uses. A labeling
job wants clips, an export wants one stream, episode features want one
recording. Building those from message rows needs a shuffle, and the
result often cannot be decoded on its own.

Change: new read_granularity argument (V2 only).
- message (default): one message per row, as before.
- window: one row per time window of a file, set by WindowSpec (length,
  stride, anchor). Each window is emitted once, by the task that owns
  the chunk its start falls in.
- topic: one row per topic of a file.
- file: one row per file.

Window, topic and file rows hold their messages as parallel lists
(log_time, topic, data, ...), plus each channel's schema, so a row
decodes on its own. Payloads stay encoded. A topic or file row over
RAY_DATA_MCAP_MAX_ROW_BYTES (1 GiB) fails with a hint to use windows.

Video lead-in: a window also carries, for each video topic in it, the
frames back to the last keyframe, so the clip decodes on its own. A
topic row of a video topic starts on a keyframe too.

  frames    K p p p K p p | p p K p p |
  lead-in           [-----)
  window                  [-----------)

- A topic is video if its schema is a known video schema (Foxglove
  CompressedVideo or CompressedImage, sensor_msgs CompressedImage), or
  if the new video_topics argument lists it.
- The codec comes from the message's format field, or else its bytes.
- The look-back is capped at 10 s (RAY_DATA_MCAP_MAX_LEAD_IN_S).

Row ids name the file, the window or topic, and a digest of the read
options, so a checkpoint never mixes rows of different reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: a video message cannot be decoded on its own. It needs the
frames back to its keyframe. Users had to read windows and decode them
again in map_batches.

Change: new video=VideoOptions(fps=..., resize=...) argument.
- message granularity: one row per decoded RGB frame, in a frame column
  (uint8, shape (height, width, 3)). Every selected topic must be video.
- window granularity: each row gets frames:<topic> and
  frame_times:<topic> for each video topic. Other topics stay in the
  lists.
- topic and file granularity refuse video.

Frames stay correct however the read is split:
- A task that starts mid-stream first decodes the frames back to the
  previous keyframe, then drops them.
- fps keeps at most one frame per interval, on a grid aligned to the
  epoch, so the kept frames do not depend on the split.
- Each frame is matched to its message by log_time, even when the
  decoder releases it late.

Decoders: PyAV for H.264, H.265, VP9 and AV1; Pillow for JPEG and PNG.
The frame shape comes from resize, or from the first keyframe. A window
whose frames exceed RAY_DATA_MCAP_MAX_ROW_BYTES fails with a hint to use
fps or resize. With the frame columns pruned, nothing is decoded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
… from statistics

Problem: topic and log_time are known from the summary before any data
is read. Still, a ds.filter on them ran only after every chunk was read,
and count() read every chunk too.

Change:
- A ds.filter on topic (== or is_in) or on log_time comparisons narrows
  the read's selection. Listing then skips the files and chunks that
  cannot match. Other filter terms stay above the read.
- At window, topic and file granularity the filter is not pushed, since
  those rows hold topic and log_time as lists.
- count() sums the summaries' per-channel message counts and reads no
  chunk. It falls back to reading when a time range, a limit, a
  partition filter or video decoding makes that count inexact.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: an MCAP file also holds attachments (named blobs such as
calibration files) and metadata records (string maps such as the
recorder version). read_mcap could not return either.

Change: read_granularity="attachment" and "metadata", one record per
row.
- The summary indexes both by byte offset, so each task seeks straight
  to its records.
- count() comes from the summary's record counts.
- topics, message_types and video do not apply and are refused.
  time_range is refused for metadata.
- Row ids are <path>#a=<offset> and <path>#md=<offset>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116

Copy link
Copy Markdown
Contributor Author

Folded into #66656, which now carries the attachment and metadata rows too, to keep the stack to four PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant