Skip to content

[Data] read_mcap: read attachments and metadata records - #67982

Draft
marwan116 wants to merge 1 commit into
marwan/mcap-dsv2-pushdownfrom
marwan/mcap-dsv2-attachments
Draft

marwan116 wants to merge 1 commit into
marwan/mcap-dsv2-pushdownfrom
marwan/mcap-dsv2-attachments

Conversation

@marwan116

@marwan116 marwan116 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Description

Problem.

  • An MCAP file can hold Attachment and Metadata records, such as a calibration file or the recorder's version. read_mcap read only messages.

Change.

calib = ray.data.read_mcap("s3://bucket/recordings/", read_granularity="attachment")
# path, name, media_type, log_time, create_time, data
meta = ray.data.read_mcap("s3://bucket/recordings/", read_granularity="metadata")
# path, name, metadata: map<string, string>
  • read_granularity is "message" (the default), "attachment" or "metadata"; a sample read refuses it. A record read refuses topics, message_types, include_paths=True and include_metadata=False, which apply only to message rows.
  • Records are read by byte offset from the summary's record index. A file without that index is scanned. time_range applies to attachments.
  • count() counts the entries of the record index, so it reads no record. With a time range on attachments, or with max_errored_blocks set, it reads instead.
  • With max_errored_blocks set, an indexed record that does not decode costs only its own row, and the task's error names its byte offset.
  • One RecordKind rule says which records a file holds, and listing, reading and counting all use it.
  • Opening a record file is charged once, to its first record. A small record file is packed as one run, so its task opens it once. A task takes at most 16 record files (RAY_DATA_MCAP_RECORD_FILES_PER_READ_TASK).
  • A task reads its record files several at a time (RAY_DATA_MCAP_RECORD_FILES_IN_FLIGHT) through make_async_gen, and closes that read-ahead explicitly.
  • A record row weighs three times its record's bytes, which is the memory reading it takes: the bytes read, the parsed record and its Arrow column. Files of large records then go one per task, so the read-ahead runs only where records are small.

Measured on S3. Read-ahead pays on small files, so it stays. Its memory cost falls on files of large records, and the working-set weight now puts those one per task, where there is nothing to read ahead. Weighed by payload, the read went (median of 3; time, then read-task peak):

Read 16 files in flight 1 file in flight
2,000 files of 3 small attachments 11.8 s, 303 MiB 23.1 s, 291 MiB
256 files of one 24 MiB attachment 11.4 s, 572 MiB 14.0 s, 402 MiB
64 files of four 24 MiB attachments 15.8 s, 486 MiB 16.2 s, 487 MiB

At 3x, the small-file set packs 12 files to a task instead of 15, and the 24 MiB set packs one file to a task instead of four.

Related issues

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py \
  python/ray/data/tests/unit/datasource_v2/test_file_partitioners.py

655 passed, 32 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116 marwan116 added data Ray Data-related issues go add ONLY when ready to merge, run all tests labels Nov 9, 2026
@marwan116
marwan116 added this pull request to stack #67983 November 9, 2026 19:35
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-attachments branch from c45c62b to c7ae8c3 Compare November 9, 2026 21:47
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-attachments branch from c7ae8c3 to 8bcc076 Compare November 10, 2026 01:48
Problem:
- An MCAP file can hold Attachment and Metadata records, such as a
  calibration file or the recorder's version, and read_mcap read only
  messages.

Change:
- read_granularity="attachment" | "metadata" returns one row per
  record, read by byte offset from the summary's record index.
- A read task opens at most 16 files of small records
  (RAY_DATA_MCAP_RECORD_FILES_PER_READ_TASK) and reads them several at
  a time (RAY_DATA_MCAP_RECORD_FILES_IN_FLIGHT).
- A record row weighs three times its record's bytes, the memory
  reading it takes: the bytes read, the parsed record and its Arrow
  column. Files of large records then go one per task.
- time_range applies to attachments. Without one, count() counts the
  entries of the summary's record index and reads no record.
- With max_errored_blocks set, an indexed record that does not decode
  costs only its own row, and the task's error names its byte offset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-attachments branch from 8bcc076 to 4e2e930 Compare November 11, 2026 19:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant