Repository navigation
Conversation
marwan116
force-pushed
the
marwan/mcap-dsv2-attachments
branch
from
November 9, 2026 21:47
c45c62b to
c7ae8c3
Compare
marwan116
force-pushed
the
marwan/mcap-dsv2-attachments
branch
from
November 10, 2026 01:48
c7ae8c3 to
8bcc076
Compare
Problem: - An MCAP file can hold Attachment and Metadata records, such as a calibration file or the recorder's version, and read_mcap read only messages. Change: - read_granularity="attachment" | "metadata" returns one row per record, read by byte offset from the summary's record index. - A read task opens at most 16 files of small records (RAY_DATA_MCAP_RECORD_FILES_PER_READ_TASK) and reads them several at a time (RAY_DATA_MCAP_RECORD_FILES_IN_FLIGHT). - A record row weighs three times its record's bytes, the memory reading it takes: the bytes read, the parsed record and its Arrow column. Files of large records then go one per task. - time_range applies to attachments. Without one, count() counts the entries of the summary's record index and reads no record. - With max_errored_blocks set, an indexed record that does not decode costs only its own row, and the task's error names its byte offset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
marwan116
force-pushed
the
marwan/mcap-dsv2-attachments
branch
from
November 11, 2026 19:43
8bcc076 to
4e2e930
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem.
read_mcapread only messages.Change.
read_granularityis"message"(the default),"attachment"or"metadata"; a sample read refuses it. A record read refusestopics,message_types,include_paths=Trueandinclude_metadata=False, which apply only to message rows.time_rangeapplies to attachments.count()counts the entries of the record index, so it reads no record. With a time range on attachments, or withmax_errored_blocksset, it reads instead.max_errored_blocksset, an indexed record that does not decode costs only its own row, and the task's error names its byte offset.RecordKindrule says which records a file holds, and listing, reading and counting all use it.RAY_DATA_MCAP_RECORD_FILES_PER_READ_TASK).RAY_DATA_MCAP_RECORD_FILES_IN_FLIGHT) throughmake_async_gen, and closes that read-ahead explicitly.Measured on S3. Read-ahead pays on small files, so it stays. Its memory cost falls on files of large records, and the working-set weight now puts those one per task, where there is nothing to read ahead. Weighed by payload, the read went (median of 3; time, then read-task peak):
At 3x, the small-file set packs 12 files to a task instead of 15, and the 24 MiB set packs one file to a task instead of four.
Related issues
Additional information
655 passed, 32 skipped. pre-commit and pyrefly clean.
AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code