Repository navigation
[Data] read_mcap: read sample rows from a file's chunks - #67927
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for sample reads in the MCAP datasource, allowing users to read specific topic steps around sample anchors. It adds a new mcap_sample_rows.py module to manage grouping, timeline reading, and building Arrow tables for sample rows, along with corresponding updates to the MCAP datasource, reader, scanner, and unit tests. The feedback highlights a compatibility issue in mcap_sample_rows.py where the key parameter in bisect.bisect_right is used (unsupported in Python 3.9), and recommends replacing an assert statement in MCAPDatasourceV2 with a standard TypeError for robust public API argument validation.
39988ef to
dfff3b3
Compare
|
bugbot run |
dfff3b3 to
06c6122
Compare
|
bugbot run |
06c6122 to
02d6e87
Compare
|
bugbot run |
02d6e87 to
9e0511d
Compare
|
bugbot run |
7fbeff9 to
3268d20
Compare
3268d20 to
2b772e0
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2b772e0. Configure here.
2b772e0 to
e2f6a58
Compare
e2f6a58 to
d7a150c
Compare
7ae336f to
c1d2a73
Compare
c1d2a73 to
d7d4c80
Compare
c2a805c to
ab44a58
Compare
Problem:
- A sample row needs each topic's messages around its anchor, read
from the recording's chunks. Reading each sample on its own reads
the same chunks many times when samples lie close together.
- The chunks a task holds at once must stay within its budget.
Change:
- mcap_sample_rows: a read task takes its samples file by file, in
anchor order, and groups samples whose windows meet into runs. A
run ends before its chunks would hold more than a read task's
budget (RAY_DATA_MCAP_BIN_PACKING_BYTES). Each run reads the chunks
its topics' steps reach once (ChunkTimeline), and forgets chunks
the next run does not need.
- A file without a chunk index is scanned once (WholeFileTimeline).
The scan keeps only what steps can take: the runs' spans, the
nearest messages around each span, and the first and last messages
inside each sample's bounds.
- A corrupt chunk fails the read task. Ray's max_errored_blocks then
decides whether the task's block is dropped or the read fails.
- SampleRows builds one row per sample: the sample's columns, then each
topic's data:, times: and is_pad: lists. A row over
RAY_DATA_MCAP_MAX_ROW_BYTES fails the read, naming the sample.
- mcap_summary.topic_chunks: the chunks that may hold a topic, one rule
for the reader and, later, for sizing tasks.
- MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...) sets
up a sample read: its schema is the sample row's, and its reader
builds sample rows from manifest rows that name each sample's file.
read_mcap wires it up in the next PR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
ab44a58 to
1c268cd
Compare

Summary
Reads sample rows from a recording's chunks: the datasource and reader side of a sample read.
read_mcapwires it up in the next PR.How to review
Start with
SampleRowsinmcap_sample_rows.py: it is the read task's loop. Then readChunkTimeline, which decides which chunks to read and when to let them go.mcap_sample_rows.pymcap_reader.py,mcap_chunks.py,mcap_scanner.py,mcap_datasource_v2.pySampleRows.mcap_summary.pytopic_chunks: which chunks may hold a topic.test_mcap_sample_rows.py,mcap_testing.py,conftest.pyBefore
The pieces below resolve steps and lay out rows, but no reader reads a sample's messages from a file.
After
MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...)sets up a sample read. Its read tasks take manifest rows that name each sample's file and build one row per sample.flowchart LR M[manifest rows: one per sample] --> G[group by file, anchor order] G --> R[runs: samples whose windows meet] R --> T[ChunkTimeline: read each chunk in reach once] T --> S[resolve_steps per topic] S --> B[SampleRows: one row per sample]RAY_DATA_MCAP_BIN_PACKING_BYTES).TopicReach, estimates from the file's statistics how farlastandnextsteps reach. The reader groups samples into runs by it, so nearbysteps.lastsamples share their chunk reads. The task sizing in [Data] read_mcap: read training and labeling samples, one row per sample #67189 uses the same rule.WholeFileTimeline). The scan keeps only what steps can take: the runs' spans, the nearest messages around each span, and the first and last messages inside each sample's bounds.max_errored_blocksset, Ray then drops that task's block instead of failing the read.mcap_summary.topic_chunksis the one rule for which chunks may hold a topic.RAY_DATA_MCAP_MAX_ROW_BYTESfails the read, naming the sample._Timelineprotocol covers both timelines, and the tuning values are named constants.Subsystems
IO (
_internal/datasource_v2/formats/mcap).Testing
444 passed, 6 skipped (the MCAP unit and datasource tests at this level). pre-commit and pyrefly clean.
Not a duplicate: split out of #67189 at a reviewer's request. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code