Repository navigation
[Data] MCAP on Datasource V2: summary indexer, chunk-level read tasks, row_id - #66654
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new V2 datasource for MCAP files (MCAPDatasourceV2) in Ray Data, enabling summary-driven listing, chunk-level read tasks, and deterministic row IDs. While the overall design is solid, several critical issues were identified in the implementation: pa.repeat does not exist in PyArrow and will cause an AttributeError during path partitioning; returning raw bytes on JSON decoding failure in decode_payload leads to mixed-type columns and pickle crashes; schema inference needs to handle None values from decode_payload to prevent type inference errors; and accessing summary.channels directly by channel ID can raise a KeyError if the ID is missing from the summary.
3da2f29 to
f606969
Compare
6191ee4 to
baf5f6e
Compare
baf5f6e to
19b765f
Compare
19b765f to
e2dd57a
Compare
e2dd57a to
8158547
Compare
88831ee to
67a3255
Compare
67a3255 to
a3979e4
Compare
5e9f139 to
0b506e3
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0b506e3. Configure here.
…, row_id Problem: - read_mcap reads each file in one task, so parallelism is capped by the number of files. - Every file is opened, even when its summary shows it holds nothing the read selects. - Rows have no stable id, so a checkpointed pipeline cannot resume. Change: read_mcap runs on Datasource V2 when DataContext.use_datasource_v2 is set (the default). The old reader stays as the fallback. file summaries -> one listing row per chunk -> ~128 MiB read tasks - Listing reads each file's summary and emits one row per chunk. A file that cannot match the selection is dropped before any data is read. - Chunks are packed into read tasks of about 128 MiB uncompressed (RAY_DATA_MCAP_BIN_PACKING_BYTES). A big file spans many tasks; small files share one. - Each task seeks straight to its chunks. The new log_time_order argument (default True) merges them by log time. - row_id names a message by its file, chunk offset and position. It is the same however the read is split. Enable it with include_row_id=True; a row_id checkpoint turns it on automatically. - Column pruning and limits are pushed into the reader. - data holds decoded JSON only when every selected channel is JSON. Otherwise it holds bytes. mcap_data_column decides this once, from the sampled files' summaries. - Columns match the old reader, plus channel_metadata (a map). schema_data is dictionary-encoded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - mcap's SeekingReader parses a summary a few bytes per read. On S3 each read of a raw file is one request, and the listing and every read task of a file read its summary. One summary read took about 20,000 requests. - Chunk records were read field by field too, and a file without a chunk index was scanned with no read buffer. Change: - New mcap_io module. read_summary_from checks both magic bytes and the footer, then parses the summary section in memory, all in one to three ranged reads. A file that is not a complete MCAP file raises a ValueError naming it. - read_chunk fetches a chunk record in one ranged read. The listing, planning and read tasks use both. - buffered wraps a linear scan in a read buffer of DataContext.streaming_read_buffer_size bytes. - Tests that patched mcap's readers now count planning scans and strip summary records through mcap_testing helpers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The per-shard packing test defined its pack and units helpers inside itself, so no other listing test could pack files into read tasks the way a ListFiles task does. Change: - pack(datasource, shard) and chunk_units(tasks) move to module level in test_mcap_listing.py. Each pack call builds its own partitioner, as each ListFiles task does. - Test code only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A file whose summary cannot be read, such as a recording cut short, failed the listing task and every file listed with it. - Planning reads some summaries on the driver, so a bad sampled file failed the whole read there, where max_errored_blocks cannot help. Change: - The listing lists such a file as one row that names no chunk and weighs a whole read task, so the packer puts no other file in its task. The reader takes that row as the whole file. - Reading it fails that task alone, with an error that names the file. max_errored_blocks can then skip just that file, as with the legacy reader. - Planning skips a sampled file whose summary cannot be read. Its own read task reports the error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - Building a table took several times the table's size. Each row's values waited in Python lists until the build, and every row carried its own copy of its channel's schema data and metadata. - schema_data was built as plain binary, a copy per row, and only then dictionary-encoded. - The lists stayed alive while the finished table was yielded. Change: - Integers go into typed arrays, and payloads and row ids into one buffer each, which the finished arrays wrap without a copy. - What a row takes from its channel and schema is stored once per table, with an int32 per row pointing at it. schema_data is built as a dictionary from the start. - build empties the builder before it returns the table, so a yielded table is the only copy of its rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The reader cut a table once its payloads, plus a flat 96 bytes a row, reached the target block size. row_id, the channel columns and appended columns such as path were not counted. - So with row_id or long channel metadata on, a table grew far past the target: on a file of small IMU messages, to nearly four times it. Change: - estimated_bytes counts what each row adds to every column the reader emits: the payload, the integers, row_id, the channel strings and metadata, and each new schema_data entry once. - The reader passes the per-row bytes of its partition and synthesized columns, such as path, as extra_row_bytes. - Decoded JSON counts six bytes per payload byte, since its values are Python objects until the table is built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The packer filled each read task up to its budget by the chunks' uncompressed bytes. A row costs more in a table than in its chunk, so a full task built tables many times its budget: up to 30 times. Change: - A chunk's listing weight is 1.2 times the bytes its rows take as a table: its uncompressed records plus, per message, the columns a row adds. Those are the channel's strings and metadata, and row_id and path when the read has them (MessageRowLayout). - A chunk's message count comes from the length of its message indexes, 16 bytes per message, and falls back to the statistics without them. - RAY_DATA_MCAP_BIN_PACKING_BYTES budgets this estimate, 128 MiB by default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - SeekLog and chunks_read, which record the chunks a read fetches, lived in the data column tests, so the reader tests could not use them. Change: - SeekLog, _SeekLoggingFile and chunks_read move unchanged from test_mcap_data_column.py to mcap_testing.py. - Test code only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A read task built every table of its share of a file, then sliced the tables down to the limit. So a small limit still fetched and decoded every chunk of the task, 125 on a task of IMU messages. Change: - The reader passes the rows still wanted to each file and stops pulling messages once it has them. No later message is added to a table, and no later chunk is fetched or decoded. - A task whose limit is met does not open its next file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- log_time_order=True did not sort a file without a chunk index at
all, though one task reads it whole.
- A file bigger than one read task is split across tasks, and each
task orders only its own chunks. The docstring did not say so.
Change:
- A file without a chunk index is sorted by log time in its task, as
the legacy reader does. The sort is stable, so equal log times keep
file order. With a limit, only the earliest messages are held.
- The log_time_order docstring says the order holds within a read
task, not across tasks or files, even with preserve_order, and that
ds.sort("log_time") orders a whole file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - channel_metadata was a map on every row. ROS 2 stores each channel's QoS profiles there, so on small, high-rate messages the metadata outweighed the message: 70% of the output on an IMU file. - A dictionary of map values would store it once, but Parquet cannot write one. Change: - channel_metadata is the channel's metadata map as JSON text with sorted keys, dictionary-encoded like schema_data: dictionary<int32, string>. A table stores each distinct value once, and a row holds an int32 index. - The listing estimate counts 4 bytes per row for it, and the builder counts each text once. - The read_mcap docstring says to read it with json.loads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The packer places a file's chunks one by one, so a file can be split across read tasks, even a small one that lands in a partly full task. - Those tasks are sent in any order. With preserve_order, a split file came out with its first messages last. Change: - When the read keeps order (DataContext.execution_options .preserve_order), OrderedFileGrouper joins each file's chunk rows into one run before the packer sees them. The packer splits no run, so one task reads the file and merges its chunks by log time. - A file heavier than a read task's budget gets a task of its own and holds that much memory, as the legacy reader does. - Without preserve_order, the rows go to the packer as listed. - The log_time_order docstring says what order a read returns, with and without preserve_order. - New tests: the grouping, and a preserve_order read of three files that an unordered read would split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - Planning decodes one selected message to type the data column. When the chunk it reads was corrupt, planning raised on the driver, so one damaged chunk failed the whole read before any task ran. - A corrupt record in a file without a chunk index did the same. Change: - Planning skips a sampled chunk that cannot be decoded and types data from the next chunk. The chunk's own read task reports the error. - A record that does not parse ends the scan of a file without a chunk index, so data is typed from the messages before it. - An error reading the file still raises. - New tests: a sampled chunk with a corrupt payload or header, and a corrupt record in a file without a chunk index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - Planning types the data column from the files it samples. Nothing said what happens when a file past the sample holds other JSON values. Change: - No code change. The read_mcap docstring says each read task types its own JSON values, as the legacy reader types each file. A field only some files have is null in the other rows, and a field whose type differs between files fails where their rows are combined. - New tests pin it. Past the sample, a new field and an int that becomes a float unify with the planned blocks, and a clashing type fails where blocks combine. A CDR file past the sample fails its read task, and mixed types in one task are refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - Planning typed data from one sampled message, while a read task types it from every message it holds. When the first message lacked a field, or had an int where later ones had a float, ds.schema() disagreed with the blocks. The legacy reader's schema() reads the first file whole, so it agreed. Change: - Planning types data from every selected message of the first chunk that holds one, up to 1,000, with the function a read task uses. In a file without a chunk index it takes the first 1,000 the scan meets. - A payload that does not decode is left out of the type, and its read task reports it, so one bad message no longer fails planning. - The read_mcap docstring says that schema() comes from the first chunk, so a field that first appears later is missing from it. - New tests: a field only some messages have, an int that is a float elsewhere, a malformed payload, and the 1,000-message cap. End to end, schema() matches the materialized data on both read paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The read-unit comment in MCAPReader._tables said a checkpoint that hands a task's unit back skips only that task's rows. Listing excludes units chunk by chunk, and a task that owns several chunks of a file names only the first, so it would skip only that chunk. Change: - The comment says so, and that nothing hands units back yet: a row_id checkpoint filters rows by id. - Comment only, no behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - With preserve_order, the grouper handed whole files to the shared packer. The packer sends a heavy file's task at once and holds light files in open bins, so a large file listed after small ones came out first. A recording split across files came out scrambled, and limit could return rows from a later file. Change: - With preserve_order, the grouper packs whole files into tasks itself, in listing order. A file joins the open task while it fits the budget; otherwise the open task leaves first. - The read_mcap docstring says files come out in listing order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The tests/unit module list depends only on tests/conftest.py, so Bazel leaves tests/unit/datasource_v2/conftest.py out of the runfiles. Premerge failed the MCAP listing, reader and data-column tests with "fixture 'chunked_file' not found". Change: - Add a py_library for that conftest and depend on it from the tests/unit module list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
bff8f7b to
ac54950
Compare
… source's own metadata (#67037) Three changes to shared Datasource V2 code so a source whose listing already knows more than a path and a size can use the common machinery instead of forking it. 1 of 5 in a stack moving `read_iceberg` onto DSv2; this one touches only shared code. **A source's own manifest columns survive bin packing** — `interfaces/file_manifest.py`, `common/online_bin_packer.py`. `FileManifest` already lets a source add columns beside the three required ones, but `OnlineBinPacker` rebuilt each bin from those three alone, so the rest were dropped. Adds `FileManifest.extra_columns` and an `extra_columns` argument to `construct_manifest`, and carries a zero-copy one-row table on each `BinItem`. Packing a manifest with two source columns and reading the names back out: ``` master: ['__path', '__file_size', '__file_chunk_metadata'] this PR: [... , '__ib_record_count', '__ib_delete_paths'] ``` ```mermaid flowchart TD L["ListFiles task"] -->|"FileManifest: 3 required + 2 source columns"| P["OnlineBinPacker"] P -->|"before: rebuilt from the 3 required"| B["ReadFiles sees 3"] P -->|"after: BinItem.extra_columns carries each row"| A["ReadFiles sees 5"] ``` **The pushdown mixins carry their own state** — `interfaces/pushdown.py`, `common/arrow_file_scanner.py`. `SupportsColumnPruning` and `SupportsLimitPushdown` were pure interfaces, so every scanner rewrote the same field and the same `replace(self, ...)` body. They are now frozen dataclasses with keyword-only fields and default implementations: pruning intersects, limit takes the smaller. `ArrowFileScanner` drops 46 lines. Intersecting also fixes a bug — the old bodies overwrote the projection and tested it for truthiness, so an empty projection (what `count()` pushes down) re-widened: ```python scanner.prune_columns([]).prune_columns(["a"]).pruned_column_names() # master: ('a',) this PR: () ``` **`ListFiles.pushdown_is_final`** — `logical/operators/read_operator.py`, `logical/rules/derive_list_files_pushdown.py`, `logical/rules/pushdown_count_files.py`. `DeriveListFilesPushdown` clears a listing's constraints when the consumer is not a `ReadFiles`, since an arbitrary consumer will not apply them. `PushdownCountFiles` rewrites `Count(ReadFiles(ListFiles))` into `MapBatches(count_rows, ListFiles)`, and its `count_rows` does apply the scanner's filter, through the same reader the deleted `ReadFiles` would have used. The flag lets that rule say so instead of silently losing the predicate and returning an unfiltered count. --- Tests: `pytest python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py test_pushdown_mixins.py test_derive_list_files_pushdown.py` — 31 passed. `test_pushdown_mixins.py` is new; the other two gain cases for extra columns surviving packing/splitting/merging and for `pushdown_is_final`. ruff 0.8.4 (default and `--select I`), black 22.10.0, pydoclint 0.9.1 and pyrefly 0.51.0 clean on these files. Not a duplicate: no open PR changes `OnlineBinPacker`, the pushdown mixins or `PushdownCountFiles`. #66654 (MCAP) and #66573 (Hive HS2) consume this shared code rather than modify it, and the mixin consolidation is what review on #66654 asked for. AI assistance (Claude Code) was used. I have reviewed every changed line and ran the tests above locally. Signed-off-by: Aarrya <aarrya.saraf@anyscale.com> 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Aarrya <aarrya.saraf@anyscale.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#67166) ## How to review Start with the module docstring of `mcap_file_cuts.py`: it states when a file is cut and when it is read whole. Then read `test_mcap_file_cuts.py`, one case per test. | File | What changes | Lines | |---|---|---| | `mcap_file_cuts.py` | Cuts a heavy file at the task budget, and keeps files in listing order. | 190 | | `mcap_datasource_v2.py`, `mcap_file_indexer.py`, `mcap_task_summary.py` | Use the cuts when listing a file. | 64 | | `read_api.py` | The docstring says when a file keeps its order. | 17 | | `test_mcap_file_cuts.py`, `test_mcap.py` | The cases, and an end-to-end ordered read. | 413 | **Problem.** - With `preserve_order`, #66654 reads each file in one task, even when the read does not keep log-time order. A large recording then loses its parallelism. **Change.** - With `preserve_order` and `log_time_order`, each file is read whole by one task. That task merges the file's chunks by log time and holds one ranged read, one chunk and one block at a time, whatever the file's size. Chunks that overlap in time are held together, as before. - Other reads cut a large file at the task budget. With `preserve_order`, its tasks are sent in order. - With `preserve_order`, files also come out in the order they were listed. Small files are packed in listing order, and a large file's tasks leave after the files listed before it. - **Behaviour change:** a file lighter than one task stays in one task, with or without `preserve_order`. Before, an unordered read split such files to fill tasks, so tasks now come out less full: 20 files of 0.6 budget make 20 tasks instead of 13. Splitting made video reads slower, because a split piece decodes back to its keyframe and drops those frames. Message rows read the same either way. **Decisions to confirm:** - With `preserve_order` and `log_time_order`, read each file whole instead of cutting it where its log-time order holds. The cuts bought parallelism, not memory, since chunks that overlap in time were never cut apart. They needed per-chunk end times, a tied-time rule and a memory warning. - Keep light files whole in unordered reads. On 128 robot episodes with two cameras, splitting to fill tasks saved 2 to 15 tasks and made the read 2% to 16% slower. ## Related issues - The summary handoff and ranged reads were split out below this PR at a reviewer's request. No other open PR touches these files. ## Additional information ``` PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \ python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \ python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py ``` 250 passed, 4 skipped. pre-commit and pyrefly clean. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Description
Part 1 of 3 of the
read_mcapmove to Datasource V2, based on master. #67166 is stacked on it, and #67167 on #67166. The rest of theread_mcapstack (#67189 onward) sits on top.Problem. One task reads one whole file:
Change.
read_mcapmoves to Datasource V2, on by default as forread_parquet(DataContext.use_datasource_v2). The legacy datasource stays as the fallback.datasource_v2/formats/mcap/. The V2 interfaces do not change.schema_datais dictionary-encoded. A newchannel_metadatacolumn holds each channel's metadata map as JSON text, dictionary-encoded, so a block stores it once per channel.dataholds decoded JSON only when every selected channel is JSON. Otherwise it holds bytes. Planning types it from the selected messages of the first chunk that holds one, as a read task types the messages it holds.row_idis<path>#c=<chunk offset>:<index>, stable however the read is split. It is on withinclude_row_id=True, or automatically underCheckpointConfig(id_column="row_id").log_time_orderargument, defaultTrue. Each task returns each of its files inlog_timeorder. Withpreserve_order, each file is read by one task, so it comes out in order from start to end, and files come out in the order they were listed. [Data] read_mcap V2: cut large files where their log-time order holds #67166 splits a large file there too, only where its order holds.max_errored_blockscan skip it alone.RAY_DATA_MCAP_BIN_PACKING_BYTES(128 MiB).Changed after the first review round. Each is its own commit:
channel_metadatais JSON text, stored once per block.preserve_order, each file is kept in one read task, and files keep their listing order.datafrom the first chunk's messages.Measured on an 845 MB recording with 1,052 chunks: reading its 125 KB summary took about 20,000 GETs and now takes 3.
Decisions to confirm:
preserve_order, this PR reads each file in one task, as the legacy reader does. [Data] read_mcap V2: cut large files where their log-time order holds #67166 lets it split a large file where its log-time order holds.ds.schema()comes from the first chunk that holds a selected message. A field that first appears later is missing from it; the docstring says so.Related issues
log_time_ordercomes from my [Data] Stream MCAP files: seekable open, size-bounded blocks, log_time_order #66653, now closed. My legacy-path PRs [Data] Estimate MCAP in-memory size from sampled messages, not file size #65912, [Data] Dictionary-encode schema_data in MCAP blocks, and document the format #65913 and [Data] Surface MCAP channel metadata as a map column #65916 are closed in favour of this one.Additional information
178 passed, 3 skipped. pre-commit and pyrefly clean.
AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code