Repository navigation
Conversation
Problem: - A sample read needs a way to say, per topic, where its steps fall around a sample and which message fills each step. - The read itself is large; landing the spec first keeps each review to one idea. Change: - TopicSpec(steps, match=None, pad="edge"), exported from ray.data.datasource. - ray.data.datasource.mcap.steps: before/after (1/fps apart around at="anchor", "clip_start" or "clip_end"), across, at, and last/next/every, which take the recorded messages themselves. - ray.data.datasource.mcap.match: nearest (default), backward, all_in_step. - The specs validate their arguments and do no I/O. The read that uses them comes later in the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - Given a sample (clip and anchor) and a topic's TopicSpec, a sample read must decide which recorded message fills each step, and what a step that no message fills holds. Change: - mcap_samples.TopicPlan: a TopicSpec compiled for one topic, with the span of log times its steps can reach for a sample. - resolve_steps / resolve_slots: the message (or, for all_in_step, the messages) that fill each step, from a sorted view of the topic's log times. nearest never looks across the anchor; backward takes the last message at or before each step. - Steps no message fills are padded by pad="edge" or "zeros", or fail with pad="raise", naming the sample. - Pure functions over log times: no I/O. The reader that feeds them comes later in the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A sample read takes a dataset of samples. Bad samples (no file, several files, an empty clip, an anchor outside its clip) must fail with an error that names the sample, before any file is read. - Planning needs the sample row's schema without reading a file. Change: - check_samples_schema: the samples dataset has files, clip_start and clip_end with the right types, an anchor when a topic measures from it, and no column the read adds. - sample_manifest: a batch of samples as manifest rows, one file each, failing on the first bad sample by its sample_id or its file and clip. - sample_row_schema: the samples' own columns, then each topic's data:, times: and valid: columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request introduces helper functions and validation logic for managing sample datasets and row layouts in the MCAP datasource. Specifically, it adds sample_manifest to construct manifest rows from a batch of samples, check_samples_schema to validate the schema of a samples dataset, and sample_row_schema to define the PyArrow schema of sample rows including topic-specific columns. Comprehensive unit tests have also been added to verify these new functionalities under various conditions. I have no feedback to provide as there are no review comments to address.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4727613. Configure here.
| missing = anchor.is_null().to_numpy(zero_copy_only=False) | ||
| if needs_anchor: | ||
| _check(batch, ~missing, _anchor_problem) | ||
| values = anchor.fill_null(0).to_numpy() |
There was a problem hiding this comment.
Chunked sample batches crash validation
High Severity
sample_manifest converts clip_start, clip_end, and anchor with to_numpy()'s default zero-copy mode. Table.column() returns a ChunkedArray, and a copy is required when there is more than one chunk, so validation raises ArrowInvalid instead of checking the samples. The files column already goes through combine_chunks(); the time columns do not, and Ray Data batches commonly have several chunks.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 4727613. Configure here.
92d18d4 to
1f257ca
Compare


Summary
Checks a samples dataset and lays out a sample row's schema, so a sample read fails on a bad sample before any file is read and plans without opening a file.
Stack: #67167 → #67924 → #67925 → #67926 (this PR) → #67927 → #67189 → #66670 → #66656 → #67193. #67189 was split into the five PRs from #67924 to #67189 to keep each review near 1k lines.
Size: 202 lines of source and 144 of tests.
Before
Step resolution (previous PR) works on one sample at a time. Nothing validates the samples dataset or says what a sample row looks like.
After
check_samples_schema: the samples havefiles,clip_startandclip_endwith the right types, ananchoronly when a topic measures from it, and no column the read adds.sample_manifest: a batch of samples becomes manifest rows, one file each. The first bad sample (no file, several files, an empty clip, an anchor outside its clip) fails, named by itssample_idor its file and clip.sample_row_schema: the samples' own columns, then each topic'sdata:<topic>,times:<topic>andvalid:<topic>.Subsystems
IO (
_internal/datasource_v2/formats/mcap/mcap_samples.py).Testing
432 passed, 7 skipped (the MCAP unit and datasource tests at this level). pre-commit and pyrefly clean.
Not a duplicate: split out of #67189 at a reviewer's request. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code