Skip to content

[Data] read_mcap: check samples and lay out sample rows - #67926

Closed
marwan116 wants to merge 3 commits into
marwan/mcap-dsv2-sample-stepsfrom
marwan/mcap-dsv2-sample-manifest
Closed

marwan116 wants to merge 3 commits into
marwan/mcap-dsv2-sample-stepsfrom
marwan/mcap-dsv2-sample-manifest

Conversation

@marwan116

@marwan116 marwan116 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Checks a samples dataset and lays out a sample row's schema, so a sample read fails on a bad sample before any file is read and plans without opening a file.

Stack: #67167 → #67924 → #67925 → #67926 (this PR) → #67927 → #67189 → #66670 → #66656 → #67193. #67189 was split into the five PRs from #67924 to #67189 to keep each review near 1k lines.

Size: 202 lines of source and 144 of tests.

Before

Step resolution (previous PR) works on one sample at a time. Nothing validates the samples dataset or says what a sample row looks like.

After

  • check_samples_schema: the samples have files, clip_start and clip_end with the right types, an anchor only when a topic measures from it, and no column the read adds.
  • sample_manifest: a batch of samples becomes manifest rows, one file each. The first bad sample (no file, several files, an empty clip, an anchor outside its clip) fails, named by its sample_id or its file and clip.
  • sample_row_schema: the samples' own columns, then each topic's data:<topic>, times:<topic> and valid:<topic>.

Subsystems

IO (_internal/datasource_v2/formats/mcap/mcap_samples.py).

Testing

pytest python/ray/data/tests/unit/datasource_v2/test_mcap_samples.py

432 passed, 7 skipped (the MCAP unit and datasource tests at this level). pre-commit and pyrefly clean.

Not a duplicate: split out of #67189 at a reviewer's request. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

marwan116 and others added 3 commits November 6, 2026 17:00
Problem:
- A sample read needs a way to say, per topic, where its steps fall
  around a sample and which message fills each step.
- The read itself is large; landing the spec first keeps each review
  to one idea.

Change:
- TopicSpec(steps, match=None, pad="edge"), exported from
  ray.data.datasource.
- ray.data.datasource.mcap.steps: before/after (1/fps apart around
  at="anchor", "clip_start" or "clip_end"), across, at, and
  last/next/every, which take the recorded messages themselves.
- ray.data.datasource.mcap.match: nearest (default), backward,
  all_in_step.
- The specs validate their arguments and do no I/O. The read that uses
  them comes later in the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Given a sample (clip and anchor) and a topic's TopicSpec, a sample
  read must decide which recorded message fills each step, and what a
  step that no message fills holds.

Change:
- mcap_samples.TopicPlan: a TopicSpec compiled for one topic, with the
  span of log times its steps can reach for a sample.
- resolve_steps / resolve_slots: the message (or, for all_in_step, the
  messages) that fill each step, from a sorted view of the topic's log
  times. nearest never looks across the anchor; backward takes the last
  message at or before each step.
- Steps no message fills are padded by pad="edge" or "zeros", or fail
  with pad="raise", naming the sample.
- Pure functions over log times: no I/O. The reader that feeds them
  comes later in the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A sample read takes a dataset of samples. Bad samples (no file,
  several files, an empty clip, an anchor outside its clip) must fail
  with an error that names the sample, before any file is read.
- Planning needs the sample row's schema without reading a file.

Change:
- check_samples_schema: the samples dataset has files, clip_start and
  clip_end with the right types, an anchor when a topic measures from
  it, and no column the read adds.
- sample_manifest: a batch of samples as manifest rows, one file each,
  failing on the first bad sample by its sample_id or its file and
  clip.
- sample_row_schema: the samples' own columns, then each topic's
  data:, times: and valid: columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116
marwan116 requested a review from a team as a code owner November 7, 2026 01:05

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces helper functions and validation logic for managing sample datasets and row layouts in the MCAP datasource. Specifically, it adds sample_manifest to construct manifest rows from a batch of samples, check_samples_schema to validate the schema of a samples dataset, and sample_row_schema to define the PyArrow schema of sample rows including topic-specific columns. Comprehensive unit tests have also been added to verify these new functionalities under various conditions. I have no feedback to provide as there are no review comments to address.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4727613. Configure here.

missing = anchor.is_null().to_numpy(zero_copy_only=False)
if needs_anchor:
_check(batch, ~missing, _anchor_problem)
values = anchor.fill_null(0).to_numpy()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chunked sample batches crash validation

High Severity

sample_manifest converts clip_start, clip_end, and anchor with to_numpy()'s default zero-copy mode. Table.column() returns a ChunkedArray, and a copy is required when there is more than one chunk, so validation raises ArrowInvalid instead of checking the samples. The files column already goes through combine_chunks(); the time columns do not, and Ray Data batches commonly have several chunks.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 4727613. Configure here.

@marwan116

Copy link
Copy Markdown
Contributor Author

Closing: folded into #67924 (samples-dataset checks) and #67925 (row schema) to keep the stack at one concern per PR.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-sample-steps branch from 92d18d4 to 1f257ca Compare November 7, 2026 01:22
@marwan116 marwan116 closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant