Skip to content

[Data] read_mcap: decode video frames of sample rows in the read task - #66670

Open
marwan116 wants to merge 1 commit into
marwan/mcap-dsv2-samplesfrom
marwan/mcap-dsv2-video
Open

marwan116 wants to merge 1 commit into
marwan/mcap-dsv2-samplesfrom
marwan/mcap-dsv2-video

Conversation

@marwan116

@marwan116 marwan116 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

Problem.

  • A sample read returned each camera step as an encoded payload. An H.264 or H.265 frame needs every frame back to its keyframe to decode, so a row could not be decoded on its own.
  • Users decoded whole streams again downstream, and kept frames they then threw away.

Change.

from ray.data.datasource.mcap import TopicSpec, VideoOptions, steps

rows = ray.data.read_mcap(
    samples=samples,
    topics={"/cam/wrist": TopicSpec(steps.before(0.5, fps=10))},
    video=VideoOptions(resize=(224, 224)),
)
# frames:/cam/wrist  uint8 tensor (steps, 224, 224, 3), in place of data:/cam/wrist
  • Samples whose steps meet form a group. Each camera is decoded once per group, from the last keyframe at or before its first frame, and only the frames that steps take become RGB.
  • A camera's decoder keeps running into the next group when no keyframe lies between them, so back-to-back clips feed each frame to the decoder once.
  • A group ends before its decoded frames would pass twice the target block size. Without resize, each read task reserves that budget as memory, 256 MiB by default, unless memory is given. Until a camera's frame size is known, the first sample that takes its frames is a group of its own. A read without decoding builds each row as its sample resolves.
  • A row's frame bytes are checked against RAY_DATA_MCAP_MAX_ROW_BYTES before its group decodes, and again as soon as a frame of a new size decodes. A camera's size change within a row fails, naming the sample.
  • A file without a chunk index keeps every message of a decoded topic, since its decoder may need any of them.
keyframes      K . . . . . . . . . K . . . . . . . . . K
clip 1             [-----]
clip 2                    [-----]
before         decode K→clip 1, then a fresh decoder K→clip 2 again
now            decode K→clip 1, then keep going into clip 2
  • Every step rule of [Data] read_mcap: read training and labeling samples, one row per sample #67189 works on decoded topics, except match.all_in_step(), which a decoded topic refuses at planning: a step holds one frame.
  • Planning settles which topics are video from the first message of each selected channel in the first four sampled files. A topic named in topics that those files do not record is looked for in the later sampled files, through their summaries. Planning shares the probe that sizes the read tasks ([Data] read_mcap: read training and labeling samples, one row per sample #67189), and runs without the checkpoint config, so it does not warn that checkpointing is disabled. The task-size estimate reuses the summaries planning read.
  • A known video topic that planning never met is read undecoded. The read warns once, naming the topic and suggesting video_topics.
  • The frames: column type follows the DataContext's tensor format.
  • A black padding frame takes the camera's decoded size as known when its row is added, or resize when it is set. A row added before any frame of its camera decodes has 0x0 frames, so the RAY_DATA_MCAP_MAX_ROW_BYTES check sees the bytes the block builds.
  • A message that holds only parameter sets is not a step. The parameter sets where a stream starts are read once per channel.
  • Each decoder runs one FFmpeg thread, since a read task reserves one CPU.
  • When a row limit stops the read, the decoders are closed explicitly. Closing drops the frames a decoder holds with flush_buffers, without decoding them.
  • A worker without av or Pillow fails with an error that says what to install. Decode warnings name the topic and the file, once per topic.
  • av is imported through an importlib helper, and the test imports carry inline ignores. pyrefly.toml therefore does not change, and CI selects only the data tests.
  • video_topics lists video topics whose schema names read_mcap does not know. It needs video.

Measured on S3: 1,024 one-minute recordings, two H.264 cameras decoded to 224×224, 2 s clips, 128 CPUs. A WindowSpec window decoded every frame of its range; a sample decodes only the frames its steps take.

rows/s read CPU-s peak worker RSS
fixed windows (the earlier design) 123 12,000 802 MB
samples 183 6,900 864 MB

The S3 rows were measured without the frame budget. It trades some speed for memory: on a 720p file, one task's peak RSS is 1.05 GiB against 2.3 GiB without it, and its read takes 4.2 s against 3.7 s.

Decisions to confirm:

  • No keyframe lead-in in a raw (undecoded) camera step: a row carries the frames its steps take, and video decodes them in the task.
  • One FFmpeg thread per decoder. A busy node spends less CPU and runs fewer threads, but a single stream on an idle node decodes slower. For scale: about 10% less CPU and 10 fewer threads per decoder, against about 3x slower on an idle node.
  • The decode tests skip in premerge CI. CI installs av only on Python 3.12, and the premerge data jobs run Python 3.10. The options are to pin av==15.1.0 for Python below 3.12 and regenerate the data locks, or to label the PR continuous-build.

Related issues

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py \
  python/ray/data/tests/unit/datasource_v2/test_file_partitioners.py

591 passed, 30 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116
marwan116 requested a review from a team as a code owner October 2, 2026 15:48
@marwan116 marwan116 added the data Ray Data-related issues label Oct 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements in-task video decoding for the V2 MCAP reader, allowing compressed video payloads (H.264, H.265, JPEG, PNG) to be decoded directly into RGB frames within read tasks. The feedback highlights several critical robustness improvements, including handling duplicate log_time values to prevent row loss, adding a total record scan limit for unindexed files to avoid driver hangs, catching broader av.FFmpegError exceptions during decoding, and clamping calculated start times to zero to prevent negative unsigned 64-bit MCAP timestamps.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch 2 times, most recently from ddae281 to 9a273ea Compare October 2, 2026 16:17

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch from 9a273ea to 0093c82 Compare October 2, 2026 16:35

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch 3 times, most recently from fd1b2ac to 93284ef Compare October 2, 2026 16:52

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch from 93284ef to a922e83 Compare October 2, 2026 17:06

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch from a922e83 to 251009d Compare October 2, 2026 17:26

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch from 251009d to d367fb7 Compare October 2, 2026 17:39

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch from 6b8a9ee to f708d4c Compare October 4, 2026 18:05

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch from f708d4c to f032d1d Compare October 4, 2026 18:20

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video branch from f032d1d to 4f27e29 Compare October 4, 2026 18:34

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116

ghost commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

1 similar comment
@marwan116

ghost commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116

ghost commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116

ghost commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116

ghost commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116

ghost commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

bugbot run

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

``add`` checked share one size, or of its black frames when it has
none."""
first = next((frame for frame in row if frame is not None), None)
return self._black_size() if first is None else first.shape[:2]

ghost Nov 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Padded rows inherit other samples' size

Medium Severity

A fully padded decoded row takes _black_size from RunDecoder.frame_size, which is whatever that camera last decoded in this file. Grouped with an earlier clip, those steps become full-size black frames; the same sample read alone stays 0x0. That breaks the invariant that a sample's frames do not depend on which samples share the task, and it can also trip RAY_DATA_MCAP_MAX_ROW_BYTES only on the grouped read.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit feb4fe4. Configure here.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

ghost left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9f44c60. Configure here.

Problem:
- A sample read returned each camera step as an encoded payload. An
  H.264 or H.265 frame needs the frames back to its keyframe to decode,
  so a row could not be decoded on its own, and users decoded whole
  streams again downstream.

Change:
- read_mcap(samples=..., video=VideoOptions(resize=...)) decodes the
  video topics in the read task. Each video topic's steps become a
  frames:<topic> uint8 tensor (steps, height, width, 3) in place of
  data:<topic>; video_topics lists topics with unknown schema names,
  and needs video.
- Samples whose steps meet form a group, and each camera is decoded
  once per group, from the last keyframe at or before its first frame
  (RAY_DATA_MCAP_MAX_LEAD_IN_S back at most). Only the frames that
  steps take become RGB.
- A group ends before its decoded frames would pass twice the target
  block size (FRAME_BUDGET_BLOCKS = 2). Until a camera's frame size is
  known, the first sample that takes its frames is a group of its own.
  A read without decoding builds each row as its sample resolves.
- With video and no resize, each read task reserves that budget as
  memory, 256 MiB by default. The caller's memory, or memory in
  ray_remote_args, wins. With resize, nothing is reserved.
- A row's frame bytes are checked against RAY_DATA_MCAP_MAX_ROW_BYTES
  before its group decodes, and again as soon as a frame of a new
  size decodes. A camera's size change within a row fails, naming the
  sample.
- A camera's decoder keeps running into the next group when no
  keyframe lies between them, so back-to-back clips feed each frame
  once.
- Every step rule works on decoded topics, except match.all_in_step,
  which a decoded topic refuses at planning: a step holds one frame.
- Planning opens the first samples' files to settle which topics are
  decoded, in the same probe that sizes the read tasks, and without
  the checkpoint config. The estimate reuses the summaries it read.
- A file without a chunk index keeps every message of a decoded
  topic, since its decoder may need any of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant