Skip to content

[Data] read_mcap: decode video frames of message rows in the read task - #67193

Draft
marwan116 wants to merge 1 commit into
marwan/mcap-dsv2-attachmentsfrom
marwan/mcap-dsv2-video-messages
Draft

marwan116 wants to merge 1 commit into
marwan/mcap-dsv2-attachmentsfrom
marwan/mcap-dsv2-video-messages

Conversation

@marwan116

@marwan116 marwan116 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

Problem.

  • A read of paths returned each video message as an encoded payload. A frame needs the frames back to its keyframe to decode, and a read task can start mid-GOP, so a row could not be decoded on its own.

Change.

rows = ray.data.read_mcap(
    "s3://bucket/recordings/",
    topics={"/cam/front"},
    video=VideoOptions(resize=(224, 224)),
)
# frame  uint8 (224, 224, 3), in place of data
  • A task that starts mid-GOP first reads back to the last keyframe at or before its first frame (RAY_DATA_MCAP_MAX_LEAD_IN_S back at most). It decodes those frames without emitting them. The read-back goes only as far as the keyframe, nearest chunk first, and cameras that share chunks parse each chunk once.
keyframes   K . . . . K . . . . K
task 2's chunks           [ . . . K . . ]
read back from            K
emitted                     [ . . . K . . ]
  • A frame goes to the message stamped with its log time, in whatever order the decoder releases it, so a stream with B-frames still gives each message one row with its own frame. A frame never lands on another message's row.
  • The rows are the same whether or not frame is selected. A picture the codec rejects, or whose frame never comes out, gets a null frame.
  • With a row limit, decoding stops once enough rows are owed.
  • Decode warnings name the topic and the file.
  • Every frame becomes a row, however the file is split into tasks. A picture with no keyframe in reach has a null frame.
  • With max_errored_blocks set, a corrupt chunk a task owns is recorded and skipped, as a plain read does. Its cameras have null frames until their next keyframe, and a look-back stops at a corrupt chunk.
  • For a lower rate, a sample read picks the frames: steps.across(fps=...) over clips cut from each recording.
  • Planning fails early when a selected channel is not decodable video, naming the topic. The frame column's type follows from resize alone, so planning decodes no keyframe. Without resize, the column is variable-shaped. With resize, it is fixed-shape, in the DataContext's tensor format.
  • count() of decoded rows reads the files, and attachment and metadata reads refuse video.

Decisions to confirm:

  • Message rows have no rate option, so fps means only a TopicSpec's step rate. A lower rate goes through a sample read.

Related issues

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py \
  python/ray/data/tests/unit/datasource_v2/test_file_partitioners.py

737 passed, 32 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from d8d3b45 to 12ece38 Compare October 20, 2026 16:26
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@marwan116
marwan116 added this pull request to stack #67196 October 20, 2026 16:31

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 12ece38 to 86820c4 Compare October 20, 2026 16:51
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 86820c4 to 4dfecca Compare October 20, 2026 17:14
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch 2 times, most recently from e737c9f to 5be43d0 Compare October 21, 2026 04:58
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 5be43d0 to f1b6a45 Compare October 29, 2026 14:07
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from f1b6a45 to 13bb72d Compare October 29, 2026 14:16
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 13bb72d to dcdb40a Compare October 29, 2026 14:37
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 201655c to 6c226ac Compare November 7, 2026 08:16
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 6c226ac to 0c77a97 Compare November 7, 2026 08:55
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 0c77a97 to acdca68 Compare November 7, 2026 10:01
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from acdca68 to 8352421 Compare November 7, 2026 10:42
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 8352421 to 1e8637f Compare November 7, 2026 11:22
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 1e8637f. Configure here.

@marwan116 marwan116 added the go add ONLY when ready to merge, run all tests label Nov 7, 2026
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 1e8637f to 55895a0 Compare November 8, 2026 23:34
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 55895a0 to 25ed92a Compare November 9, 2026 02:07
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 25ed92a to 63aa1ac Compare November 9, 2026 05:45
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-video-messages branch from 63aa1ac to 31cc5fc Compare November 9, 2026 15:18
Problem:
- A read of paths returned each video message as an encoded payload.
  A frame needs the frames back to its keyframe to decode, and a read
  task can start mid-GOP, so a row could not be decoded on its own.

Change:
- read_mcap(paths, video=VideoOptions(resize=...)) decodes
  the video topics in the read task. Each video message becomes a row
  with a frame column, a uint8 (height, width, 3) image, in place of
  data; video_topics lists topics with unknown schema names.
- A task that starts mid-GOP first reads back to the last keyframe at
  or before its first frame (RAY_DATA_MCAP_MAX_LEAD_IN_S back at
  most), and decodes those frames without emitting them.
- Every frame becomes a row, however the file is split into tasks. A
  picture with no keyframe in reach has a null frame. For a lower
  rate, a sample read with steps.across(fps=...) over clips of each
  recording picks the frames.
- With max_errored_blocks set, a corrupt chunk a task owns is recorded
  and skipped, as a plain read does, and its cameras have null frames
  until their next keyframe. A look-back stops at a corrupt chunk.
- Planning fails early when a selected channel is not decodable video.
  Without resize, it types the frame column from the first keyframes
  of the sampled files.
- count() of decoded rows reads the files, and attachment and metadata
  reads refuse video.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant