Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements in-task video decoding for the V2 MCAP reader, allowing compressed video payloads (H.264, H.265, JPEG, PNG) to be decoded directly into RGB frames within read tasks. The feedback highlights several critical robustness improvements, including handling duplicate log_time values to prevent row loss, adding a total record scan limit for unindexed files to avoid driver hangs, catching broader av.FFmpegError exceptions during decoding, and clamping calculated start times to zero to prevent negative unsigned 64-bit MCAP timestamps.
ddae281 to
9a273ea
Compare
9a273ea to
0093c82
Compare
fd1b2ac to
93284ef
Compare
93284ef to
a922e83
Compare
a922e83 to
251009d
Compare
251009d to
d367fb7
Compare
d367fb7 to
fad7f6d
Compare
fad7f6d to
6b8a9ee
Compare
6b8a9ee to
f708d4c
Compare
f708d4c to
f032d1d
Compare
f032d1d to
4f27e29
Compare
commented
Nov 7, 2026
|
bugbot run |
1 similar comment
commented
Nov 7, 2026
|
bugbot run |
commented
Nov 7, 2026
|
bugbot run |
commented
Nov 7, 2026
|
bugbot run |
commented
Nov 7, 2026
|
bugbot run |
commented
Nov 7, 2026
|
bugbot run |
| ``add`` checked share one size, or of its black frames when it has | ||
| none.""" | ||
| first = next((frame for frame in row if frame is not None), None) | ||
| return self._black_size() if first is None else first.shape[:2] |
There was a problem hiding this comment.
Padded rows inherit other samples' size
Medium Severity
A fully padded decoded row takes _black_size from RunDecoder.frame_size, which is whatever that camera last decoded in this file. Grouped with an earlier clip, those steps become full-size black frames; the same sample read alone stays 0x0. That breaks the invariant that a sample's frames do not depend on which samples share the task, and it can also trip RAY_DATA_MCAP_MAX_ROW_BYTES only on the grouped read.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit feb4fe4. Configure here.
left a comment
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9f44c60. Configure here.
Problem: - A sample read returned each camera step as an encoded payload. An H.264 or H.265 frame needs the frames back to its keyframe to decode, so a row could not be decoded on its own, and users decoded whole streams again downstream. Change: - read_mcap(samples=..., video=VideoOptions(resize=...)) decodes the video topics in the read task. Each video topic's steps become a frames:<topic> uint8 tensor (steps, height, width, 3) in place of data:<topic>; video_topics lists topics with unknown schema names, and needs video. - Samples whose steps meet form a group, and each camera is decoded once per group, from the last keyframe at or before its first frame (RAY_DATA_MCAP_MAX_LEAD_IN_S back at most). Only the frames that steps take become RGB. - A group ends before its decoded frames would pass twice the target block size (FRAME_BUDGET_BLOCKS = 2). Until a camera's frame size is known, the first sample that takes its frames is a group of its own. A read without decoding builds each row as its sample resolves. - With video and no resize, each read task reserves that budget as memory, 256 MiB by default. The caller's memory, or memory in ray_remote_args, wins. With resize, nothing is reserved. - A row's frame bytes are checked against RAY_DATA_MCAP_MAX_ROW_BYTES before its group decodes, and again as soon as a frame of a new size decodes. A camera's size change within a row fails, naming the sample. - A camera's decoder keeps running into the next group when no keyframe lies between them, so back-to-back clips feed each frame once. - Every step rule works on decoded topics, except match.all_in_step, which a decoded topic refuses at planning: a step holds one frame. - Planning opens the first samples' files to settle which topics are decoded, in the same probe that sizes the read tasks, and without the checkpoint config. The estimate reuses the summaries it read. - A file without a chunk index keeps every message of a decoded topic, since its decoder may need any of them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>


Description
Problem.
Change.
resize, each read task reserves that budget as memory, 256 MiB by default, unlessmemoryis given. Until a camera's frame size is known, the first sample that takes its frames is a group of its own. A read without decoding builds each row as its sample resolves.RAY_DATA_MCAP_MAX_ROW_BYTESbefore its group decodes, and again as soon as a frame of a new size decodes. A camera's size change within a row fails, naming the sample.match.all_in_step(), which a decoded topic refuses at planning: a step holds one frame.topicsthat those files do not record is looked for in the later sampled files, through their summaries. Planning shares the probe that sizes the read tasks ([Data] read_mcap: read training and labeling samples, one row per sample #67189), and runs without the checkpoint config, so it does not warn that checkpointing is disabled. The task-size estimate reuses the summaries planning read.video_topics.frames:column type follows theDataContext's tensor format.resizewhen it is set. A row added before any frame of its camera decodes has 0x0 frames, so theRAY_DATA_MCAP_MAX_ROW_BYTEScheck sees the bytes the block builds.flush_buffers, without decoding them.avor Pillow fails with an error that says what to install. Decode warnings name the topic and the file, once per topic.avis imported through animportlibhelper, and the test imports carry inline ignores.pyrefly.tomltherefore does not change, and CI selects only the data tests.video_topicslists video topics whose schema namesread_mcapdoes not know. It needsvideo.Measured on S3: 1,024 one-minute recordings, two H.264 cameras decoded to 224×224, 2 s clips, 128 CPUs. A
WindowSpecwindow decoded every frame of its range; a sample decodes only the frames its steps take.The S3 rows were measured without the frame budget. It trades some speed for memory: on a 720p file, one task's peak RSS is 1.05 GiB against 2.3 GiB without it, and its read takes 4.2 s against 3.7 s.
Decisions to confirm:
videodecodes them in the task.avonly on Python 3.12, and the premerge data jobs run Python 3.10. The options are to pinav==15.1.0for Python below 3.12 and regenerate the data locks, or to label the PRcontinuous-build.Related issues
Additional information
591 passed, 30 skipped. pre-commit and pyrefly clean.
AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code