Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces filter pushdown for 'topic' and 'log_time' conjuncts and metadata-based row counting for the V2 MCAP datasource in Ray Data. By folding these filters into the MCAP selection, the file indexer can prune files and chunks during listing, and the reader can avoid reading unnecessary payloads. Additionally, the reader now supports metadata-based row counting to answer count queries efficiently using MCAP statistics. The review feedback correctly identifies a potential correctness issue in 'read_metadata' where message counts could be over-counted or double-counted if a file's chunks are split across multiple manifests or if some chunks are excluded, as it currently sums the entire file's statistics.
bf84a43 to
00c9095
Compare
00c9095 to
7a194ff
Compare
7a194ff to
1c0d1c9
Compare
1c0d1c9 to
43e2723
Compare
43e2723 to
86a68dc
Compare
86a68dc to
5717191
Compare
5717191 to
345de8f
Compare
345de8f to
5613406
Compare
5613406 to
789c37b
Compare
Problem: - read_mcap switches to the V2 datasource by default, but its docstring did not say so, or how its rows differ. - The use_datasource_v2 docstring did not name MCAP. Change: - The read_mcap docstring lists the three differences from the legacy reader, how to switch back, and an example that stores schema_data as plain bytes. - The use_datasource_v2 docstring names MCAP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The V2 warning names override_num_blocks even when the caller passed parallelism. - It hard-codes 128 MiB instead of the configured budget. Change: - Name the argument the caller gave. - Print the budget from RAY_DATA_MCAP_BIN_PACKING_BYTES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The chunk-fetch test ran four read configs, but one is its own baseline and two make the same single read on the test file. - The parallelism warning had its own test beside the override_num_blocks test, which varied only the value. - The generated-id test ran two names that hit the same raise. Change: - Compare one chunk per read with small reads only. - Parametrize the override_num_blocks test over the argument name and drop the separate parallelism test. - Check the generated-id raise with the name row_id only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - Listing threads read and parsed summaries, and up to twice the summary concurrency of parsed summaries were held at once. Parsed summaries are much larger than their bytes, and nothing bounded their size. Parsing holds the GIL, so the threads gained nothing from it. - A read task's peak was estimated as 1.2 times its table. That misses the chunk being decompressed and parsed, the log-time heap and the ranged read, so a task of large chunks or small messages peaked at up to 2.6 times its estimate. Change: - Summary reads are split in two: a tail read, whose footer gives the section's exact length, and a section read. Listing threads only fetch bytes, and the caller parses each section in order. - A section is fetched once it fits the read task budget (RAY_DATA_MCAP_BIN_PACKING_BYTES) with the sections held. With nothing held, the next one is fetched whatever its size. A failed read is still that file's own error. - A read task's peak is estimated as its table, plus the largest working set of one chunk, plus one ranged read. Each chunk weighs its table scaled so that a full task leaves room for the rest. - New tests pin the estimate's constants by measurement, check the estimate against a measured read, and check the byte budget of summary reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The estimate counts a payload's raw bytes, but a JSON payload is decoded into Python values that hold several times as many bytes until its table is built. A read of small JSON messages peaks above the estimate. Change: - When every selected channel of a file is JSON-encoded, each payload byte counts the reader's own bytes-per-payload-byte for decoded JSON. - The peak test also reads JSON payloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - An ordered read (preserve_order) keeps each file in one read task, so a large recording is read by one task and held in memory whole. - In other reads the packer places a file's chunks one by one, so a small file can be split across two shared tasks. Change: - OrderedFileCutter replaces OrderedFileGrouper. It cuts each large file into read tasks and sends them at once, in order. It cuts only where every chunk before the cut ends no later than every chunk after it starts, so with preserve_order each file still comes out in log-time order from start to end. - In a recording with one writer every boundary allows a cut, so tasks are as full as in an unordered read. Where chunks overlap in time, as when cameras and robot state are written to chunks of their own, the overlapping stretch stays in one task. - A small file goes to the packer whole, so it is never split. - A file whose rows cannot be placed in time goes to the packer as listed: one read whole, an unreadable one, or rows without a chunk index record. - With log_time_order=False a file is cut in file order, where every boundary keeps the order. - The log_time_order docstring says where an ordered read cuts a file. - New tests: the cuts and the tasks' summary records. The preserve_order read of three files now checks that each file is split. Tests that force small tasks now check that they still split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A file whose chunks overlap in time has few cuts that keep log-time order, so it was read by one task on every read. Without preserve_order its blocks come out in any order anyway, so that one task bought nothing and cost parallelism. - In an ordered read, that task's merge holds every chunk that overlaps the current log time. A file whose chunks all overlap is held whole, and nothing told the user. Change: - The cutter keeps a file's cuts in log-time order only when the read keeps its blocks in order. It reads DataContext.execution_options.preserve_order in the listing task, which runs under the dataset's own context. - Otherwise every boundary allows a cut. Chunks are still taken by first log time, so each task spans one stretch of time. - A recording with one writer is cut the same way in either case. - On a tied log time, an ordered cut is allowed only when the chunks before it that end then lie earlier in the file, since the merge yields equal log times in file order. - An ordered task whose overlapping chunks would hold more than four read-task budgets at once logs one warning per process. It names the file and suggests read_mcap(memory=...) or dropping preserve_order. - The warning stays quiet when each read task already reserves at least that much. read_mcap passes memory, or else ray_remote_args["memory"], to the datasource, which hands it to the cutter. - The read_mcap docstring says that an overlapping stretch stays in one task only with preserve_order, that the task holds it whole, and when the warning fires. - New tests: an unordered read may cut a file whose chunks overlap in time at any boundary; the warning fires once for a file with no clean cut, and not for shallow overlap, an unordered read, one big chunk or a reservation that covers it; read_mcap's memory reaches the cutter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - With preserve_order, the cutter sent a heavy file's tasks at once and left light files in the packer's open bins. A large file listed after small ones came out first. Change: - With preserve_order, the cutter packs light files and files it cannot place in time itself, in listing order. A heavy file's tasks leave after the files listed before it. - Without preserve_order, light files go to the packer as before. - The read_mcap docstring says files come out in listing order with preserve_order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
This is 2a's fix applied to 2c's code. The 2c-specific part is the docstring of OrderedFileCutter's _rows_by_file. Problem: - The reader merged the manifest rows of one path. A path listed twice was read once when its copies shared a read task and twice when they did not, so the row count depended on the task budget. - The packer keeps one row per listed copy, and Parquet reads a repeated path twice. - Three fallbacks could turn off the summary handoff unnoticed: a missing task summary column, a path named by two rows, and a listing without record columns all made read tasks read summaries again. Change: - The reader reads each manifest row on its own, in manifest order. - task_summaries asserts the task summary column, and the partitioner asserts the listing record columns. A null record only marks a file the listing could not read. - Tests pack listing rows into a read task (as_read_task) instead of reading listing manifests directly. - OrderedFileCutter's _rows_by_file states why copies stay apart: each listing manifest holds one listed file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The cutter checked for a missing chunk index column and for rows naming several chunks. The listing never makes such rows, and TaskSummaryPartitioner reads the column before the cutter runs. - Several cutter tests repeated what other tests already assert. - The memory test mocked MCAPDatasourceV2.__init__ to see one argument. Change: - Read the chunk index column directly, and drop the one-chunk check. - Drop the empty-record and two-chunk rows from unplaceable_files(). - Drop tests whose cases other tests cover, and merge the light-file test and the two cut-anywhere tests into parametrized tests. - Check the memory reservation on the cutter that read_mcap's plan holds, with no mock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - An ordered read warned only once its log-time merge held four read task budgets. Nothing measured chose four, and a task with eight overlapping chunks peaked at 2.5 budgets with no warning. - The warning named `preserve_order`, but the merge comes from `log_time_order`, and it gave no value to reserve. Change: - Warn once the merge holds more than the budget a task is sized for. - Name the reservation to pass, `read_mcap(memory=...)` rounded up to a whole MiB, and `log_time_order=False` as the other way out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - An ordered log-time read cut a large file only where its log-time order held. That needed per-chunk end times, a tied-time rule and a memory warning with a reservation handed down from read_mcap. - The cuts bought parallelism, not memory. A whole-file task holds one ranged read, one chunk and one block at a time, whatever the file's size. Chunks that overlap in time were never cut apart, so a task held them together either way. Change: - With preserve_order and log_time_order, each file is read whole by one task, in listing order. - Other reads still cut a large file at the budget, and files still keep their listing order with preserve_order. - Remove the ordered-task memory warning, read_task_memory on the datasource, and their tests. - chunk_start_time replaces chunk_log_times, since only the first log time is still read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A map task buffers its output until a block reaches its target size. When the task raised, its buffered rows were dropped, even when max_errored_blocks let the job skip the error. A read task that hit one corrupt file lost the rows of the files it had read before it. Change: - When the job skips errors (max_errored_blocks is not 0) and the task cannot run again, _BlockShapingIterator emits the buffered rows as a last block, then raises the error. - A task that may run again raises without a partial block: on an error that retried_map_errors matches while _map_task has attempts left, on a RaySystemError, or on an error the task's retry_exceptions names (or one it wraps). An actor task's retry_exceptions come from actor_task_retry_on_errors. A rerun skips as many blocks as were already emitted, so a partial block would hide rows of the rerun. - TaskContext carries the task's Ray retry_exceptions (False when max_retries is 0), and _map_task sets each attempt's retries left and retried_map_errors patterns on it. - BlockOutputBuffer.has_buffered_rows says whether rows are waiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - The listing cannot isolate a file whose summary reads but whose chunk is corrupt. That chunk failed the read task holding it: the reader stopped at the bad file, and the small files sharing the task lost their rows too. Change: - When max_errored_blocks lets errors be skipped, the reader skips a corrupt chunk and reads the rest of its file. Each chunk is compressed on its own and found through the chunk index, so one bad chunk costs only its own rows. - A chunk read back for a declaration that does not decode is skipped, and the look-back goes on to the next earlier chunk, so the chunk being read is not blamed for it. - A retried file read starts with no corrupt chunks recorded. - A file that fails as a whole does not stop the task's other files. A file without a chunk index still fails whole, and an error reading the file still raises, so the read retry applies. - Once the task's files are read, it raises one error naming every file and chunk that failed, so the task counts once against max_errored_blocks. With max_errored_blocks=0 the first failure still fails the task at once. - With the map task's flush of buffered rows, the job keeps every other row, also when a map is fused into the read. - New tests: a chunk whose payload does not decompress and one whose record does not parse, in both read orders, also when fetched in one ranged read with its neighbours. End to end, a file with two corrupt chunks loses only their rows under max_errored_blocks=1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - An actor task counted as rerun on any error its retry_exceptions names, even when its actor's max_task_retries is 0. Ray then does not rerun it, so a skipped error dropped the task's buffered rows. Change: - An actor task counts as rerun only while max_task_retries allows it. The task option overrides the actor's; Ray defaults it to 0, as it stays with max_restarts=0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A map task counted as rerun on every RaySystemError, even with max_retries=0 or max_task_retries=0. Ray then does not rerun it, so a skipped error dropped the task's buffered rows. Change: - RaySystemError counts as rerun only through the task's retry_exceptions, which Ray Data always extends with it and which are empty when retries are off. - The task-pool retry check adds RaySystemError as cached_remote_fn does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…error Problem: - A fused downstream batcher dropped the rows it held when an upstream stage raised an error the job skips. Map then MapBatches(batch_size=4) kept rows 0 to 3 of 0 to 6. - A TaskContext built without retry settings meant "never rerun", so a shuffle reduce task emitted partial output it did not opt in to. - The retry settings were sentinels on TaskContext, and the pattern check was a copy of iterate_with_retry's. - The max_errored_blocks docs still said failed blocks are dropped. Change: - A batch transform ends its input at an error the task skips, hands on a short last batch, then raises the error. - A frozen RerunPolicy replaces the TaskContext retry fields. Its default reruns on any error; map operators and _map_task fill it in. - iterate_with_retry and RerunPolicy share is_retryable_error. - _emit_rows_before_error is renamed _should_emit_buffered_rows. - The context docstring and the docs page say which rows are kept. - Dataset-level tests cover a skipped map_batches error and a fused batcher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…once Problem: - A corrupt chunk read back for a declaration was always skipped. When it alone declared a schema, rows came out with no schema, even with max_errored_blocks=0. - Any error while a chunk was read, such as the reader's own missing channel error, a MemoryError or a bug, was reported as a skipped corrupt chunk. - A task joined every failed file into one ValueError caused by the first. A later OSError was hidden, so the task was skipped instead of rerun. Change: - is_corrupt_data_error says which errors mean the bytes do not decode. Only those are skipped, and only around decoding a chunk. - A chunk read back for a declaration is skipped only when corrupt chunks are skipped. A schema still missing after the look-back raises, naming the corrupt chunk. - A file's corrupt data is deferred to the task's one error. Any other error raises at once. _read_owned_file holds one file's read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…uplicates Problem: - Ten unit tests each ran one rerun policy and checked whether the buffered rows came out first. - The errored-rows test ran max_errored_blocks 1 and -1, which take the same path. - Two read_mcap tests also ran with a fused map, which the fused batcher test already covers. Change: - Make the ten tests one test over a small case dataclass of the rerun policy and whether the rows come out, with readable ids. - Let the failing transformer helper raise a given error type, and drop the separate system-error transformer. - Keep only max_errored_blocks=-1, which also catches a check that skips only on a positive limit. - Drop the fused_map parameter from the two read_mcap tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - With max_errored_blocks != 0, one message whose payload is not valid JSON dropped the rest of its file, rows already built included. - The error did not say that the rest of the file was dropped. Change: - decode_payload raises InvalidJsonPayload, a ValueError, for a payload that is not valid JSON. A wrong message encoding still raises a ValueError that fails the file. - When errors can be skipped, the reader skips that one message and keeps every other row of the file, as it does for a corrupt chunk. - One error per file names the skipped chunks by byte offset, counts the skipped messages and names the first by row id, so the task counts once. The record keeps only the first message. - The table builder decodes a payload before it adds anything, so a skipped message leaves it consistent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - With max_errored_blocks set, a message that is not valid JSON used up a place in the row limit, so a limited read could return fewer rows than asked although later messages were valid. - A file that failed mid-way, for example on a topic of another encoding, dropped the rows already built into its current table. What the file kept depended on the block size. Change: - The limit counts rows built. A skipped message frees its place, and no message is pulled once the limit is met. - An ordered scan of a file without a chunk index holds every message when it can skip, so a skipped one cannot shorten the result. - When errors are skipped, a file error yields the rows built before it, then fails the rest of the file as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A missing schema raised for every message, before the selection check ran. - So a topic the read filters out could fail the rest of its file when its schema lived only in a skipped corrupt chunk. The later selected rows were lost and a max_errored_blocks slot was spent. Change: - Split the channel and schema lookups. A message whose channel the listing or the topic filter rejects, or whose log time is out of range, is skipped before its schema is looked up. - A message that may be read still fails on a missing schema and names the corrupt chunk read back for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A read that skips corrupt chunks keeps each chunk's error until the file is done. An error's traceback holds its chunk's bytes, so every corrupt chunk adds its size to the read's memory. Change: - Keep the byte offsets of a file's corrupt chunks and only the error at the lowest offset. The error the read raises is unchanged. - A test reads a file whose every chunk is corrupt and bounds its peak. - Cut four reader tests whose cases other tests already cover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…ples checks Problem: - A sample read takes two inputs: a TopicSpec per topic, saying where its steps fall and which message fills each, and a dataset of samples. Both need a shape and checks before anything reads a file. Change: - TopicSpec(steps, match=None, pad="edge"), exported from ray.data.datasource.mcap beside steps and match. - ray.data.datasource.mcap.steps: before/after (1/fps apart around at="anchor", "clip_start" or "clip_end"), across, at, and last/next/every, which take the recorded messages themselves. - ray.data.datasource.mcap.match: nearest (default), backward, all_in_step. - mcap_samples.check_samples_schema: files, clip_start and clip_end with the right types, an anchor when a topic measures from it, and no column the read adds. - mcap_samples.sample_manifest: a batch of samples as manifest rows, one file each, failing on the first bad sample by its sample_id or its file and clip. - No I/O. The read that uses them comes later in the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Given a sample (clip and anchor) and a topic's TopicSpec, a sample
read must decide which recorded message fills each step, what a step
no message fills holds, and what the row looks like.
Change:
- mcap_samples.TopicPlan: a TopicSpec compiled for one topic, with the
span of log times its steps can reach for a sample.
- resolve_steps / resolve_slots: the message (or, for all_in_step, the
messages) that fill each step. nearest never looks across the
anchor; backward takes the last message at or before each step.
- Steps no message fills are padded by pad="edge" or "zeros", or fail
with pad="raise", naming the sample.
- sample_row_schema: the samples' own columns, then each topic's data:,
times: and is_pad: lists. is_pad is True for a padded step, as in
read_lerobot's {key}_is_pad.
- Pure functions over log times: no I/O. The reader that feeds them
comes later in the stack.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A sample row needs each topic's messages around its anchor, read
from the recording's chunks. Reading each sample on its own reads
the same chunks many times when samples lie close together.
- The chunks a task holds at once must stay within its budget.
Change:
- mcap_sample_rows: a read task takes its samples file by file, in
anchor order, and groups samples whose windows meet into runs. A
run ends before its chunks would hold more than a read task's
budget (RAY_DATA_MCAP_BIN_PACKING_BYTES). Each run reads the chunks
its topics' steps reach once (ChunkTimeline), and forgets chunks
the next run does not need.
- A file without a chunk index is scanned once (WholeFileTimeline).
The scan keeps only what steps can take: the runs' spans, the
nearest messages around each span, and the first and last messages
inside each sample's bounds.
- A corrupt chunk fails the read task. Ray's max_errored_blocks then
decides whether the task's block is dropped or the read fails.
- SampleRows builds one row per sample: the sample's columns, then each
topic's data:, times: and is_pad: lists. A row over
RAY_DATA_MCAP_MAX_ROW_BYTES fails the read, naming the sample.
- mcap_summary.topic_chunks: the chunks that may hold a topic, one rule
for the reader and, later, for sizing tasks.
- MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...) sets
up a sample read: its schema is the sample row's, and its reader
builds sample rows from manifest rows that name each sample's file.
read_mcap wires it up in the next PR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Robot learning reads examples, not messages: a clip of a recording,
or a moment inside one, with each topic read over its own window.
- The stack below adds the spec, step resolution, sample checks and
the sample reader; nothing reaches them from read_mcap yet.
Change:
- read_mcap(samples=ds, topics={topic: TopicSpec(...)}) reads one row
per sample. The samples feed ReadFiles as its manifest; planning
lists no files. Options that only a read of paths takes are refused.
- Planning runs the samples once: their schema comes from their plan
when it is known, else from the samples probed for task size.
- A task takes as many samples as read about
RAY_DATA_MCAP_BIN_PACKING_BYTES of chunks, estimated from the first
256 samples (mcap_sample_tasks).
- A checkpoint keyed on a sample column skips finished samples before
their files are opened: ReadFiles(checkpoint_on_input=True), a
checkpoint filter only at the read nearest the sink, and ReadFiles
skipping empty manifest blocks. A checkpoint keyed on a column the
samples lack, such as row_id, is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A sample read returned each camera step as an encoded payload. An H.264 or H.265 frame needs the frames back to its keyframe to decode, so a row could not be decoded on its own, and users decoded whole streams again downstream. Change: - read_mcap(samples=..., video=VideoOptions(resize=...)) decodes the video topics in the read task. Each video topic's steps become a frames:<topic> uint8 tensor (steps, height, width, 3) in place of data:<topic>; video_topics lists topics with unknown schema names, and needs video. - Samples whose steps meet form a group, and each camera is decoded once per group, from the last keyframe at or before its first frame (RAY_DATA_MCAP_MAX_LEAD_IN_S back at most). Only the frames that steps take become RGB. - A group ends before its decoded frames would pass twice the target block size (FRAME_BUDGET_BLOCKS = 2). Until a camera's frame size is known, the first sample that takes its frames is a group of its own. A read without decoding builds each row as its sample resolves. - With video and no resize, each read task reserves that budget as memory, 256 MiB by default. The caller's memory, or memory in ray_remote_args, wins. With resize, nothing is reserved. - A row's frame bytes are checked against RAY_DATA_MCAP_MAX_ROW_BYTES before its group decodes, and again as soon as a frame of a new size decodes. A camera's size change within a row fails, naming the sample. - A camera's decoder keeps running into the next group when no keyframe lies between them, so back-to-back clips feed each frame once. - Every step rule works on decoded topics, except match.all_in_step, which a decoded topic refuses at planning: a step holds one frame. - Planning opens the first samples' files to settle which topics are decoded, in the same probe that sizes the read tasks, and without the checkpoint config. The estimate reuses the summaries it read. - A file without a chunk index keeps every message of a decoded topic, since its decoder may need any of them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem: - A filter on topic or log_time above read_mcap read every chunk, then dropped most rows. - count() read every payload, though each file's summary already counts messages per channel. Change: - MCAPScanner pushes topic and log_time conjuncts into the selection of a read of message rows. Listing prunes files and chunks with the same selection; the rest of the predicate stays in a Filter. A sample read keeps every filter above the read. - count() of message rows sums the summary's per-channel statistics when they cover the selection (no time range, no limit, and max_errored_blocks is 0). A file whose summary cannot answer is counted by reading it. A count task counts 16 files at the same time (RAY_DATA_MCAP_READER_COUNT_ROWS_BATCH_SIZE). - Each manifest row counts, so a path listed twice counts twice, as it is read twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Description
Problem.
topicorlog_timeaboveread_mcapread every chunk, then dropped most rows.count()read every payload, though each file's summary already counts messages per channel.Change.
MCAPScannerpushestopicandlog_timeconjuncts into the selection of a read of message rows. Listing prunes files and chunks with the same selection, and the rest of the predicate stays in aFilter. A sample read keeps every filter above the read.count()of message rows sums the summary's per-channel statistics when they cover the selection: no time range, no limit, andmax_errored_blocksis 0. Otherwise a read may skip corrupt chunks, socount()reads, and it equals what a read returns.RAY_DATA_MCAP_READER_COUNT_ROWS_BATCH_SIZE), counts them on a thread pool, and retries transient errors.Related issues
Additional information
615 passed, 31 skipped. pre-commit and pyrefly clean.
AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.
🤖 Generated with Claude Code