Skip to content

[Data] read_mcap: push filters into listing and count from statistics - #66656

Draft
marwan116 wants to merge 39 commits into
marwan/mcap-dsv2-videofrom
marwan/mcap-dsv2-pushdown
Draft

marwan116 wants to merge 39 commits into
marwan/mcap-dsv2-videofrom
marwan/mcap-dsv2-pushdown

Conversation

@marwan116

@marwan116 marwan116 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

Problem.

  • A filter on topic or log_time above read_mcap read every chunk, then dropped most rows.
  • count() read every payload, though each file's summary already counts messages per channel.

Change.

ds = ray.data.read_mcap("s3://bucket/recordings/")
ds.filter(expr=col("topic") == "/imu").count()
# summed from each file's statistics; no chunk is opened
ds.filter(expr=(col("topic") == "/imu") & (col("log_time") >= t0)).count()
# a time range is counted by reading, but only the files and chunks that can match
  • MCAPScanner pushes topic and log_time conjuncts into the selection of a read of message rows. Listing prunes files and chunks with the same selection, and the rest of the predicate stays in a Filter. A sample read keeps every filter above the read.
  • count() of message rows sums the summary's per-channel statistics when they cover the selection: no time range, no limit, and max_errored_blocks is 0. Otherwise a read may skip corrupt chunks, so count() reads, and it equals what a read returns.
  • Each manifest row counts, so a path listed twice counts twice, as a read returns its rows twice.
  • A file whose summary cannot answer is counted by reading it. A count task lists 16 files (RAY_DATA_MCAP_READER_COUNT_ROWS_BATCH_SIZE), counts them on a thread pool, and retries transient errors.

Related issues

Additional information

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py \
  python/ray/data/tests/unit/datasource_v2/test_file_partitioners.py

615 passed, 31 skipped. pre-commit and pyrefly clean.

AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116
marwan116 requested a review from a team as a code owner October 2, 2026 04:33
@marwan116 marwan116 added the data Ray Data-related issues label Oct 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces filter pushdown for 'topic' and 'log_time' conjuncts and metadata-based row counting for the V2 MCAP datasource in Ray Data. By folding these filters into the MCAP selection, the file indexer can prune files and chunks during listing, and the reader can avoid reading unnecessary payloads. Additionally, the reader now supports metadata-based row counting to answer count queries efficiently using MCAP statistics. The review feedback correctly identifies a potential correctness issue in 'read_metadata' where message counts could be over-counted or double-counted if a file's chunks are split across multiple manifests or if some chunks are excluded, as it currently sums the entire file's statistics.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 1c0d1c9 to 43e2723 Compare October 5, 2026 00:37
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 43e2723 to 86a68dc Compare October 5, 2026 02:20
@marwan116
marwan116 added this pull request to stack #66712 October 5, 2026 02:25
@marwan116 marwan116 self-assigned this Oct 5, 2026
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 86a68dc to 5717191 Compare October 5, 2026 02:32

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 5717191 to 345de8f Compare October 5, 2026 02:45
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 345de8f to 5613406 Compare October 5, 2026 02:56
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-pushdown branch from 5613406 to 789c37b Compare October 5, 2026 03:01
marwan116 and others added 4 commits November 8, 2026 13:38
Problem:
- read_mcap switches to the V2 datasource by default, but its
  docstring did not say so, or how its rows differ.
- The use_datasource_v2 docstring did not name MCAP.

Change:
- The read_mcap docstring lists the three differences from the legacy
  reader, how to switch back, and an example that stores schema_data
  as plain bytes.
- The use_datasource_v2 docstring names MCAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The V2 warning names override_num_blocks even when the caller
  passed parallelism.
- It hard-codes 128 MiB instead of the configured budget.

Change:
- Name the argument the caller gave.
- Print the budget from RAY_DATA_MCAP_BIN_PACKING_BYTES.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The chunk-fetch test ran four read configs, but one is its own
  baseline and two make the same single read on the test file.
- The parallelism warning had its own test beside the
  override_num_blocks test, which varied only the value.
- The generated-id test ran two names that hit the same raise.

Change:
- Compare one chunk per read with small reads only.
- Parametrize the override_num_blocks test over the argument name and
  drop the separate parallelism test.
- Check the generated-id raise with the name row_id only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Listing threads read and parsed summaries, and up to twice the
  summary concurrency of parsed summaries were held at once. Parsed
  summaries are much larger than their bytes, and nothing bounded
  their size. Parsing holds the GIL, so the threads gained nothing
  from it.
- A read task's peak was estimated as 1.2 times its table. That
  misses the chunk being decompressed and parsed, the log-time heap
  and the ranged read, so a task of large chunks or small messages
  peaked at up to 2.6 times its estimate.

Change:
- Summary reads are split in two: a tail read, whose footer gives the
  section's exact length, and a section read. Listing threads only
  fetch bytes, and the caller parses each section in order.
- A section is fetched once it fits the read task budget
  (RAY_DATA_MCAP_BIN_PACKING_BYTES) with the sections held. With
  nothing held, the next one is fetched whatever its size. A failed
  read is still that file's own error.
- A read task's peak is estimated as its table, plus the largest
  working set of one chunk, plus one ranged read. Each chunk weighs
  its table scaled so that a full task leaves room for the rest.
- New tests pin the estimate's constants by measurement, check the
  estimate against a measured read, and check the byte budget of
  summary reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
marwan116 and others added 25 commits November 10, 2026 17:48
Problem:
- The estimate counts a payload's raw bytes, but a JSON payload is
  decoded into Python values that hold several times as many bytes
  until its table is built. A read of small JSON messages peaks above
  the estimate.

Change:
- When every selected channel of a file is JSON-encoded, each payload
  byte counts the reader's own bytes-per-payload-byte for decoded
  JSON.
- The peak test also reads JSON payloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- An ordered read (preserve_order) keeps each file in one read task, so
  a large recording is read by one task and held in memory whole.
- In other reads the packer places a file's chunks one by one, so a
  small file can be split across two shared tasks.

Change:
- OrderedFileCutter replaces OrderedFileGrouper. It cuts each large
  file into read tasks and sends them at once, in order. It cuts only
  where every chunk before the cut ends no later than every chunk after
  it starts, so with preserve_order each file still comes out in
  log-time order from start to end.
- In a recording with one writer every boundary allows a cut, so tasks
  are as full as in an unordered read. Where chunks overlap in time, as
  when cameras and robot state are written to chunks of their own, the
  overlapping stretch stays in one task.
- A small file goes to the packer whole, so it is never split.
- A file whose rows cannot be placed in time goes to the packer as
  listed: one read whole, an unreadable one, or rows without a chunk
  index record.
- With log_time_order=False a file is cut in file order, where every
  boundary keeps the order.
- The log_time_order docstring says where an ordered read cuts a file.
- New tests: the cuts and the tasks' summary records. The preserve_order
  read of three files now checks that each file is split. Tests that
  force small tasks now check that they still split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A file whose chunks overlap in time has few cuts that keep log-time
  order, so it was read by one task on every read. Without
  preserve_order its blocks come out in any order anyway, so that one
  task bought nothing and cost parallelism.
- In an ordered read, that task's merge holds every chunk that overlaps
  the current log time. A file whose chunks all overlap is held whole,
  and nothing told the user.

Change:
- The cutter keeps a file's cuts in log-time order only when the read
  keeps its blocks in order. It reads
  DataContext.execution_options.preserve_order in the listing task,
  which runs under the dataset's own context.
- Otherwise every boundary allows a cut. Chunks are still taken by
  first log time, so each task spans one stretch of time.
- A recording with one writer is cut the same way in either case.
- On a tied log time, an ordered cut is allowed only when the chunks
  before it that end then lie earlier in the file, since the merge
  yields equal log times in file order.
- An ordered task whose overlapping chunks would hold more than four
  read-task budgets at once logs one warning per process. It names the
  file and suggests read_mcap(memory=...) or dropping preserve_order.
- The warning stays quiet when each read task already reserves at
  least that much. read_mcap passes memory, or else
  ray_remote_args["memory"], to the datasource, which hands it to the
  cutter.
- The read_mcap docstring says that an overlapping stretch stays in
  one task only with preserve_order, that the task holds it whole, and
  when the warning fires.
- New tests: an unordered read may cut a file whose chunks overlap in
  time at any boundary; the warning fires once for a file with no clean
  cut, and not for shallow overlap, an unordered read, one big chunk or
  a reservation that covers it; read_mcap's memory reaches the cutter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- With preserve_order, the cutter sent a heavy file's tasks at once
  and left light files in the packer's open bins. A large file listed
  after small ones came out first.

Change:
- With preserve_order, the cutter packs light files and files it
  cannot place in time itself, in listing order. A heavy file's tasks
  leave after the files listed before it.
- Without preserve_order, light files go to the packer as before.
- The read_mcap docstring says files come out in listing order with
  preserve_order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
This is 2a's fix applied to 2c's code. The 2c-specific part is the
docstring of OrderedFileCutter's _rows_by_file.

Problem:
- The reader merged the manifest rows of one path. A path listed twice
  was read once when its copies shared a read task and twice when they
  did not, so the row count depended on the task budget.
- The packer keeps one row per listed copy, and Parquet reads a
  repeated path twice.
- Three fallbacks could turn off the summary handoff unnoticed: a
  missing task summary column, a path named by two rows, and a listing
  without record columns all made read tasks read summaries again.

Change:
- The reader reads each manifest row on its own, in manifest order.
- task_summaries asserts the task summary column, and the partitioner
  asserts the listing record columns. A null record only marks a
  file the listing could not read.
- Tests pack listing rows into a read task (as_read_task) instead of
  reading listing manifests directly.
- OrderedFileCutter's _rows_by_file states why copies stay apart: each
  listing manifest holds one listed file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The cutter checked for a missing chunk index column and for rows
  naming several chunks. The listing never makes such rows, and
  TaskSummaryPartitioner reads the column before the cutter runs.
- Several cutter tests repeated what other tests already assert.
- The memory test mocked MCAPDatasourceV2.__init__ to see one argument.

Change:
- Read the chunk index column directly, and drop the one-chunk check.
- Drop the empty-record and two-chunk rows from unplaceable_files().
- Drop tests whose cases other tests cover, and merge the light-file
  test and the two cut-anywhere tests into parametrized tests.
- Check the memory reservation on the cutter that read_mcap's plan
  holds, with no mock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- An ordered read warned only once its log-time merge held four read
  task budgets. Nothing measured chose four, and a task with eight
  overlapping chunks peaked at 2.5 budgets with no warning.
- The warning named `preserve_order`, but the merge comes from
  `log_time_order`, and it gave no value to reserve.

Change:
- Warn once the merge holds more than the budget a task is sized for.
- Name the reservation to pass, `read_mcap(memory=...)` rounded up to
  a whole MiB, and `log_time_order=False` as the other way out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- An ordered log-time read cut a large file only where its log-time
  order held. That needed per-chunk end times, a tied-time rule and a
  memory warning with a reservation handed down from read_mcap.
- The cuts bought parallelism, not memory. A whole-file task holds one
  ranged read, one chunk and one block at a time, whatever the file's
  size. Chunks that overlap in time were never cut apart, so a task
  held them together either way.

Change:
- With preserve_order and log_time_order, each file is read whole by
  one task, in listing order.
- Other reads still cut a large file at the budget, and files still
  keep their listing order with preserve_order.
- Remove the ordered-task memory warning, read_task_memory on the
  datasource, and their tests.
- chunk_start_time replaces chunk_log_times, since only the first log
  time is still read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A map task buffers its output until a block reaches its target size.
  When the task raised, its buffered rows were dropped, even when
  max_errored_blocks let the job skip the error. A read task that hit
  one corrupt file lost the rows of the files it had read before it.

Change:
- When the job skips errors (max_errored_blocks is not 0) and the task
  cannot run again, _BlockShapingIterator emits the buffered rows as a
  last block, then raises the error.
- A task that may run again raises without a partial block: on an
  error that retried_map_errors matches while _map_task has attempts
  left, on a RaySystemError, or on an error the task's
  retry_exceptions names (or one it wraps). An actor task's
  retry_exceptions come from actor_task_retry_on_errors. A rerun skips
  as many blocks as were already emitted, so a partial block would
  hide rows of the rerun.
- TaskContext carries the task's Ray retry_exceptions (False when
  max_retries is 0), and _map_task sets each attempt's retries left
  and retried_map_errors patterns on it.
- BlockOutputBuffer.has_buffered_rows says whether rows are waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The listing cannot isolate a file whose summary reads but whose
  chunk is corrupt. That chunk failed the read task holding it: the
  reader stopped at the bad file, and the small files sharing the task
  lost their rows too.

Change:
- When max_errored_blocks lets errors be skipped, the reader skips a
  corrupt chunk and reads the rest of its file. Each chunk is
  compressed on its own and found through the chunk index, so one bad
  chunk costs only its own rows.
- A chunk read back for a declaration that does not decode is skipped,
  and the look-back goes on to the next earlier chunk, so the chunk
  being read is not blamed for it.
- A retried file read starts with no corrupt chunks recorded.
- A file that fails as a whole does not stop the task's other files.
  A file without a chunk index still fails whole, and an error reading
  the file still raises, so the read retry applies.
- Once the task's files are read, it raises one error naming every
  file and chunk that failed, so the task counts once against
  max_errored_blocks. With max_errored_blocks=0 the first failure
  still fails the task at once.
- With the map task's flush of buffered rows, the job keeps every
  other row, also when a map is fused into the read.
- New tests: a chunk whose payload does not decompress and one whose
  record does not parse, in both read orders, also when fetched in
  one ranged read with its neighbours. End to end, a file with two
  corrupt chunks loses only their rows under max_errored_blocks=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- An actor task counted as rerun on any error its retry_exceptions
  names, even when its actor's max_task_retries is 0. Ray then does
  not rerun it, so a skipped error dropped the task's buffered rows.

Change:
- An actor task counts as rerun only while max_task_retries allows it.
  The task option overrides the actor's; Ray defaults it to 0, as it
  stays with max_restarts=0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A map task counted as rerun on every RaySystemError, even with
  max_retries=0 or max_task_retries=0. Ray then does not rerun it, so
  a skipped error dropped the task's buffered rows.

Change:
- RaySystemError counts as rerun only through the task's
  retry_exceptions, which Ray Data always extends with it and which
  are empty when retries are off.
- The task-pool retry check adds RaySystemError as cached_remote_fn
  does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…error

Problem:
- A fused downstream batcher dropped the rows it held when an upstream
  stage raised an error the job skips. Map then MapBatches(batch_size=4)
  kept rows 0 to 3 of 0 to 6.
- A TaskContext built without retry settings meant "never rerun", so a
  shuffle reduce task emitted partial output it did not opt in to.
- The retry settings were sentinels on TaskContext, and the pattern
  check was a copy of iterate_with_retry's.
- The max_errored_blocks docs still said failed blocks are dropped.

Change:
- A batch transform ends its input at an error the task skips, hands
  on a short last batch, then raises the error.
- A frozen RerunPolicy replaces the TaskContext retry fields. Its
  default reruns on any error; map operators and _map_task fill it in.
- iterate_with_retry and RerunPolicy share is_retryable_error.
- _emit_rows_before_error is renamed _should_emit_buffered_rows.
- The context docstring and the docs page say which rows are kept.
- Dataset-level tests cover a skipped map_batches error and a fused
  batcher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…once

Problem:
- A corrupt chunk read back for a declaration was always skipped. When
  it alone declared a schema, rows came out with no schema, even with
  max_errored_blocks=0.
- Any error while a chunk was read, such as the reader's own missing
  channel error, a MemoryError or a bug, was reported as a skipped
  corrupt chunk.
- A task joined every failed file into one ValueError caused by the
  first. A later OSError was hidden, so the task was skipped instead
  of rerun.

Change:
- is_corrupt_data_error says which errors mean the bytes do not decode.
  Only those are skipped, and only around decoding a chunk.
- A chunk read back for a declaration is skipped only when corrupt
  chunks are skipped. A schema still missing after the look-back
  raises, naming the corrupt chunk.
- A file's corrupt data is deferred to the task's one error. Any other
  error raises at once. _read_owned_file holds one file's read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…uplicates

Problem:
- Ten unit tests each ran one rerun policy and checked whether the
  buffered rows came out first.
- The errored-rows test ran max_errored_blocks 1 and -1, which take
  the same path.
- Two read_mcap tests also ran with a fused map, which the fused
  batcher test already covers.

Change:
- Make the ten tests one test over a small case dataclass of the
  rerun policy and whether the rows come out, with readable ids.
- Let the failing transformer helper raise a given error type, and
  drop the separate system-error transformer.
- Keep only max_errored_blocks=-1, which also catches a check that
  skips only on a positive limit.
- Drop the fused_map parameter from the two read_mcap tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- With max_errored_blocks != 0, one message whose payload is not valid
  JSON dropped the rest of its file, rows already built included.
- The error did not say that the rest of the file was dropped.

Change:
- decode_payload raises InvalidJsonPayload, a ValueError, for a payload
  that is not valid JSON. A wrong message encoding still raises a
  ValueError that fails the file.
- When errors can be skipped, the reader skips that one message and
  keeps every other row of the file, as it does for a corrupt chunk.
- One error per file names the skipped chunks by byte offset, counts
  the skipped messages and names the first by row id, so the task
  counts once. The record keeps only the first message.
- The table builder decodes a payload before it adds anything, so a
  skipped message leaves it consistent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- With max_errored_blocks set, a message that is not valid JSON used
  up a place in the row limit, so a limited read could return fewer
  rows than asked although later messages were valid.
- A file that failed mid-way, for example on a topic of another
  encoding, dropped the rows already built into its current table.
  What the file kept depended on the block size.

Change:
- The limit counts rows built. A skipped message frees its place, and
  no message is pulled once the limit is met.
- An ordered scan of a file without a chunk index holds every message
  when it can skip, so a skipped one cannot shorten the result.
- When errors are skipped, a file error yields the rows built before
  it, then fails the rest of the file as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A missing schema raised for every message, before the selection
  check ran.
- So a topic the read filters out could fail the rest of its file
  when its schema lived only in a skipped corrupt chunk. The later
  selected rows were lost and a max_errored_blocks slot was spent.

Change:
- Split the channel and schema lookups. A message whose channel the
  listing or the topic filter rejects, or whose log time is out of
  range, is skipped before its schema is looked up.
- A message that may be read still fails on a missing schema and
  names the corrupt chunk read back for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A read that skips corrupt chunks keeps each chunk's error until the
  file is done. An error's traceback holds its chunk's bytes, so every
  corrupt chunk adds its size to the read's memory.

Change:
- Keep the byte offsets of a file's corrupt chunks and only the error
  at the lowest offset. The error the read raises is unchanged.
- A test reads a file whose every chunk is corrupt and bounds its peak.
- Cut four reader tests whose cases other tests already cover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…ples checks

Problem:
- A sample read takes two inputs: a TopicSpec per topic, saying where
  its steps fall and which message fills each, and a dataset of
  samples. Both need a shape and checks before anything reads a file.

Change:
- TopicSpec(steps, match=None, pad="edge"), exported from
  ray.data.datasource.mcap beside steps and match.
- ray.data.datasource.mcap.steps: before/after (1/fps apart around
  at="anchor", "clip_start" or "clip_end"), across, at, and
  last/next/every, which take the recorded messages themselves.
- ray.data.datasource.mcap.match: nearest (default), backward,
  all_in_step.
- mcap_samples.check_samples_schema: files, clip_start and clip_end
  with the right types, an anchor when a topic measures from it, and no
  column the read adds.
- mcap_samples.sample_manifest: a batch of samples as manifest rows,
  one file each, failing on the first bad sample by its sample_id or
  its file and clip.
- No I/O. The read that uses them comes later in the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Given a sample (clip and anchor) and a topic's TopicSpec, a sample
  read must decide which recorded message fills each step, what a step
  no message fills holds, and what the row looks like.

Change:
- mcap_samples.TopicPlan: a TopicSpec compiled for one topic, with the
  span of log times its steps can reach for a sample.
- resolve_steps / resolve_slots: the message (or, for all_in_step, the
  messages) that fill each step. nearest never looks across the
  anchor; backward takes the last message at or before each step.
- Steps no message fills are padded by pad="edge" or "zeros", or fail
  with pad="raise", naming the sample.
- sample_row_schema: the samples' own columns, then each topic's data:,
  times: and is_pad: lists. is_pad is True for a padded step, as in
  read_lerobot's {key}_is_pad.
- Pure functions over log times: no I/O. The reader that feeds them
  comes later in the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A sample row needs each topic's messages around its anchor, read
  from the recording's chunks. Reading each sample on its own reads
  the same chunks many times when samples lie close together.
- The chunks a task holds at once must stay within its budget.

Change:
- mcap_sample_rows: a read task takes its samples file by file, in
  anchor order, and groups samples whose windows meet into runs. A
  run ends before its chunks would hold more than a read task's
  budget (RAY_DATA_MCAP_BIN_PACKING_BYTES). Each run reads the chunks
  its topics' steps reach once (ChunkTimeline), and forgets chunks
  the next run does not need.
- A file without a chunk index is scanned once (WholeFileTimeline).
  The scan keeps only what steps can take: the runs' spans, the
  nearest messages around each span, and the first and last messages
  inside each sample's bounds.
- A corrupt chunk fails the read task. Ray's max_errored_blocks then
  decides whether the task's block is dropped or the read fails.
- SampleRows builds one row per sample: the sample's columns, then each
  topic's data:, times: and is_pad: lists. A row over
  RAY_DATA_MCAP_MAX_ROW_BYTES fails the read, naming the sample.
- mcap_summary.topic_chunks: the chunks that may hold a topic, one rule
  for the reader and, later, for sizing tasks.
- MCAPDatasourceV2(topics={topic: TopicSpec}, samples_schema=...) sets
  up a sample read: its schema is the sample row's, and its reader
  builds sample rows from manifest rows that name each sample's file.
  read_mcap wires it up in the next PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Robot learning reads examples, not messages: a clip of a recording,
  or a moment inside one, with each topic read over its own window.
- The stack below adds the spec, step resolution, sample checks and
  the sample reader; nothing reaches them from read_mcap yet.

Change:
- read_mcap(samples=ds, topics={topic: TopicSpec(...)}) reads one row
  per sample. The samples feed ReadFiles as its manifest; planning
  lists no files. Options that only a read of paths takes are refused.
- Planning runs the samples once: their schema comes from their plan
  when it is known, else from the samples probed for task size.
- A task takes as many samples as read about
  RAY_DATA_MCAP_BIN_PACKING_BYTES of chunks, estimated from the first
  256 samples (mcap_sample_tasks).
- A checkpoint keyed on a sample column skips finished samples before
  their files are opened: ReadFiles(checkpoint_on_input=True), a
  checkpoint filter only at the read nearest the sink, and ReadFiles
  skipping empty manifest blocks. A checkpoint keyed on a column the
  samples lack, such as row_id, is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A sample read returned each camera step as an encoded payload. An
  H.264 or H.265 frame needs the frames back to its keyframe to decode,
  so a row could not be decoded on its own, and users decoded whole
  streams again downstream.

Change:
- read_mcap(samples=..., video=VideoOptions(resize=...)) decodes the
  video topics in the read task. Each video topic's steps become a
  frames:<topic> uint8 tensor (steps, height, width, 3) in place of
  data:<topic>; video_topics lists topics with unknown schema names,
  and needs video.
- Samples whose steps meet form a group, and each camera is decoded
  once per group, from the last keyframe at or before its first frame
  (RAY_DATA_MCAP_MAX_LEAD_IN_S back at most). Only the frames that
  steps take become RGB.
- A group ends before its decoded frames would pass twice the target
  block size (FRAME_BUDGET_BLOCKS = 2). Until a camera's frame size is
  known, the first sample that takes its frames is a group of its own.
  A read without decoding builds each row as its sample resolves.
- With video and no resize, each read task reserves that budget as
  memory, 256 MiB by default. The caller's memory, or memory in
  ray_remote_args, wins. With resize, nothing is reserved.
- A row's frame bytes are checked against RAY_DATA_MCAP_MAX_ROW_BYTES
  before its group decodes, and again as soon as a frame of a new
  size decodes. A camera's size change within a row fails, naming the
  sample.
- A camera's decoder keeps running into the next group when no
  keyframe lies between them, so back-to-back clips feed each frame
  once.
- Every step rule works on decoded topics, except match.all_in_step,
  which a decoded topic refuses at planning: a step holds one frame.
- Planning opens the first samples' files to settle which topics are
  decoded, in the same probe that sizes the read tasks, and without
  the checkpoint config. The estimate reuses the summaries it read.
- A file without a chunk index keeps every message of a decoded
  topic, since its decoder may need any of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- A filter on topic or log_time above read_mcap read every chunk, then
  dropped most rows.
- count() read every payload, though each file's summary already
  counts messages per channel.

Change:
- MCAPScanner pushes topic and log_time conjuncts into the selection
  of a read of message rows. Listing prunes files and chunks with the
  same selection; the rest of the predicate stays in a Filter. A
  sample read keeps every filter above the read.
- count() of message rows sums the summary's per-channel statistics
  when they cover the selection (no time range, no limit, and
  max_errored_blocks is 0). A file whose summary cannot answer is
  counted by reading it. A count task counts 16 files at the same
  time (RAY_DATA_MCAP_READER_COUNT_ROWS_BATCH_SIZE).
- Each manifest row counts, so a path listed twice counts twice, as
  it is read twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant