Skip to content

[Data] read_mcap V2: carry read tasks' summary records in the manifest - #67929

Merged
richardliaw merged 5 commits into
masterfrom
marwan/mcap-dsv2-task-summary
Nov 11, 2026
Merged

richardliaw merged 5 commits into
masterfrom
marwan/mcap-dsv2-task-summary

Conversation

@marwan116

@marwan116 marwan116 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Read tasks take their files' summary records from the listing's manifest instead of reading each file's summary again.

How to review

Start with mcap_task_summary.py: it holds the records each read task needs, and its module docstring shows the flow. Then read how the reader uses them.

File What changes Lines
mcap_task_summary.py New. Packs each task's summary records into the manifest. 240
mcap_reader.py Reads the records from the manifest instead of the file. 120
mcap_summary.py, mcap_file_indexer.py The listing keeps the raw records it already read. 111
mcap_io.py, mcap_chunks.py, mcap_datasource_v2.py Small hooks. 94
online_bin_packer.py, mcap_file_cuts.py Docstrings only. 21
test_mcap_task_summary.py Rows match reading the summary from the file. 427

Before

The listing reads every file's summary to plan the read, then each read task reads the same summaries again: one to three ranged requests per file, per task.

After

The listing hands each read task the summary records its chunks need, in the manifest (mcap_task_summary). A read task opens its files' chunks directly.

flowchart LR
    L[listing: read summary once] -->|records in manifest| T[read task]
    T --> C[read chunks]
Loading
  • Each manifest row is read on its own, so a path listed twice is read twice. The packer contract and Parquet read such a path the same way.
  • The reader asserts that its task carries the summary-records column. It reads a file's summary itself only for a file the listing could not read, and that read fails with the file's own error.
  • Also: the OnlineBinPacker docstring describes per-shard packing correctly, and the MCAP tests share one pack helper. Neither changes behaviour.

Subsystems

IO (_internal/datasource_v2/formats/mcap).

Testing

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py

206 passed, 3 skipped. pre-commit and pyrefly clean.

Not a duplicate: split out of #67166 at a reviewer's request. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

marwan116 and others added 3 commits November 6, 2026 11:36
Problem:
- The docstring promised at most one under-filled bin per listing
  shard. A shard's flush can leave several.
- It did not say when packing per shard pays off.

Change:
- The docstring says that listing then runs as parallel tasks: the
  listing planner splits the paths passed to read_* evenly across up
  to DEFAULT_MAX_NUM_LIST_FILES_TASKS (200) tasks, and a folder counts
  as one path. It still uses one task when the read shuffles files.
  The cost is that bins may not fill as well, which may mean more
  read tasks.
- The same claim in MCAPDatasourceV2.get_file_partitioner's comment is
  fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Packing listing rows into read tasks, as a ListFiles task does, was
  a helper of the listing tests alone. The reader and task summary
  tests need the same steps.

Change:
- pack(datasource, paths=None, **kwargs) moves to mcap_testing.py. It
  lists the datasource's own paths by default and passes kwargs, such
  as preserve_order, to the listing.
- Test code only. The listing tests call it as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The listing reads every file's summary, and then each read task read
  its files' summaries again, one to three requests per file, to use a
  small part of them.

Change:
- Listing rows carry the summary records their read tasks need: each
  chunk row its ChunkIndex record, and each file its schema and channel
  records, stored once (mcap_task_summary).
- TaskSummaryPartitioner wraps the bin packer. To each packed row it
  adds the file's records and the index records of the chunks the row
  names, compressed with zstd. A carried ChunkIndex record leaves out
  its message index offsets, which the listing has already applied.
- The reader parses those records instead of reading the summary. A
  task that reads back a declaration from an earlier chunk reads the
  whole summary once. A file named by two rows of a task, or a manifest
  without the records, reads the summary from the file.
- A whole-file row carries every record, or none for a file without a
  summary. An unreadable file carries null, so its task still reads the
  summary and fails naming the file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes MCAP reading by carrying the summary records each read task needs directly in its manifest, eliminating the need for read tasks to fetch and parse the entire summary from the file. It introduces TaskSummaryPartitioner to pack rows and attach compressed summary records, and updates MCAPReader and related MCAP data source utilities to support this flow. Feedback on the changes suggests adding a defensive check in TaskSummaryPartitioner._hold_records to verify that metadata is not None before subscripting it, preventing potential TypeError exceptions.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@ray-gardener ray-gardener Bot added the data Ray Data-related issues label Nov 7, 2026
Problem:
- The reader merged the manifest rows of one path. A path listed twice
  was read once when its copies shared a read task and twice when they
  did not, so the row count depended on the task budget.
- The packer keeps one row per listed copy, and Parquet reads a
  repeated path twice.
- Three fallbacks could turn off the summary handoff unnoticed: a
  missing task summary column, a path named by two rows, and a listing
  without record columns all made read tasks read summaries again.

Change:
- The reader reads each manifest row on its own, in manifest order.
- task_summaries asserts the task summary column, and the partitioner
  asserts the listing record columns. A null record only marks a
  file the listing could not read.
- Tests pack listing rows into a read task (as_read_task) instead of
  reading listing manifests directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit dc87f27. Configure here.

@marwan116 marwan116 added the go add ONLY when ready to merge, run all tests label Nov 7, 2026
Problem:
- The listing copied every record of a whole-file row's summary, and
  rewrote each ChunkIndex record without its message index offsets. Both
  need hand-written byte offsets for little gain.
- Opcodes were written out as numbers that mcap already names.
- read_summaries and read_listed_summaries read the same summaries.
- Several tests checked what other tests already cover.

Change:
- A whole-file row carries the raw summary section. A chunk row carries
  each ChunkIndex record unchanged.
- Opcodes come from mcap.opcode.Opcode, imported inside the functions.
- read_summaries returns ListedSummary for every caller.
- Cut or merge duplicate tests. Keep one check that a file's records
  are stored once, the unreadable-file test, and the empty-records test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116
marwan116 removed this pull request from stack #67931 November 9, 2026 19:35
@marwan116
marwan116 added this pull request to stack #67983 November 9, 2026 19:35
@marwan116 marwan116 self-assigned this Nov 10, 2026
@richardliaw
richardliaw merged commit 0eb5672 into master Nov 11, 2026
6 checks passed
@richardliaw
richardliaw deleted the marwan/mcap-dsv2-task-summary branch November 11, 2026 22:36
eugenepaniot added a commit to eugenepaniot/ray that referenced this pull request Nov 19, 2026
…floop-input

* upstream/master:
  [Data][3/n] update progress bar on lineage reconstruction (ray-project#67366)
  [Data] Enable the Top-N rewrite by default and expose it on DataContext (ray-project#68329)
  [Data] Adaptively skip map-side partial aggregation for high-cardinality keys (ray-project#68118)
  [Core] Spawn specific servers without preexec_fn to fix intermittent startup crashes (ray-project#67947)
  [Serve] Stop the __wrapped__ walk at non-callables (ray-project#67117)
  [Data] Deflake test_read_parquet_v2 by cutting redundant Ray Data executions (ray-project#68318)
  [Data] Retry FooterReader actor deaths with Ray Core and yield footers in listing order (ray-project#67988)
  [Data] Fix order-dependent schema unification when a list<null> arm is present (ray-project#68259)
  [Serve] Make @serve.batch work when wrapped in another decorator (ray-project#68177)
  [serve] Deflake test_metrics_3 - record the batch queue length once a batch is taken (ray-project#68302)
  [serve] Drop completed requests from the multiplexed model ID index (ray-project#68085)
  [Data] Fix flaky test_hudi: read each table once instead of sorting in Ray Data, keep size small (ray-project#68162)
  [Data] Fix flaky test_context: start 2 Ray clusters instead of 5 and keep size small (ray-project#68161)
  [Data] Fix flaky test_kafka: use recent timestamps in test_read_kafka_datetime_partial_range (ray-project#68248)
  [Data] Iceberg DSv2 [2/5]: The listing-to-reading manifest for Iceberg (ray-project#67951)
  [Train] Report training errors when the post-mortem debugger raises a BaseException (ray-project#68317)
  [RLlib] Skip the PPOConfig docstring example that crashes the rllib doctest target (ray-project#68306)
  [Train] Support post-mortem debugging for training function errors (ray-project#68296)
  [train][1/n] Give each health probe its own timeout (ray-project#68191)
  [Data] Fix unifying a fixed-shape tensor array with a lower-ndim variable-shaped one (ray-project#67266)
  [Train] Add a guide to debugging failures and hangs (ray-project#68299)
  [Data] Persist BlockMetadata.num_rows and batch ray.get in _calculate_blocks_rows (ray-project#67662)
  [Data] Vectorize LeRobot delta segment preparation and validate index contiguity (ray-project#67845)
  [Data] Avoid full-DataFrame apply in _AbstractKBinsDiscretizer._transform_pandas (ray-project#67862)
  [doc] Refresh intersphinx inventory snapshots (ray-project#68303)
  [Data] Fix SimpleImputer on categorical columns in append mode and with existing category values (ray-project#67467)
  [Data] Replace O(N) list.pop(0) with deque.popleft() in PushBasedShuffleTaskScheduler (ray-project#67243)
  [Train] Reject an elastic ScalingConfig with max_workers=0 (ray-project#68218)
  [core][move semantics][3/n]Populate task args with is_move for movable refs (ray-project#66803)
  [core][sandbox] Remove the readonly option (ray-project#68088)
  [Data](1/n) Implement the line-chunk reader for CSV and JSONL on DataSource V2 (ray-project#67907)
  [Data] Set MIMALLOC_PURGE_DELAY=0 on random_shuffle to fix the pyarrow 25 OOM regression (ray-project#68151)
  [core] Quote each `uv run` argument in the uv runtime env hook (ray-project#68156)
  [core] Don't let gRPC cancel calls that wait over 30s in the server queue (ray-project#68210)
  [Data] Fix shuffle and join on native tensor columns with pyarrow 25+ (ray-project#68072)
  [serve][llm] Remove decode load progress (ray-project#68258)
  [ci] Bump sglang to 0.5.23 in release tests (ray-project#68250)
  [Core] Build the test auth token file locally so Windows tests can read it (ray-project#68262)
  [image] Bump EFA installer to 1.51.0 in CPU base-extra images (ray-project#67730)
  [serve] Observe the result of rank reassignment reconfigures (ray-project#67923)
  [Serve][Docs] Explain imperative vs declarative deploys and in-place updates from Python (ray-project#68243)
  [Data] (Fault tolerance) Ray Data lineage reconstruction docs (ray-project#68061)
  [RLlib] Add a multi-node footsies APPO memory release test (ray-project#67595)
  [Train] Remove local mode (ray-project#68239)
  [Data] Fix filtered Lance row-count metadata (ray-project#67228)
  [Serve][2/2] Propagate TracingConfig to proxies at startup (ray-project#65437)
  [Data] Fix synthesized path filtering in DataSourceV2 (ray-project#67079)
  [RLlib] Run the BCConfig Tune example one trial at a time to unblock the rllib doctest target (ray-project#68242)
  [Data][Delta DSv2][2/7] List Delta Lake table files from the log (ray-project#67888)
  [Data] Parallelize Parquet and ORC full-schema reads (ray-project#67542)
  [Data] Random Shuffle on Shuffle V2 Map/Reduce Operators (ray-project#68004)
  [Data] Propagate metadata prefetch thread failures to executor (ray-project#68171)
  [core] Send generator consumed-progress updates from io_service_ (ray-project#67160)
  [Data] Add @ray-data-connectors as codeowner for datasource/sinks files (ray-project#68154)
  [data][llm] Keep the prompt in vLLM error rows (ray-project#67665)
  [data][actor-only][3/n] Actor-only backend: operator fusion (ray-project#67699)
  [Data] Read text files on DataSource v2 (ray-project#68063)
  [Autoscaler][KubeRay] Never patch worker group replicas below minReplicas (ray-project#67332)
  [Serve] Show progress of updating (ray-project#67107)
  [core][test] Trigger usage-stats OOM test without exhausting host memory (ray-project#68009)
  [Serve] Remove unused RAY_SERVE_HAPROXY_INGRESS_RETRY_ON and RAY_SERV… (ray-project#68175)
  [Data] Implement lineage tracker garbage collection upon marking operator durable (ray-project#67788)
  [Core] Fix recovery suppression race in ObjectRecoveryManager (ray-project#64974)
  [serve] Pass grpc.so_reuseport to gRPC as an integer (ray-project#66816)
  [Autoscaler][v2] Preserve retry accounting when trimming status history (ray-project#67360)
  [doc] Refresh intersphinx inventory snapshots (ray-project#68069)
  [Doc] Add item to AGENTS.md to update documentation when modifying API-facing code. (ray-project#68204)
  [Train] Preserve `num_boost_round=0` and avoid mutating `callbacks` in legacy LightGBM/XGBoost trainers (ray-project#68145)
  [Train] Raise in prepare_data_loader instead of dropping a custom batch_sampler (ray-project#68201)
  [air] Return None in skip_exceptions when exc or StartTraceback cause is None (ray-project#68172)
  [Train] Fix `checkpoint_manager` to support metrics only reported in a validation function (ray-project#67909)
  [Train] Count elastic workers in resource units for fractional resources (ray-project#68139)
  [Data] Refactor Disk shuffle map to use map Operator (ray-project#67130)
  [core][move semantics][2/n] Change state to moved on consumption and add enforcements (ray-project#66767)
  [Data] Propagate labels to object-store SHUFFLE_V2 map/reduce tasks (ray-project#67135)
  feat(core): add Cambricon MLU resource support (ray-project#66368)
  [Data] Make `zero_copy_batch=False` copy default batches (ray-project#67154)
  [data] Fix Parquet filename suffix detection (ray-project#67156)
  [data] Fix docstring/behavior mismatches in read_json and Dataset.sort (ray-project#66927)
  [Data] Run video_object_detection.ray on T4s with more memory (ray-project#68149)
  [core][sandbox] Let a node override network="public" sandboxes' default resolvers (ray-project#67691)
  [Data] read_mcap: resolve each sample's steps and lay out its row (ray-project#67925)
  [Data] read_mcap: sample read inputs: TopicSpec, steps, match and samples checks (ray-project#67924)
  [Data] Keep a map task's buffered rows when max_errored_blocks skips its error; read past corrupt MCAP chunks (ray-project#67167)
  [core][move semantics][1/n] Adding consume_once flag in actor methods (ray-project#66620)
  [Data] Size the schema pin and the columns Ray adds in the Parquet decoded-size estimate [3/6] (ray-project#67838)
  [Serve][LLM][1/3] Token based autoscaling: track autoscaling workload in KV router (ray-project#67937)
  [Data] Add native direct reads for nonpartitioned Hive tables (ray-project#67114)
  [Train] Count elastic workers in resource units for fractional resources (ray-project#68139)
  [Data] Add `condition` to `Dataset.join` and extract equi-join keys from it (ray-project#67590)
  [data] Add Top-N execution (ray-project#66931)
  [build] Bazel 8.8.1: bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0 (ray-project#66483)
  [build] Use Arena::Create and pin Java 8 bytecode ahead of protobuf 29 / rules_java 8 (ray-project#67720)
  [core][sandbox] Don't join a sandbox's cgroup on every exec (ray-project#67690)
  [Data] Run `training_ingest_regression_test` on Instances that can be Provisioned (ray-project#68110)
  [Data] Run Sort on Disk-based Shuffle under SHUFFLE_V2 (ray-project#67850)
  [core] Deflake tests that race on timing (ray-project#67717)
  [ci] Bump pypa manylinux2014 base image to 2026.10.22-0 for CPython 3.15 (ray-project#67285)
  [Data] Make Mcap Data Column Test Insensitive to Struct Field Order (ray-project#68109)
  [Data] read_mcap V2: cut large files where their log-time order holds (ray-project#67166)
  [core][sandbox] Commit sandbox as new registry image (ray-project#67972)
  [Data] read_mcap V2: read neighbouring chunks in one ranged request (ray-project#67930)
  [train][4/n] Wire WorkerProbes through the HealthManager (ray-project#67915)
  [train][3/n] Add WorkerProbe results and health reports to the HealthState (ray-project#67879)
  [Data] (Data fault tolerance ray-project#6/6) Add a manual release test for lineage reconstruction (ray-project#67519)
  [train][6/n] Act on health decisions in the Ray Train controller (ray-project#67856)
  [Train] Do not start an elastic worker group with zero workers (ray-project#68101)
  [Train] Keep the last checkpoint after a metrics-only report in local mode (ray-project#68070)
  [Data][2/n] Migrate GPU Shuffle to Unified Phase Metrics (ray-project#67359)
  [Data][Chkpt] Fix `ArrowInvalid` when loading large generated-ID checkpoints (ray-project#68065)
  [core][test] Create auth token in test_redis_wrong_password (ray-project#68014)
  [data][actor-only][2/n] Actor-only backend: promote task pools to actor pools (ray-project#67698)
  [Data] Fix typing mismatches in Dataset public API (ray-project#62077)
  [Data] read_mcap V2: carry read tasks' summary records in the manifest (ray-project#67929)
  [Data] Distinguish generated ID collisions with synthesized columns (ray-project#67164)
  [Data] Fix incorrect scale-up in rate-based autoscaler for custom resources (ray-project#66647)
  [Data][Delta DSv2][1/7] Shared Parquet and partitioning fixes for the Delta reader (ray-project#67887)
  [Data] Eliminate redundant aggregation shuffle (ray-project#67198)
  [Data] Raise minimum supported pyarrow to 23.0.0 and test up to 26.0.1 (ray-project#68005)
  [Train] Use the default validation timeout when a report's ValidationTaskConfig has none (ray-project#68045)
  [doc][train] Use the global batch size in the TensorFlow Train examples (ray-project#68006)
  [TPU] Default multi-slice MASTER_ADDR to slice 0 host 0 in SlicePlacementGroup (ray-project#67878)
  [Data] Estimate a Parquet row group's decoded Arrow bytes from its footer [2/6] (ray-project#67837)
  [Data] Predicate pushdown through joins: split conjuncts, derive one-sided OR weakenings (ray-project#67122)
  [Data] Add partition_filters to read_delta and take file sizes from the log (ray-project#67900)
  [Data] Stop writing Unity Catalog vended credentials to the environment (ray-project#67899)
  [Data] Raise an error for Delta tables with column mapping in write_delta (ray-project#67898)
  [Core] Add initial foundational configurations and flags for fork server (ray-project#67356)
  [core][sandbox] Image layers caching (ray-project#67874)
  [Doc][Train] Add device stream batch prefetching section to accelerator guide (ray-project#67779)
  [Train] Use user-defined train function name in training thread name and error stack traces (ray-project#64292)
  [serve][docs] Remove beta label from custom autoscaling policies (ray-project#68046)
  [core] Don't treat a gRPC client with calls in flight as idle (ray-project#67718)
  [train] Fix checkpoint sorting to support nested metrics (ray-project#59130)
  [train][5/n] Run ControllerProbes on the controller's node monitor (ray-project#67671)
  [Train] Rank a checkpoint without the score attribute as worst in min order (ray-project#67901)
  [Core][RuntimeEnv] Stream runtime environment package downloads (ray-project#67184)
  [core] Deflake `_run_hanging_command` tests (ray-project#67461)
  [Data] Add Gravitino Iceberg REST catalog provider (ray-project#67007)
  [Data] Fix mixed-root partition filtering in Parquet and ORC V2 (ray-project#67067)
  [core] Keep absl logging at ERROR unless GRPC_VERBOSITY is set (ray-project#67719)
  [Data][1/n]Add Shuffle Metrics Unification Instrumentation (ray-project#66968)
  [ci][release] Stop publishing Python 3.10 wheels and images (ray-project#67722)
  [serve] Copy the ASGI lifespan state into each request scope (ray-project#67844)
  [Serve][LLM][4/5] Support scale-to-zero application routing (ray-project#67087)
  [core][sandbox] Let GPU sandboxes request NVIDIA profiling capabilities (ray-project#67974)
  [Serve][LLM] Exclude LLMRouter from dashboard QPS (ray-project#67873)
  [Data] Read the Parquet SizeStatistics PyArrow doesn't expose [1/6] (ray-project#67836)
  Add perf metrics for 2.60.0 (ray-project#67727)
  [Observability / Train] Add Grafana Annotation support with Ray Train usage (ray-project#66958)

Signed-off-by: Eugene Paniot <e.paniot@gmail.com>

# Conflicts:
#	python/ray/data/_internal/planner/plan_udf_map_op.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants