Skip to content

[Data] read_mcap V2: read neighbouring chunks in one ranged request - #67930

Merged
richardliaw merged 10 commits into
masterfrom
marwan/mcap-dsv2-ranged-reads
Nov 12, 2026
Merged

richardliaw merged 10 commits into
masterfrom
marwan/mcap-dsv2-ranged-reads

Conversation

@marwan116

@marwan116 marwan116 commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A read task fetches neighbouring chunks in one ranged request instead of one request per chunk. The listing holds at most one read task's budget of summaries, and a read task's memory is estimated from what its chunks hold. Also four small read_mcap fixes that fail early or speak up instead of failing late or quietly.

How to review

Start with mcap_io.plan_chunk_reads: it groups neighbouring chunks into one read. read_chunks then frees each read's bytes after its last chunk.

chunks a task reads   [c3][c4][c5]      [c9]
before                 req  req  req     req
after                 [  one request ]  [req]
File What changes Lines
mcap_io.py Plans and makes the ranged reads. Reads a summary's tail, then its section. 202
mcap_chunks.py Reads a task's chunks through them. 70
mcap_summary.py, mcap_file_indexer.py, mcap_data_column.py, mcap_datasource_v2.py Bound the summaries a listing holds. Estimate a task's peak from its chunks. 417
read_api.py, context.py The small read_mcap items and the docstring. 89
test_mcap_io.py, test_mcap_reader.py, test_mcap.py Reads match one request per chunk. 298
test_mcap_listing.py, test_mcap_read_memory.py The listing's byte bound, and the estimate against measured peaks. 388

Before

  • Each chunk was its own request, even when chunks sat next to each other in the file.
  • The listing read summaries a few at a time.
  • Each chunk was weighed as 1.2 times the table it produces. Large chunks and small messages peak well above that.
  • override_num_blocks was ignored without a word, and a generated checkpoint id column failed with a misleading error.

After

  • Chunks that sit next to each other are read in one ranged request (mcap_io).
  • The listing reads 32 summary tails at once by default. A tail gives the summary section's exact length, and sections are fetched in file order while they fit one read task's budget. Threads only fetch, and the listing parses one summary at a time.
  • A chunk weighs its share of the task's table, scaled so that each task leaves room for its largest chunk's working set and one ranged read. Every term is sized from the summary.
  • A file whose selected channels are all JSON counts each payload byte as the Python values it decodes into, which hold several times its bytes.
  • override_num_blocks or parallelism logs a warning on V2, since tasks are sized by bytes. The warning names the argument the caller passed and prints the configured byte budget.
  • A generated checkpoint id column raises InvalidCheckpointingConfig. The legacy path refuses a row_id checkpoint as it refuses include_row_id.
  • The read_mcap docstring describes the V2 datasource and how its rows differ from the legacy reader's.

Measured (local disk):

Case Before After
Listing 100 copies of a 10,000-chunk, 20-channel file: summaries held 1.3 GiB 240 MiB, same time
Listing 600 copies of a real recording: memory added 81 MiB 37 MiB
Read of 20,000 small messages: estimate ÷ measured peak 0.25 to 1.08 1.45 to 1.50

Decisions to confirm:

  • The estimate errs high on small messages, so they pack into more tasks: 600 copies of a real recording make 5,405 tasks instead of 4,803.
  • The per-message sizes are upper bounds for CPython 3.10, which also covers 3.12.
  • The listing task doesn't reserve its summary budget as memory yet. That needs a change in the shared plan_list_files_op.py.

Subsystems

IO (_internal/datasource_v2/formats/mcap), API (read_api.py).

Testing

PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \
  python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \
  python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py

249 passed, 4 skipped. pre-commit and pyrefly clean.

Not a duplicate: split out of #67166 at a reviewer's request. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally.

🤖 Generated with Claude Code

@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes MCAP file reading in the DataSourceV2 pipeline by implementing ranged reads, which group nearby chunk records into a single read operation to reduce I/O latency. It introduces configuration options for chunk grouping, updates the message reader to utilize these ranged reads, and enhances error handling and documentation for read_mcap. The review feedback recommends wrapping the chunk iteration in read_chunks with a try...finally block to ensure the underlying BytesIO stream is safely closed and memory is freed even if the generator is terminated early or encounters an exception.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@ray-gardener ray-gardener Bot added docs An issue or change related to documentation data Ray Data-related issues labels Nov 7, 2026
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-ranged-reads branch from 3416715 to 07ac1ed Compare November 7, 2026 08:16
@marwan116

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 07ac1ed. Configure here.

@marwan116 marwan116 added the go add ONLY when ready to merge, run all tests label Nov 7, 2026
@marwan116
marwan116 force-pushed the marwan/mcap-dsv2-ranged-reads branch 2 times, most recently from b6ea1d4 to 6d1086e Compare November 9, 2026 17:56
@marwan116
marwan116 removed this pull request from stack #67931 November 9, 2026 19:35
@marwan116
marwan116 added this pull request to stack #67983 November 9, 2026 19:35
@marwan116 marwan116 self-assigned this Nov 10, 2026
Base automatically changed from marwan/mcap-dsv2-task-summary to master November 11, 2026 22:36
marwan116 and others added 10 commits November 11, 2026 14:36
Problem:
- A read task fetched each chunk in its own request. On object storage
  each request waits for a round trip, so a task of many chunks spent
  most of its time waiting.

Change:
- mcap_io.read_chunks groups chunks, in the order the reader needs
  them, into ranged reads (ChunkReads). A chunk joins the read before
  it when it starts after that read, within a gap of 1 MiB, and the
  read stays within 16 MiB.
- A file's first read spans at most 4 MiB, and each later one may span
  twice as much, so a read that stops early, as under a limit, fetches
  little.
- A read is fetched when its first chunk is needed, and its bytes are
  released once its last chunk is parsed.
- Both the file-order path and the log-time merge read this way. The
  merge plans its reads in the order it expands chunks.
- RAY_DATA_MCAP_CHUNK_READ_BYTES, RAY_DATA_MCAP_CHUNK_READ_GAP_BYTES and
  RAY_DATA_MCAP_CHUNK_READ_FIRST_BYTES set the limits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- One listing task reads every summary under its path, 16 at a time.
  Each summary read waits on request latency, so on a folder of many
  small files that wait was the slow part of the read.

Change:
- RAY_DATA_MCAP_SUMMARY_IO_CONCURRENCY defaults to 32. Up to twice that
  many read summaries are held at once. Planning reads summaries with
  the same default.
- A test checks that the listing rows come out in the same order with
  one summary read at a time or 32, so the packer forms the same read
  tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The comment on _READ_PEAK_FACTOR named only the table being built
  and the chunk being decompressed. A read task also holds the ranged
  read that fetched the chunk, so one table's read peaks a little
  higher.

Change:
- The comment names the ranged read and gives the measured range. The
  factor stays 1.2.
- Comment only, no behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The V2 datasource sizes read tasks by bytes, so it ignores
  override_num_blocks without saying so.

Change:
- read_mcap logs a warning when override_num_blocks (or parallelism)
  is set on the V2 path, and the docstring says so.
- test_read_mcap_override_num_blocks checks the warning on V2 and its
  absence on the legacy path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- CheckpointConfig(generated_id_column="row_id") also reports
  id_column == "row_id". read_mcap then turns on its own text row_id,
  where checkpointing expects the struct ids only read_parquet builds.
- With any other generated name, the write fails later with a
  misleading "ID column ... is absent" error.
- On the V1 legacy path, CheckpointConfig(id_column="row_id") neither
  turned row_id on nor raised, so the read had no ids for the
  checkpoint.

Change:
- read_mcap raises InvalidCheckpointingConfig for a generated id
  column, on both read paths, and points to
  CheckpointConfig(id_column="row_id").
- row_id is turned on only for a user id_column of "row_id", on both
  read paths. The V1 legacy path then raises the NotImplementedError it
  raises for include_row_id=True.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- read_mcap switches to the V2 datasource by default, but its
  docstring did not say so, or how its rows differ.
- The use_datasource_v2 docstring did not name MCAP.

Change:
- The read_mcap docstring lists the three differences from the legacy
  reader, how to switch back, and an example that stores schema_data
  as plain bytes.
- The use_datasource_v2 docstring names MCAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The V2 warning names override_num_blocks even when the caller
  passed parallelism.
- It hard-codes 128 MiB instead of the configured budget.

Change:
- Name the argument the caller gave.
- Print the budget from RAY_DATA_MCAP_BIN_PACKING_BYTES.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The chunk-fetch test ran four read configs, but one is its own
  baseline and two make the same single read on the test file.
- The parallelism warning had its own test beside the
  override_num_blocks test, which varied only the value.
- The generated-id test ran two names that hit the same raise.

Change:
- Compare one chunk per read with small reads only.
- Parametrize the override_num_blocks test over the argument name and
  drop the separate parallelism test.
- Check the generated-id raise with the name row_id only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- Listing threads read and parsed summaries, and up to twice the
  summary concurrency of parsed summaries were held at once. Parsed
  summaries are much larger than their bytes, and nothing bounded
  their size. Parsing holds the GIL, so the threads gained nothing
  from it.
- A read task's peak was estimated as 1.2 times its table. That
  misses the chunk being decompressed and parsed, the log-time heap
  and the ranged read, so a task of large chunks or small messages
  peaked at up to 2.6 times its estimate.

Change:
- Summary reads are split in two: a tail read, whose footer gives the
  section's exact length, and a section read. Listing threads only
  fetch bytes, and the caller parses each section in order.
- A section is fetched once it fits the read task budget
  (RAY_DATA_MCAP_BIN_PACKING_BYTES) with the sections held. With
  nothing held, the next one is fetched whatever its size. A failed
  read is still that file's own error.
- A read task's peak is estimated as its table, plus the largest
  working set of one chunk, plus one ranged read. Each chunk weighs
  its table scaled so that a full task leaves room for the rest.
- New tests pin the estimate's constants by measurement, check the
  estimate against a measured read, and check the byte budget of
  summary reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Problem:
- The estimate counts a payload's raw bytes, but a JSON payload is
  decoded into Python values that hold several times as many bytes
  until its table is built. A read of small JSON messages peaks above
  the estimate.

Change:
- When every selected channel of a file is JSON-encoded, each payload
  byte counts the reader's own bytes-per-payload-byte for decoded
  JSON.
- The peak test also reads JSON payloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@richardliaw
richardliaw merged commit dd26df2 into master Nov 12, 2026
6 checks passed
@richardliaw
richardliaw deleted the marwan/mcap-dsv2-ranged-reads branch November 12, 2026 19:33
eugenepaniot added a commit to eugenepaniot/ray that referenced this pull request Nov 19, 2026
…floop-input

* upstream/master:
  [Data][3/n] update progress bar on lineage reconstruction (ray-project#67366)
  [Data] Enable the Top-N rewrite by default and expose it on DataContext (ray-project#68329)
  [Data] Adaptively skip map-side partial aggregation for high-cardinality keys (ray-project#68118)
  [Core] Spawn specific servers without preexec_fn to fix intermittent startup crashes (ray-project#67947)
  [Serve] Stop the __wrapped__ walk at non-callables (ray-project#67117)
  [Data] Deflake test_read_parquet_v2 by cutting redundant Ray Data executions (ray-project#68318)
  [Data] Retry FooterReader actor deaths with Ray Core and yield footers in listing order (ray-project#67988)
  [Data] Fix order-dependent schema unification when a list<null> arm is present (ray-project#68259)
  [Serve] Make @serve.batch work when wrapped in another decorator (ray-project#68177)
  [serve] Deflake test_metrics_3 - record the batch queue length once a batch is taken (ray-project#68302)
  [serve] Drop completed requests from the multiplexed model ID index (ray-project#68085)
  [Data] Fix flaky test_hudi: read each table once instead of sorting in Ray Data, keep size small (ray-project#68162)
  [Data] Fix flaky test_context: start 2 Ray clusters instead of 5 and keep size small (ray-project#68161)
  [Data] Fix flaky test_kafka: use recent timestamps in test_read_kafka_datetime_partial_range (ray-project#68248)
  [Data] Iceberg DSv2 [2/5]: The listing-to-reading manifest for Iceberg (ray-project#67951)
  [Train] Report training errors when the post-mortem debugger raises a BaseException (ray-project#68317)
  [RLlib] Skip the PPOConfig docstring example that crashes the rllib doctest target (ray-project#68306)
  [Train] Support post-mortem debugging for training function errors (ray-project#68296)
  [train][1/n] Give each health probe its own timeout (ray-project#68191)
  [Data] Fix unifying a fixed-shape tensor array with a lower-ndim variable-shaped one (ray-project#67266)
  [Train] Add a guide to debugging failures and hangs (ray-project#68299)
  [Data] Persist BlockMetadata.num_rows and batch ray.get in _calculate_blocks_rows (ray-project#67662)
  [Data] Vectorize LeRobot delta segment preparation and validate index contiguity (ray-project#67845)
  [Data] Avoid full-DataFrame apply in _AbstractKBinsDiscretizer._transform_pandas (ray-project#67862)
  [doc] Refresh intersphinx inventory snapshots (ray-project#68303)
  [Data] Fix SimpleImputer on categorical columns in append mode and with existing category values (ray-project#67467)
  [Data] Replace O(N) list.pop(0) with deque.popleft() in PushBasedShuffleTaskScheduler (ray-project#67243)
  [Train] Reject an elastic ScalingConfig with max_workers=0 (ray-project#68218)
  [core][move semantics][3/n]Populate task args with is_move for movable refs (ray-project#66803)
  [core][sandbox] Remove the readonly option (ray-project#68088)
  [Data](1/n) Implement the line-chunk reader for CSV and JSONL on DataSource V2 (ray-project#67907)
  [Data] Set MIMALLOC_PURGE_DELAY=0 on random_shuffle to fix the pyarrow 25 OOM regression (ray-project#68151)
  [core] Quote each `uv run` argument in the uv runtime env hook (ray-project#68156)
  [core] Don't let gRPC cancel calls that wait over 30s in the server queue (ray-project#68210)
  [Data] Fix shuffle and join on native tensor columns with pyarrow 25+ (ray-project#68072)
  [serve][llm] Remove decode load progress (ray-project#68258)
  [ci] Bump sglang to 0.5.23 in release tests (ray-project#68250)
  [Core] Build the test auth token file locally so Windows tests can read it (ray-project#68262)
  [image] Bump EFA installer to 1.51.0 in CPU base-extra images (ray-project#67730)
  [serve] Observe the result of rank reassignment reconfigures (ray-project#67923)
  [Serve][Docs] Explain imperative vs declarative deploys and in-place updates from Python (ray-project#68243)
  [Data] (Fault tolerance) Ray Data lineage reconstruction docs (ray-project#68061)
  [RLlib] Add a multi-node footsies APPO memory release test (ray-project#67595)
  [Train] Remove local mode (ray-project#68239)
  [Data] Fix filtered Lance row-count metadata (ray-project#67228)
  [Serve][2/2] Propagate TracingConfig to proxies at startup (ray-project#65437)
  [Data] Fix synthesized path filtering in DataSourceV2 (ray-project#67079)
  [RLlib] Run the BCConfig Tune example one trial at a time to unblock the rllib doctest target (ray-project#68242)
  [Data][Delta DSv2][2/7] List Delta Lake table files from the log (ray-project#67888)
  [Data] Parallelize Parquet and ORC full-schema reads (ray-project#67542)
  [Data] Random Shuffle on Shuffle V2 Map/Reduce Operators (ray-project#68004)
  [Data] Propagate metadata prefetch thread failures to executor (ray-project#68171)
  [core] Send generator consumed-progress updates from io_service_ (ray-project#67160)
  [Data] Add @ray-data-connectors as codeowner for datasource/sinks files (ray-project#68154)
  [data][llm] Keep the prompt in vLLM error rows (ray-project#67665)
  [data][actor-only][3/n] Actor-only backend: operator fusion (ray-project#67699)
  [Data] Read text files on DataSource v2 (ray-project#68063)
  [Autoscaler][KubeRay] Never patch worker group replicas below minReplicas (ray-project#67332)
  [Serve] Show progress of updating (ray-project#67107)
  [core][test] Trigger usage-stats OOM test without exhausting host memory (ray-project#68009)
  [Serve] Remove unused RAY_SERVE_HAPROXY_INGRESS_RETRY_ON and RAY_SERV… (ray-project#68175)
  [Data] Implement lineage tracker garbage collection upon marking operator durable (ray-project#67788)
  [Core] Fix recovery suppression race in ObjectRecoveryManager (ray-project#64974)
  [serve] Pass grpc.so_reuseport to gRPC as an integer (ray-project#66816)
  [Autoscaler][v2] Preserve retry accounting when trimming status history (ray-project#67360)
  [doc] Refresh intersphinx inventory snapshots (ray-project#68069)
  [Doc] Add item to AGENTS.md to update documentation when modifying API-facing code. (ray-project#68204)
  [Train] Preserve `num_boost_round=0` and avoid mutating `callbacks` in legacy LightGBM/XGBoost trainers (ray-project#68145)
  [Train] Raise in prepare_data_loader instead of dropping a custom batch_sampler (ray-project#68201)
  [air] Return None in skip_exceptions when exc or StartTraceback cause is None (ray-project#68172)
  [Train] Fix `checkpoint_manager` to support metrics only reported in a validation function (ray-project#67909)
  [Train] Count elastic workers in resource units for fractional resources (ray-project#68139)
  [Data] Refactor Disk shuffle map to use map Operator (ray-project#67130)
  [core][move semantics][2/n] Change state to moved on consumption and add enforcements (ray-project#66767)
  [Data] Propagate labels to object-store SHUFFLE_V2 map/reduce tasks (ray-project#67135)
  feat(core): add Cambricon MLU resource support (ray-project#66368)
  [Data] Make `zero_copy_batch=False` copy default batches (ray-project#67154)
  [data] Fix Parquet filename suffix detection (ray-project#67156)
  [data] Fix docstring/behavior mismatches in read_json and Dataset.sort (ray-project#66927)
  [Data] Run video_object_detection.ray on T4s with more memory (ray-project#68149)
  [core][sandbox] Let a node override network="public" sandboxes' default resolvers (ray-project#67691)
  [Data] read_mcap: resolve each sample's steps and lay out its row (ray-project#67925)
  [Data] read_mcap: sample read inputs: TopicSpec, steps, match and samples checks (ray-project#67924)
  [Data] Keep a map task's buffered rows when max_errored_blocks skips its error; read past corrupt MCAP chunks (ray-project#67167)
  [core][move semantics][1/n] Adding consume_once flag in actor methods (ray-project#66620)
  [Data] Size the schema pin and the columns Ray adds in the Parquet decoded-size estimate [3/6] (ray-project#67838)
  [Serve][LLM][1/3] Token based autoscaling: track autoscaling workload in KV router (ray-project#67937)
  [Data] Add native direct reads for nonpartitioned Hive tables (ray-project#67114)
  [Train] Count elastic workers in resource units for fractional resources (ray-project#68139)
  [Data] Add `condition` to `Dataset.join` and extract equi-join keys from it (ray-project#67590)
  [data] Add Top-N execution (ray-project#66931)
  [build] Bazel 8.8.1: bump protobuf to v29.6, rules_java to 8.12.0 and gRPC to 1.69.0 (ray-project#66483)
  [build] Use Arena::Create and pin Java 8 bytecode ahead of protobuf 29 / rules_java 8 (ray-project#67720)
  [core][sandbox] Don't join a sandbox's cgroup on every exec (ray-project#67690)
  [Data] Run `training_ingest_regression_test` on Instances that can be Provisioned (ray-project#68110)
  [Data] Run Sort on Disk-based Shuffle under SHUFFLE_V2 (ray-project#67850)
  [core] Deflake tests that race on timing (ray-project#67717)
  [ci] Bump pypa manylinux2014 base image to 2026.10.22-0 for CPython 3.15 (ray-project#67285)
  [Data] Make Mcap Data Column Test Insensitive to Struct Field Order (ray-project#68109)
  [Data] read_mcap V2: cut large files where their log-time order holds (ray-project#67166)
  [core][sandbox] Commit sandbox as new registry image (ray-project#67972)
  [Data] read_mcap V2: read neighbouring chunks in one ranged request (ray-project#67930)
  [train][4/n] Wire WorkerProbes through the HealthManager (ray-project#67915)
  [train][3/n] Add WorkerProbe results and health reports to the HealthState (ray-project#67879)
  [Data] (Data fault tolerance ray-project#6/6) Add a manual release test for lineage reconstruction (ray-project#67519)
  [train][6/n] Act on health decisions in the Ray Train controller (ray-project#67856)
  [Train] Do not start an elastic worker group with zero workers (ray-project#68101)
  [Train] Keep the last checkpoint after a metrics-only report in local mode (ray-project#68070)
  [Data][2/n] Migrate GPU Shuffle to Unified Phase Metrics (ray-project#67359)
  [Data][Chkpt] Fix `ArrowInvalid` when loading large generated-ID checkpoints (ray-project#68065)
  [core][test] Create auth token in test_redis_wrong_password (ray-project#68014)
  [data][actor-only][2/n] Actor-only backend: promote task pools to actor pools (ray-project#67698)
  [Data] Fix typing mismatches in Dataset public API (ray-project#62077)
  [Data] read_mcap V2: carry read tasks' summary records in the manifest (ray-project#67929)
  [Data] Distinguish generated ID collisions with synthesized columns (ray-project#67164)
  [Data] Fix incorrect scale-up in rate-based autoscaler for custom resources (ray-project#66647)
  [Data][Delta DSv2][1/7] Shared Parquet and partitioning fixes for the Delta reader (ray-project#67887)
  [Data] Eliminate redundant aggregation shuffle (ray-project#67198)
  [Data] Raise minimum supported pyarrow to 23.0.0 and test up to 26.0.1 (ray-project#68005)
  [Train] Use the default validation timeout when a report's ValidationTaskConfig has none (ray-project#68045)
  [doc][train] Use the global batch size in the TensorFlow Train examples (ray-project#68006)
  [TPU] Default multi-slice MASTER_ADDR to slice 0 host 0 in SlicePlacementGroup (ray-project#67878)
  [Data] Estimate a Parquet row group's decoded Arrow bytes from its footer [2/6] (ray-project#67837)
  [Data] Predicate pushdown through joins: split conjuncts, derive one-sided OR weakenings (ray-project#67122)
  [Data] Add partition_filters to read_delta and take file sizes from the log (ray-project#67900)
  [Data] Stop writing Unity Catalog vended credentials to the environment (ray-project#67899)
  [Data] Raise an error for Delta tables with column mapping in write_delta (ray-project#67898)
  [Core] Add initial foundational configurations and flags for fork server (ray-project#67356)
  [core][sandbox] Image layers caching (ray-project#67874)
  [Doc][Train] Add device stream batch prefetching section to accelerator guide (ray-project#67779)
  [Train] Use user-defined train function name in training thread name and error stack traces (ray-project#64292)
  [serve][docs] Remove beta label from custom autoscaling policies (ray-project#68046)
  [core] Don't treat a gRPC client with calls in flight as idle (ray-project#67718)
  [train] Fix checkpoint sorting to support nested metrics (ray-project#59130)
  [train][5/n] Run ControllerProbes on the controller's node monitor (ray-project#67671)
  [Train] Rank a checkpoint without the score attribute as worst in min order (ray-project#67901)
  [Core][RuntimeEnv] Stream runtime environment package downloads (ray-project#67184)
  [core] Deflake `_run_hanging_command` tests (ray-project#67461)
  [Data] Add Gravitino Iceberg REST catalog provider (ray-project#67007)
  [Data] Fix mixed-root partition filtering in Parquet and ORC V2 (ray-project#67067)
  [core] Keep absl logging at ERROR unless GRPC_VERBOSITY is set (ray-project#67719)
  [Data][1/n]Add Shuffle Metrics Unification Instrumentation (ray-project#66968)
  [ci][release] Stop publishing Python 3.10 wheels and images (ray-project#67722)
  [serve] Copy the ASGI lifespan state into each request scope (ray-project#67844)
  [Serve][LLM][4/5] Support scale-to-zero application routing (ray-project#67087)
  [core][sandbox] Let GPU sandboxes request NVIDIA profiling capabilities (ray-project#67974)
  [Serve][LLM] Exclude LLMRouter from dashboard QPS (ray-project#67873)
  [Data] Read the Parquet SizeStatistics PyArrow doesn't expose [1/6] (ray-project#67836)
  Add perf metrics for 2.60.0 (ray-project#67727)
  [Observability / Train] Add Grafana Annotation support with Ray Train usage (ray-project#66958)

Signed-off-by: Eugene Paniot <e.paniot@gmail.com>

# Conflicts:
#	python/ray/data/_internal/planner/plan_udf_map_op.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues docs An issue or change related to documentation go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants