Skip to content

[Data] Collect write stats in-flight instead of teeing write-task input - #66708

Open
FreemanDane wants to merge 3 commits into
ray-project:masterfrom
FreemanDane:features/optimize-data-sink-memory-usage
Open

FreemanDane wants to merge 3 commits into
ray-project:masterfrom
FreemanDane:features/optimize-data-sink-memory-usage

Conversation

@FreemanDane

@FreemanDane FreemanDane commented Oct 4, 2026 •

Copy link
Copy Markdown

Why are these changes needed?

generate_write_fn uses itertools.tee so the write and the write-stats stage can each read the write task's blocks. The two readers are consumed one after the other — the sink drains its copy first — so tee buffers the entire task input until the stats stage reads it, and generate_collect_write_stats_fn then holds every block again to count rows/bytes. A write task therefore keeps its whole input in memory while writing.

The stats are just two integers, so this PR accumulates them while the blocks stream to the sink (_WriteStats / _StatsCollectingBlocks, stashed on TaskContext.kwargs) and stops re-emitting the written blocks. The final stages take these values off ctx.kwargs with pop, releasing the references (notably the captured ID blocks below) as soon as they are consumed. A sink that stops early is still followed by a drain, so the reported stats cover the whole task input exactly as before. Direct callers of generate_collect_write_stats_fn (see test_bigquery.py) fall back to counting the blocks they are handed.

One subtlety pinned by a test: the write stage is lazy — it runs when the stats stage first pulls on its output — so the stats must be read off ctx only after draining.

Checkpointing adapts to the write no longer re-emitting blocks:

  • Non-file datasinks write their checkpoint after the data and took the ID column from the re-emitted blocks. A new pre-write transform now picks up the ID columns (only that one column) while the blocks stream, and the post-write transform writes the checkpoint from them. Checkpoint timing and content are unchanged. One observable difference: the ID-column existence check now runs during the stream instead of after the write (the task still fails either way).
  • commit_checkpoints drains instead of materializing the blocks it no longer receives.

Follow-up (not in this PR): the 2-phase-commit pre-write transform still materializes the task's blocks (_combine_blocks), because the pending checkpoint must cover the whole task before any data is written. Fixing that needs a recovery-semantics change.

Duplicate check (per AGENTS.md)

Searched open PRs/issues for write memory, itertools.tee, write stats iterator, plan_write_op, generate_write_fn, tee write blocks, write task memory blocks — no in-flight work touches this. Closest history: #57108 introduced the tee-based iterator (this PR fixes its memory behavior); #61821 added the checkpoint module adapted here; #66146 is a different (driver-side) memory issue. Base is current with master for both files (2a18b892).

Tests

  • pytest python/ray/data/tests/unit/test_plan_write_op.py — 9 passed (new file; includes a weakref regression test that fails on the old tee implementation, and a test that drives the real transform chain to pin the lazy "write runs on first pull" ordering)
  • pytest python/ray/data/tests/datasource/test_bigquery.py::TestWriteBigQuery — 3 passed
  • pytest python/ray/data/tests/test_checkpoint.py::test_checkpoint_end_to_end ::test_prepare_checkpoint_transform_writes_pending ::test_write_block_checkpoint_with_pandas_df ::test_2pc_fail_retry_cleans_pending_checkpoints ::test_partial_failure_no_duplicates — 6 passed, 1 skipped
  • black / ruff (incl. import order) / pydoclint clean

This change was developed with AI assistance and reviewed line-by-line by the submitter.

@FreemanDane
FreemanDane requested a review from a team as a code owner October 4, 2026 20:16

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes memory usage during write operations and checkpoints in Ray Data by streaming blocks and collecting metadata on-the-fly instead of buffering entire blocks in memory. It introduces a mechanism to capture only ID columns for post-write checkpoints and accumulates write statistics as blocks flow to the sink. The feedback suggests further optimizing memory by using 'pop' instead of 'get' when retrieving checkpoint blocks, write statistics, and write return values from 'ctx.kwargs' in the final stages, which immediately releases these references.

@ray-gardener ray-gardener Bot added data Ray Data-related issues community-contribution Contributed by the community labels Oct 5, 2026
`generate_write_fn` used `itertools.tee` so the write and the stats stage
could each read the task's blocks. The two readers run one after the other
-- the sink drains its copy first -- so tee buffers the entire task input
until the stats stage reads it, and `generate_collect_write_stats_fn` then
held every block again to count rows and bytes.

The stats are two integers, so accumulate them as the blocks stream to the
sink (`_WriteStats` / `_StatsCollectingBlocks`) and stash them on
`TaskContext.kwargs`. The write no longer re-emits the blocks; a sink that
stops early is followed by a drain, so the stats still cover the whole task
input exactly as before. The final stages take these values off `ctx.kwargs`
with `pop`, releasing the references -- notably the captured ID blocks below
-- as soon as they are consumed.

`generate_collect_write_stats_fn` reads the stats after draining its input
-- draining is what runs the write stage, and the write is what sets the
stats -- and falls back to counting the blocks it is handed directly (see
test_bigquery).

Checkpointing adapts to the write no longer re-emitting blocks:

- Non-file datasinks write their checkpoint after the data and took the ID
  column from the re-emitted blocks. A new pre-write transform now picks the
  ID columns up while the blocks stream and keeps only them; the post-write
  transform writes the checkpoint from those. Checkpoint timing and content
  are unchanged; the ID column check runs during the stream instead of after
  the write.
- `commit_checkpoints` drains instead of materializing blocks it no longer
  receives.

Tests: new python/ray/data/tests/unit/test_plan_write_op.py (stats parity,
partial sink consumption, a weakref regression test for the tee buffering,
the deferred write stage, checkpoint ID capture), plus
test_bigquery.py::TestWriteBigQuery and the test_checkpoint.py 2PC tests.

Developed with AI assistance (see PR description).

Signed-off-by: FreemanDane <thss15_fengs@163.com>
@FreemanDane
FreemanDane force-pushed the features/optimize-data-sink-memory-usage branch from a00000f to 88952da Compare October 5, 2026 04:22
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Oct 19, 2026
@github-actions github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant