Repository navigation
[Data] Collect write stats in-flight instead of teeing write-task input - #66708
FreemanDane wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request optimizes memory usage during write operations and checkpoints in Ray Data by streaming blocks and collecting metadata on-the-fly instead of buffering entire blocks in memory. It introduces a mechanism to capture only ID columns for post-write checkpoints and accumulates write statistics as blocks flow to the sink. The feedback suggests further optimizing memory by using 'pop' instead of 'get' when retrieving checkpoint blocks, write statistics, and write return values from 'ctx.kwargs' in the final stages, which immediately releases these references.
`generate_write_fn` used `itertools.tee` so the write and the stats stage could each read the task's blocks. The two readers run one after the other -- the sink drains its copy first -- so tee buffers the entire task input until the stats stage reads it, and `generate_collect_write_stats_fn` then held every block again to count rows and bytes. The stats are two integers, so accumulate them as the blocks stream to the sink (`_WriteStats` / `_StatsCollectingBlocks`) and stash them on `TaskContext.kwargs`. The write no longer re-emits the blocks; a sink that stops early is followed by a drain, so the stats still cover the whole task input exactly as before. The final stages take these values off `ctx.kwargs` with `pop`, releasing the references -- notably the captured ID blocks below -- as soon as they are consumed. `generate_collect_write_stats_fn` reads the stats after draining its input -- draining is what runs the write stage, and the write is what sets the stats -- and falls back to counting the blocks it is handed directly (see test_bigquery). Checkpointing adapts to the write no longer re-emitting blocks: - Non-file datasinks write their checkpoint after the data and took the ID column from the re-emitted blocks. A new pre-write transform now picks the ID columns up while the blocks stream and keeps only them; the post-write transform writes the checkpoint from those. Checkpoint timing and content are unchanged; the ID column check runs during the stream instead of after the write. - `commit_checkpoints` drains instead of materializing blocks it no longer receives. Tests: new python/ray/data/tests/unit/test_plan_write_op.py (stats parity, partial sink consumption, a weakref regression test for the tee buffering, the deferred write stage, checkpoint ID capture), plus test_bigquery.py::TestWriteBigQuery and the test_checkpoint.py 2PC tests. Developed with AI assistance (see PR description). Signed-off-by: FreemanDane <thss15_fengs@163.com>
a00000f to
88952da
Compare
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
Why are these changes needed?
generate_write_fnusesitertools.teeso the write and the write-stats stage can each read the write task's blocks. The two readers are consumed one after the other — the sink drains its copy first — so tee buffers the entire task input until the stats stage reads it, andgenerate_collect_write_stats_fnthen holds every block again to count rows/bytes. A write task therefore keeps its whole input in memory while writing.The stats are just two integers, so this PR accumulates them while the blocks stream to the sink (
_WriteStats/_StatsCollectingBlocks, stashed onTaskContext.kwargs) and stops re-emitting the written blocks. The final stages take these values offctx.kwargswithpop, releasing the references (notably the captured ID blocks below) as soon as they are consumed. A sink that stops early is still followed by a drain, so the reported stats cover the whole task input exactly as before. Direct callers ofgenerate_collect_write_stats_fn(seetest_bigquery.py) fall back to counting the blocks they are handed.One subtlety pinned by a test: the write stage is lazy — it runs when the stats stage first pulls on its output — so the stats must be read off
ctxonly after draining.Checkpointing adapts to the write no longer re-emitting blocks:
commit_checkpointsdrains instead of materializing the blocks it no longer receives.Follow-up (not in this PR): the 2-phase-commit pre-write transform still materializes the task's blocks (
_combine_blocks), because the pending checkpoint must cover the whole task before any data is written. Fixing that needs a recovery-semantics change.Duplicate check (per AGENTS.md)
Searched open PRs/issues for
write memory,itertools.tee,write stats iterator,plan_write_op,generate_write_fn,tee write blocks,write task memory blocks— no in-flight work touches this. Closest history: #57108 introduced the tee-based iterator (this PR fixes its memory behavior); #61821 added the checkpoint module adapted here; #66146 is a different (driver-side) memory issue. Base is current with master for both files (2a18b892).Tests
pytest python/ray/data/tests/unit/test_plan_write_op.py— 9 passed (new file; includes a weakref regression test that fails on the old tee implementation, and a test that drives the real transform chain to pin the lazy "write runs on first pull" ordering)pytest python/ray/data/tests/datasource/test_bigquery.py::TestWriteBigQuery— 3 passedpytest python/ray/data/tests/test_checkpoint.py::test_checkpoint_end_to_end ::test_prepare_checkpoint_transform_writes_pending ::test_write_block_checkpoint_with_pandas_df ::test_2pc_fail_retry_cleans_pending_checkpoints ::test_partial_failure_no_duplicates— 6 passed, 1 skippedblack/ruff(incl. import order) /pydoclintcleanThis change was developed with AI assistance and reviewed line-by-line by the submitter.