Repository navigation
[Data] Distribute the Iceberg filtered-overwrite rewrite across Ray tasks - #66144
Conversation
…asks `write_iceberg(mode=OVERWRITE, overwrite_filter=...)` handed the delete side to PyIceberg's `Transaction.delete()`, whose copy-on-write path reads and rewrites every partially matched data file one at a time in the calling process. That call happens in `on_write_complete`, which runs on the driver, so the row-level rewrite of a partial overwrite was serial single-process work on the driver: with 8 data files the driver read all 8 of them (800k rows) to write a 2-row dataset. Keep PyIceberg's two steps -- drop the files the filter covers entirely using manifest metadata, then rewrite the ones it covers only partially -- but dispatch one Ray task per file for the rewrite, the way `_commit_upsert_scan_merge` already does for UPSERT. Preserved rows are selected with PyIceberg's complement of the bound filter, so rows whose filter column is NULL are kept. Snapshot semantics are unchanged: the same `['overwrite', 'append']` (or `['delete', 'append']` when only whole files are dropped) pair is committed, and the files the metadata step already dropped are not planned for a rewrite. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
There was a problem hiding this comment.
Code Review
This pull request implements distributed row-level overwrites for Iceberg tables by executing partial file rewrites inside Ray tasks instead of on the driver. It introduces _rewrite_iceberg_file_by_filter to handle the streaming rewrite of individual files and updates _commit_overwrite to perform metadata-only deletes for fully covered files before dispatching Ray tasks for partially covered ones. Feedback suggests optimizing the file rewrite logic by collecting filtered Arrow tables in a list and performing a single pa.concat_tables call at the end of the loop to avoid incremental copy overhead.
| preserved_rows: Optional["pa.Table"] = None | ||
| total_in_rows = 0 | ||
| total_preserved_rows = 0 | ||
|
|
||
| for rb in record_batches: | ||
| batch_table = pa.Table.from_batches([rb]) | ||
| if len(batch_table) == 0: | ||
| continue | ||
| total_in_rows += len(batch_table) | ||
|
|
||
| kept = batch_table.filter(preserve_row_filter) | ||
| if len(kept) > 0: | ||
| if preserved_rows is None: | ||
| preserved_rows = kept | ||
| else: | ||
| preserved_rows = pa.concat_tables( | ||
| [preserved_rows, kept], promote_options="permissive" | ||
| ) | ||
| total_preserved_rows += len(kept) | ||
|
|
||
| if total_in_rows == 0: | ||
| return (None, []) | ||
|
|
||
| if total_preserved_rows == 0: | ||
| # Every row matches the filter, so the whole file goes away. | ||
| return (file_scan_task.file, []) | ||
|
|
||
| if total_preserved_rows == total_in_rows: | ||
| # The metrics said the file might match, but no row actually does. | ||
| return (None, []) |
There was a problem hiding this comment.
Repeatedly calling pa.concat_tables in a loop can be highly inefficient because it performs incremental copies and metadata updates for each batch. Instead, collect the filtered tables in a list and perform a single concatenation at the end of the loop, right before writing the data files. This also avoids any concatenation overhead if the function returns early.
preserved_batches: List["pa.Table"] = []
total_in_rows = 0
total_preserved_rows = 0
for rb in record_batches:
batch_table = pa.Table.from_batches([rb])
if len(batch_table) == 0:
continue
total_in_rows += len(batch_table)
kept = batch_table.filter(preserve_row_filter)
if len(kept) > 0:
preserved_batches.append(kept)
total_preserved_rows += len(kept)
if total_in_rows == 0:
return (None, [])
if total_preserved_rows == 0:
# Every row matches the filter, so the whole file goes away.
return (file_scan_task.file, [])
if total_preserved_rows == total_in_rows:
# The metrics said the file might match, but no row actually does.
return (None, [])
preserved_rows = pa.concat_tables(
preserved_batches, promote_options="permissive"
)There was a problem hiding this comment.
Applied in 328bbea, and to the sibling _rewrite_iceberg_file used by UPSERT since it had the same pattern.
For the record on the magnitude: with a uniform schema pa.concat_tables appends chunks without copying buffers, so what the repeated call costs is chunk-list bookkeeping rather than data copies. Measured on this machine: 1000 batches of 1000 rows took 12.6ms incrementally versus 1.0ms as a single concatenation, and 50 batches of 20,000 rows took 0.2ms versus 0.0ms. Worth doing, and the code is shorter, but it is not where the time in this path goes.
Both rewrite tasks accumulated preserved rows by calling `pa.concat_tables` on every RecordBatch, so each batch walked the chunk list built up so far. Collect the batches in a list and concatenate once, after the early returns, so a file that is deleted or left untouched does no concatenation at all. This is a small win: `concat_tables` on a uniform schema appends chunks without copying buffers, so the cost is chunk-list bookkeeping, measured at 12.6ms versus 1.0ms for 1000 batches of 1000 rows, and 0.2ms versus 0.0ms for 50 batches of 20,000 rows. The shorter code is the main benefit. Applied to the new `_rewrite_iceberg_file_by_filter` and, since it is the same pattern in the sibling function, to `_rewrite_iceberg_file` used by UPSERT. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
|
Thank you for your contribution! I am reviewing the changes for this PR and #66146 Will get back on this by the end of tomorrow! |
|
@ayushk7102 @AarryaSaraf, gentle ping for review when you have a chance. The branch still merges cleanly onto current master, and CI was green on the last run. Happy to rebase or split anything if that makes it easier to review. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit b100836. Configure here.
| results = ray.get(refs) | ||
|
|
There was a problem hiding this comment.
Consider using a loop with ray.wait() for the done/pending tasks, so that we can have observability into the rewrite progress and catch timeouts.
A bare ray.get(refs) without any timeout blocks forever if a rewrite task stalls.
| results = ray.get(refs) | |
| results = [] | |
| pending = list(refs) | |
| try: | |
| while pending: | |
| done, pending = ray.wait( | |
| pending, num_returns=len(pending), | |
| timeout=_REWRITE_STALL_TIMEOUT_S | |
| ) | |
| if not done: | |
| raise TimeoutError( | |
| f"No Iceberg overwrite rewrite task finished in " | |
| f"{_REWRITE_STALL_TIMEOUT_S}s; | |
| {len(pending)}/{len(refs)} " | |
| "still pending." | |
| ) | |
| results.extend(ray.get(done)) | |
| logger.debug( | |
| "[overwrite] rewrite progress: %d/%d file(s) done | |
| (%.1fs elapsed)", | |
| len(results), | |
| len(refs), | |
| time.perf_counter() - t0, | |
| ) | |
| except BaseException: | |
| for ref in pending: | |
| ray.cancel(ref) | |
| raise |
There was a problem hiding this comment.
I think we can modify the existing UPSERT rewrite loop so that we do the same thing there too. I noticed we are currently not catching a timeout and rather retrying infinitely.
To protect scope let's keep that out of this PR, and I can take this as a follow up!
| n_partial, | ||
| n_untouched, | ||
| ) | ||
|
|
There was a problem hiding this comment.
Can we early return here?
if n_whole_delete == 0 and n_partial == 0:
return
| ) | ||
| assert rows_same(result, expected) | ||
|
|
||
| @staticmethod |
There was a problem hiding this comment.
Should we keep this as module level instead of making this a @staticmethod? The other helpers in this test suite all seem to be module level.
| @staticmethod | ||
| def _append_files(table: Table, rows: List[Dict[str, Any]]) -> None: |
Collect the filtered-overwrite rewrite results with a ray.wait loop that logs progress and raises a TimeoutError, cancelling the remaining tasks, when no task finishes within _REWRITE_STALL_TIMEOUT_S, instead of a bare ray.get that can block forever. Return early when every rewrite task comes back untouched. File metrics can admit a filter that no row matches, and committing the OVERWRITE snapshot in that case added an empty snapshot before the append. Move the overwrite test helpers to module level to match the rest of the suite, and cover both cases with tests. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
…write' into iceberg-overwrite-distributed-rewrite
…sks) (#66146) ## Description `Dataset.write_iceberg(mode=SaveMode.UPSERT, ...)` aggregated the join keys of every upserted row on the driver: each write task returned the key columns of every row it wrote, `on_write_complete` concatenated all of them into one table, and the driver then filtered NULL keys and ran a distinct over the result. Driver memory therefore grew with the size of the upsert -- ~185 MiB per million rows for a two-column key, 1.4 GiB at 8M rows -- and it was paid even when the coarse filter matched no file and no rewrite was needed at all. What the driver actually needs is only the per-column min/max used to build the coarse range filter. So: 1. The write tasks fold their block's keys into running per-column `(min, max)` bounds and return only those bounds. Rows with a NULL in any join column are left out of the bounds, because `NULL != NULL` means they can never match an existing row -- that is where the driver used to filter them. 2. The driver merges the bounds (min of mins, max of maxes) and builds the same coarse filter as before. When the bounds are empty -- every row had a NULL join key -- the commit short-circuits to a plain append, and when `plan_files()` then finds no candidate file the keys are never needed. 3. When there *are* candidate files, one task reads the join key columns back out of the data files this write just produced and deduplicates them; every rewrite task shares that single result. The driver passes the reference along without ever resolving it. On step 3: the keys are columns of the dataset that was just written, so they are already in the new data files and a projected read gets them back without the driver touching them. This is a task return value rather than a `ray.put`, deliberately: `ray.put` objects are owned by the worker that created them and are not lineage-reconstructable, so losing that worker after the write phase would fail the commit with `ObjectLostError`, whereas a task return value is owned by the submitter and Ray can rebuild it by re-running the task. Doing the read inside each rewrite task instead would turn one broadcast into (candidate files x new files) reads, hence the single collect task. The coarse filter that comes out is identical to before: `pc.min_max` skips nulls, and the distinct that used to run on the driver cannot move either bound. ## Related issues Related to #66145. This covers steps 1 and 2 of that issue plus the key-broadcast half of step 3. It deliberately does not change where the rewrite tasks build their anti-join, which is the remaining scaling limit discussed in [my design comment on the issue](#66145 (comment)) and is why the issue stays open -- see "What this does not fix" below. Not a duplicate: no issue or PR mentioned the UPSERT key aggregation before #66145, and no open PR touches `iceberg_datasink.py`'s upsert path (#62749 adds CREATE mode, #61753 adds checkpointing, #63337 is a Delta/write-abstraction stack). #66144 is mine and changes the OVERWRITE commit path in the same file; this PR is independent of it -- the two touch different functions, the branches merge cleanly and the merged tree passes the Iceberg tests, so they can land in either order and whichever is second just needs a rebase. ## Additional information ### Driver memory Measured with the script from #66145: the Iceberg table stays at 1000 rows and only the upsert dataset grows. The dataset is generated with `ray.data.range().map_batches()` so its blocks live in the object store; what the driver holds is the key aggregation under measurement. Peak driver RSS delta over the `write_iceberg` call, sampled every 20ms. Both arms run back to back on the same machine, `num_cpus=8`. `overlap` means the existing rows' ids fall inside the upsert range, so the coarse filter makes the existing file a rewrite candidate and the keys really are needed; `no overlap` is the pure-insert case where they are not. | upserted rows | before (no overlap) | after | before (overlap) | after | |---|---|---|---|---| | 1,000,000 | 191 MiB | 10 MiB | 207 MiB | 11 MiB | | 2,000,000 | 368 MiB | 10 MiB | 377 MiB | 11 MiB | | 4,000,000 | 786 MiB | 11 MiB | 793 MiB | 11 MiB | | 8,000,000 | 1,457 MiB | 12 MiB | 1,534 MiB | 12 MiB | Linear before, flat after. Row counts after the write are identical in every pair (1,001,000 / 2,001,000 / 4,001,000 / 8,001,000 for the insert case, 1M / 2M / 4M / 8M for the overlap case, where the upsert replaces the 1000 existing rows). Wall time is not what this change is about and is within noise of before (for example 8M: 5.9s -> 5.5s no overlap, 5.5s -> 6.6s overlap); the failure mode being removed is a driver OOM, not a slowdown. ### What this does not fix Memory per rewrite task is unchanged: each rewrite task still builds an anti-join table over the whole key set, ~19 MiB per million rows, so a 100M-row upsert would still ask about 1.9 GiB of each rewrite task. Fixing that needs the broadcast anti-join replaced by a distributed join, which is a design change with its own trade-offs (for a small upsert against a large table the broadcast is cheaper than a shuffle, so it likely wants a broadcast-or-shuffle choice rather than a replacement). That is step 3 of #66145 and I have asked there whether maintainers want it in the same effort. ### Tests Two cases added to `TestUpsertScanMerge` in `python/ray/data/tests/datasource/test_iceberg.py`: - `test_upsert_write_tasks_do_not_ship_keys_to_the_driver` -- wraps `on_write_complete` to inspect what the write tasks actually returned, and asserts none of them carried a key table, that the bounds cover the written keys, and that the resulting rows are correct. This is the regression test for the behaviour being changed. - `test_upsert_row_with_null_join_key_is_inserted` -- a row whose join key is NULL must be inserted rather than matched against anything. NULL filtering moved from the driver into the write tasks, so this guards that the move did not change the semantics; it passes before and after by design. Commands and results: ``` # with master's iceberg_datasink.py, new tests only pytest python/ray/data/tests/datasource/test_iceberg.py \ -k "upsert_write_tasks_do_not_ship_keys or upsert_row_with_null_join_key" -> 1 failed, 1 passed AssertionError: write tasks returned upsert key tables to the driver ([2] rows); the driver only needs the per-column bounds # with this change pytest python/ray/data/tests/datasource/test_iceberg.py \ -k "upsert_write_tasks_do_not_ship_keys or upsert_row_with_null_join_key" -> 2 passed pytest python/ray/data/tests/datasource/test_iceberg.py -> 85 passed pre-commit run --files python/ray/data/_internal/datasource/iceberg_datasink.py \ python/ray/data/tests/datasource/test_iceberg.py -> all hooks pass ``` Run on Python 3.12.11, macOS arm64, with the pinned `pyiceberg==0.11.0` / `pyarrow==23.0.1`. ### AI assistance AI assistance was used for this change: the investigation, the implementation, this description and the measurements above were produced with Claude Code, and every number quoted here comes from the runs listed. I have reviewed every changed line and run the tests locally. --------- Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com> Signed-off-by: Ayush Kumar <ayushk7102@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Ayush Kumar <ayushk7102@gmail.com> Co-authored-by: Isaac <no-reply@databricks.com>
| assert files_read_on_driver == 0, ( | ||
| "a filter that covers whole files must stay on the metadata-only path, " | ||
| f"but {files_read_on_driver} data file(s) were read" | ||
| ) |
There was a problem hiding this comment.
Nit: This test asserts that no file was read on the driver. Is it misleading to say that we are asserting that whole file deletes must stay on the metadata-only path?
ayushk7102
left a comment
There was a problem hiding this comment.
Approved with minor nit

Description
Dataset.write_iceberg(mode=SaveMode.OVERWRITE, overwrite_filter=...)handed the delete side toPyIceberg's
Transaction.delete(). For a filter that does not align with the partition spec thatcall is copy-on-write: it reads and rewrites every partially matched data file one at a time in the
calling process.
_commit_overwriteruns fromon_write_complete, which runs on the driver, so therow-level rewrite of a partial overwrite was serial single-process work on the driver -- with 8 data
files the driver read all 8 of them (800,000 rows) in order to write a 2-row dataset.
This keeps PyIceberg's two steps and moves only the rewrite:
update_snapshot().delete()withdelete_by_predicate. The producer commits nothing when no whole file is affected.rewrites_needed: one Ray task per partially covered file reads it, keeps the rows thefilter does not match and writes them back; the driver commits those deletes and appends as a
single OVERWRITE snapshot, which is the snapshot PyIceberg's own rewrite path produces.
_commit_upsert_scan_mergealready works this way for UPSERT; the OVERWRITE path was the one left onPyIceberg's in-process rewrite.
Two details worth calling out:
_expression_to_complementary_pyarrowapplied to thebound filter, so rows whose filter column is NULL -- which do not match the filter -- are kept. A
naive
~exprwould drop them.txn._scan), the way PyIceberg does between its own twosteps, so files the metadata step already dropped are not planned for a rewrite.
Related issues
Closes #66133.
Not a duplicate: no issue or PR mentioned
overwrite_filterbefore #66133, and no open PR touchesiceberg_datasink.py's overwrite path (#62749 adds CREATE mode, #61753 adds checkpointing). Theconflict-detection filter for PyIceberg 0.12's concurrency validation is deliberately out of scope
here and tracked in #66134.
Additional information
Behaviour is unchanged apart from where the rewrite runs
Measured through
ray.data.write_icebergagainst a local SQLite catalog: 8 data files x 100k rows,overwrite_filter=col("col_c") == 2matching 10% of the rows in every file, writing 2 rows. Readsare counted by wrapping
ArrowScan.to_tablein the driver process only; Ray write tasks run inseparate processes, so a driver count of 0 means the driver read nothing.
['overwrite', 'append']['overwrite', 'append'][720000, 720002][720000, 720002]A full overwrite (no filter) is also unchanged: 0 driver-side reads and
['delete', 'append'],because whole files are dropped from metadata.
Object storage
Measured against a real S3-backed Iceberg table (
us-west-2, 6 data files / 185 MiB, filter matching10% of the rows in every file,
num_cpus=8, driven from a laptop). Five rounds, alternating the twoversions and rolling the table back to the same snapshot before every run, so both arms see the same
data and comparable network conditions. Each run's driver-side read count confirms which version ran
(6 files before, 0 after):
Uplink throughput degraded during rounds 4 and 5, which is why those numbers are much larger; the
paired ratio stays at 1.27-1.31x in every round where both arms ran under similar conditions, and
round 4's 3.86x is an artefact of the baseline landing in the slow window. The median paired ratio,
1.30x, is the number I would quote.
The gain here is bounded by one laptop's uplink shared by the rewrite tasks; on a cluster the rewrite
spreads across nodes, while the previous behaviour stayed serial on the driver no matter how large
the cluster was. On a local NVMe filesystem the two are within noise of each other (16 files x 200k
rows: 2.06s before, 2.15s after), because there the per-file cost is smaller than task dispatch --
this change is about removing the driver bottleneck, not about local-disk throughput.
Tests
Four cases added to
TestOverwriteModeinpython/ray/data/tests/datasource/test_iceberg.py:test_write_overwrite_with_filter_rewrites_off_the_driver-- three partially matched files; assertsno data file is read in the driver process, plus the resulting rows.
test_write_overwrite_with_filter_skips_files_already_dropped-- a filter covering one file whollyand another partly; asserts exactly one rewrite task is dispatched.
test_write_overwrite_with_filter_keeps_whole_file_deletes_metadata_only-- a partition-alignedfilter stays on the metadata-only path.
test_write_overwrite_with_filter_preserves_null_rows-- NULL rows in the filter column survive.Commands and results:
Run on Python 3.12.11, macOS arm64, with the pinned
pyiceberg==0.11.0/pyarrow==23.0.1.One behaviour note
Keys in
overwrite_kwargsother thanbranchandcase_sensitiveused to be forwarded toTransaction.delete(); they are now logged as unsupported and ignored, matching the argument'sdocumented set (
case_sensitive,branch) and howupsert_kwargsalready behaves.AI assistance
AI assistance was used for this change: the investigation, the implementation, this description and
the measurements above were produced with Claude Code, and every number quoted here comes from the
runs listed. I have reviewed every changed line and run the tests locally.