Skip to content

[Data] Distribute the Iceberg filtered-overwrite rewrite across Ray tasks - #66144

Merged
goutamvenkat-anyscale merged 6 commits into
ray-project:masterfrom
moomindani:iceberg-overwrite-distributed-rewrite
Oct 15, 2026
Merged

goutamvenkat-anyscale merged 6 commits into
ray-project:masterfrom
moomindani:iceberg-overwrite-distributed-rewrite

Conversation

@moomindani

Copy link
Copy Markdown
Contributor

Description

Dataset.write_iceberg(mode=SaveMode.OVERWRITE, overwrite_filter=...) handed the delete side to
PyIceberg's Transaction.delete(). For a filter that does not align with the partition spec that
call is copy-on-write: it reads and rewrites every partially matched data file one at a time in the
calling process. _commit_overwrite runs from on_write_complete, which runs on the driver, so the
row-level rewrite of a partial overwrite was serial single-process work on the driver -- with 8 data
files the driver read all 8 of them (800,000 rows) in order to write a 2-row dataset.

This keeps PyIceberg's two steps and moves only the rewrite:

  1. metadata only: drop the files the filter covers entirely, via update_snapshot().delete() with
    delete_by_predicate. The producer commits nothing when no whole file is affected.
  2. when rewrites_needed: one Ray task per partially covered file reads it, keeps the rows the
    filter does not match and writes them back; the driver commits those deletes and appends as a
    single OVERWRITE snapshot, which is the snapshot PyIceberg's own rewrite path produces.

_commit_upsert_scan_merge already works this way for UPSERT; the OVERWRITE path was the one left on
PyIceberg's in-process rewrite.

Two details worth calling out:

  • Preserved rows are selected with PyIceberg's _expression_to_complementary_pyarrow applied to the
    bound filter, so rows whose filter column is NULL -- which do not match the filter -- are kept. A
    naive ~expr would drop them.
  • Step 2 scans through the transaction (txn._scan), the way PyIceberg does between its own two
    steps, so files the metadata step already dropped are not planned for a rewrite.

Related issues

Closes #66133.

Not a duplicate: no issue or PR mentioned overwrite_filter before #66133, and no open PR touches
iceberg_datasink.py's overwrite path (#62749 adds CREATE mode, #61753 adds checkpointing). The
conflict-detection filter for PyIceberg 0.12's concurrency validation is deliberately out of scope
here and tracked in #66134.

Additional information

Behaviour is unchanged apart from where the rewrite runs

Measured through ray.data.write_iceberg against a local SQLite catalog: 8 data files x 100k rows,
overwrite_filter=col("col_c") == 2 matching 10% of the rows in every file, writing 2 rows. Reads
are counted by wrapping ArrowScan.to_table in the driver process only; Ray write tasks run in
separate processes, so a driver count of 0 means the driver read nothing.

before after
data files read in the driver process 8 (800,000 rows) 0
snapshots added ['overwrite', 'append'] ['overwrite', 'append']
rows visible per new snapshot [720000, 720002] [720000, 720002]
rows after the write 720,002 720,002

A full overwrite (no filter) is also unchanged: 0 driver-side reads and ['delete', 'append'],
because whole files are dropped from metadata.

Object storage

Measured against a real S3-backed Iceberg table (us-west-2, 6 data files / 185 MiB, filter matching
10% of the rows in every file, num_cpus=8, driven from a laptop). Five rounds, alternating the two
versions and rolling the table back to the same snapshot before every run, so both arms see the same
data and comparable network conditions. Each run's driver-side read count confirms which version ran
(6 files before, 0 after):

round before (wall) after (wall) ratio
1 50.0s 39.1s 1.28x
2 49.0s 37.3s 1.31x
3 52.5s 41.3s 1.27x
4 137.4s 35.6s 3.86x
5 120.7s 93.1s 1.30x
median 52.5s 39.1s 1.30x

Uplink throughput degraded during rounds 4 and 5, which is why those numbers are much larger; the
paired ratio stays at 1.27-1.31x in every round where both arms ran under similar conditions, and
round 4's 3.86x is an artefact of the baseline landing in the slow window. The median paired ratio,
1.30x, is the number I would quote.

The gain here is bounded by one laptop's uplink shared by the rewrite tasks; on a cluster the rewrite
spreads across nodes, while the previous behaviour stayed serial on the driver no matter how large
the cluster was. On a local NVMe filesystem the two are within noise of each other (16 files x 200k
rows: 2.06s before, 2.15s after), because there the per-file cost is smaller than task dispatch --
this change is about removing the driver bottleneck, not about local-disk throughput.

Tests

Four cases added to TestOverwriteMode in python/ray/data/tests/datasource/test_iceberg.py:

  • test_write_overwrite_with_filter_rewrites_off_the_driver -- three partially matched files; asserts
    no data file is read in the driver process, plus the resulting rows.
  • test_write_overwrite_with_filter_skips_files_already_dropped -- a filter covering one file wholly
    and another partly; asserts exactly one rewrite task is dispatched.
  • test_write_overwrite_with_filter_keeps_whole_file_deletes_metadata_only -- a partition-aligned
    filter stays on the metadata-only path.
  • test_write_overwrite_with_filter_preserves_null_rows -- NULL rows in the filter column survive.

Commands and results:

# before the change
pytest python/ray/data/tests/datasource/test_iceberg.py -k overwrite_with_filter
  -> 2 failed, 3 passed  (the two new behavioural tests fail: "the driver read 3 data file(s)")

# after the change
pytest python/ray/data/tests/datasource/test_iceberg.py -k overwrite_with_filter
  -> 5 passed
pytest python/ray/data/tests/datasource/test_iceberg.py
  -> 87 passed
pre-commit run --files python/ray/data/_internal/datasource/iceberg_datasink.py python/ray/data/tests/datasource/test_iceberg.py
  -> all hooks pass

Run on Python 3.12.11, macOS arm64, with the pinned pyiceberg==0.11.0 / pyarrow==23.0.1.

One behaviour note

Keys in overwrite_kwargs other than branch and case_sensitive used to be forwarded to
Transaction.delete(); they are now logged as unsupported and ignored, matching the argument's
documented set (case_sensitive, branch) and how upsert_kwargs already behaves.

AI assistance

AI assistance was used for this change: the investigation, the implementation, this description and
the measurements above were produced with Claude Code, and every number quoted here comes from the
runs listed. I have reviewed every changed line and run the tests locally.

…asks

`write_iceberg(mode=OVERWRITE, overwrite_filter=...)` handed the delete side to
PyIceberg's `Transaction.delete()`, whose copy-on-write path reads and rewrites
every partially matched data file one at a time in the calling process. That call
happens in `on_write_complete`, which runs on the driver, so the row-level rewrite
of a partial overwrite was serial single-process work on the driver: with 8 data
files the driver read all 8 of them (800k rows) to write a 2-row dataset.

Keep PyIceberg's two steps -- drop the files the filter covers entirely using
manifest metadata, then rewrite the ones it covers only partially -- but dispatch
one Ray task per file for the rewrite, the way `_commit_upsert_scan_merge` already
does for UPSERT. Preserved rows are selected with PyIceberg's complement of the
bound filter, so rows whose filter column is NULL are kept.

Snapshot semantics are unchanged: the same `['overwrite', 'append']` (or
`['delete', 'append']` when only whole files are dropped) pair is committed, and
the files the metadata step already dropped are not planned for a rewrite.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
@moomindani
moomindani requested a review from a team as a code owner September 13, 2026 07:57

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements distributed row-level overwrites for Iceberg tables by executing partial file rewrites inside Ray tasks instead of on the driver. It introduces _rewrite_iceberg_file_by_filter to handle the streaming rewrite of individual files and updates _commit_overwrite to perform metadata-only deletes for fully covered files before dispatching Ray tasks for partially covered ones. Feedback suggests optimizing the file rewrite logic by collecting filtered Arrow tables in a list and performing a single pa.concat_tables call at the end of the loop to avoid incremental copy overhead.

Comment on lines +83 to +112
preserved_rows: Optional["pa.Table"] = None
total_in_rows = 0
total_preserved_rows = 0

for rb in record_batches:
batch_table = pa.Table.from_batches([rb])
if len(batch_table) == 0:
continue
total_in_rows += len(batch_table)

kept = batch_table.filter(preserve_row_filter)
if len(kept) > 0:
if preserved_rows is None:
preserved_rows = kept
else:
preserved_rows = pa.concat_tables(
[preserved_rows, kept], promote_options="permissive"
)
total_preserved_rows += len(kept)

if total_in_rows == 0:
return (None, [])

if total_preserved_rows == 0:
# Every row matches the filter, so the whole file goes away.
return (file_scan_task.file, [])

if total_preserved_rows == total_in_rows:
# The metrics said the file might match, but no row actually does.
return (None, [])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Repeatedly calling pa.concat_tables in a loop can be highly inefficient because it performs incremental copies and metadata updates for each batch. Instead, collect the filtered tables in a list and perform a single concatenation at the end of the loop, right before writing the data files. This also avoids any concatenation overhead if the function returns early.

    preserved_batches: List["pa.Table"] = []
    total_in_rows = 0
    total_preserved_rows = 0

    for rb in record_batches:
        batch_table = pa.Table.from_batches([rb])
        if len(batch_table) == 0:
            continue
        total_in_rows += len(batch_table)

        kept = batch_table.filter(preserve_row_filter)
        if len(kept) > 0:
            preserved_batches.append(kept)
            total_preserved_rows += len(kept)

    if total_in_rows == 0:
        return (None, [])

    if total_preserved_rows == 0:
        # Every row matches the filter, so the whole file goes away.
        return (file_scan_task.file, [])

    if total_preserved_rows == total_in_rows:
        # The metrics said the file might match, but no row actually does.
        return (None, [])

    preserved_rows = pa.concat_tables(
        preserved_batches, promote_options="permissive"
    )

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 328bbea, and to the sibling _rewrite_iceberg_file used by UPSERT since it had the same pattern.

For the record on the magnitude: with a uniform schema pa.concat_tables appends chunks without copying buffers, so what the repeated call costs is chunk-list bookkeeping rather than data copies. Measured on this machine: 1000 batches of 1000 rows took 12.6ms incrementally versus 1.0ms as a single concatenation, and 50 batches of 20,000 rows took 0.2ms versus 0.0ms. Worth doing, and the code is shorter, but it is not where the time in this path goes.

Both rewrite tasks accumulated preserved rows by calling `pa.concat_tables` on
every RecordBatch, so each batch walked the chunk list built up so far. Collect the
batches in a list and concatenate once, after the early returns, so a file that is
deleted or left untouched does no concatenation at all.

This is a small win: `concat_tables` on a uniform schema appends chunks without
copying buffers, so the cost is chunk-list bookkeeping, measured at 12.6ms versus
1.0ms for 1000 batches of 1000 rows, and 0.2ms versus 0.0ms for 50 batches of
20,000 rows. The shorter code is the main benefit.

Applied to the new `_rewrite_iceberg_file_by_filter` and, since it is the same
pattern in the sibling function, to `_rewrite_iceberg_file` used by UPSERT.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Sep 30, 2026
@richardliaw richardliaw removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Sep 30, 2026
@ayushk7102

Copy link
Copy Markdown
Contributor

Thank you for your contribution! I am reviewing the changes for this PR and #66146 Will get back on this by the end of tomorrow!

@moomindani

Copy link
Copy Markdown
Contributor Author

@ayushk7102 @AarryaSaraf, gentle ping for review when you have a chance. The branch still merges cleanly onto current master, and CI was green on the last run. Happy to rebase or split anything if that makes it easier to review.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit b100836. Configure here.

Comment on lines +1005 to +1006
results = ray.get(refs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using a loop with ray.wait() for the done/pending tasks, so that we can have observability into the rewrite progress and catch timeouts.

A bare ray.get(refs) without any timeout blocks forever if a rewrite task stalls.

Suggested change
results = ray.get(refs)
results = []
pending = list(refs)
try:
while pending:
done, pending = ray.wait(
pending, num_returns=len(pending),
timeout=_REWRITE_STALL_TIMEOUT_S
)
if not done:
raise TimeoutError(
f"No Iceberg overwrite rewrite task finished in "
f"{_REWRITE_STALL_TIMEOUT_S}s;
{len(pending)}/{len(refs)} "
"still pending."
)
results.extend(ray.get(done))
logger.debug(
"[overwrite] rewrite progress: %d/%d file(s) done
(%.1fs elapsed)",
len(results),
len(refs),
time.perf_counter() - t0,
)
except BaseException:
for ref in pending:
ray.cancel(ref)
raise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can modify the existing UPSERT rewrite loop so that we do the same thing there too. I noticed we are currently not catching a timeout and rather retrying infinitely.

To protect scope let's keep that out of this PR, and I can take this as a follow up!

n_partial,
n_untouched,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we early return here?

        if n_whole_delete == 0 and n_partial == 0:
            return

)
assert rows_same(result, expected)

@staticmethod

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep this as module level instead of making this a @staticmethod? The other helpers in this test suite all seem to be module level.

Comment on lines +1855 to +1856
@staticmethod
def _append_files(table: Table, rows: List[Dict[str, Any]]) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this helper

moomindani and others added 3 commits October 7, 2026 18:51
Collect the filtered-overwrite rewrite results with a ray.wait loop that
logs progress and raises a TimeoutError, cancelling the remaining tasks,
when no task finishes within _REWRITE_STALL_TIMEOUT_S, instead of a bare
ray.get that can block forever.

Return early when every rewrite task comes back untouched. File metrics
can admit a filter that no row matches, and committing the OVERWRITE
snapshot in that case added an empty snapshot before the append.

Move the overwrite test helpers to module level to match the rest of
the suite, and cover both cases with tests.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
…write' into iceberg-overwrite-distributed-rewrite
richardliaw pushed a commit that referenced this pull request Oct 14, 2026
…sks) (#66146)

## Description

`Dataset.write_iceberg(mode=SaveMode.UPSERT, ...)` aggregated the join
keys of every upserted row on
the driver: each write task returned the key columns of every row it
wrote, `on_write_complete`
concatenated all of them into one table, and the driver then filtered
NULL keys and ran a distinct
over the result. Driver memory therefore grew with the size of the
upsert -- ~185 MiB per million rows
for a two-column key, 1.4 GiB at 8M rows -- and it was paid even when
the coarse filter matched no
file and no rewrite was needed at all.

What the driver actually needs is only the per-column min/max used to
build the coarse range filter.
So:

1. The write tasks fold their block's keys into running per-column
`(min, max)` bounds and return only
those bounds. Rows with a NULL in any join column are left out of the
bounds, because `NULL != NULL`
means they can never match an existing row -- that is where the driver
used to filter them.
2. The driver merges the bounds (min of mins, max of maxes) and builds
the same coarse filter as
before. When the bounds are empty -- every row had a NULL join key --
the commit short-circuits to
a plain append, and when `plan_files()` then finds no candidate file the
keys are never needed.
3. When there *are* candidate files, one task reads the join key columns
back out of the data files
this write just produced and deduplicates them; every rewrite task
shares that single result. The
   driver passes the reference along without ever resolving it.

On step 3: the keys are columns of the dataset that was just written, so
they are already in the new
data files and a projected read gets them back without the driver
touching them. This is a task return
value rather than a `ray.put`, deliberately: `ray.put` objects are owned
by the worker that created
them and are not lineage-reconstructable, so losing that worker after
the write phase would fail the
commit with `ObjectLostError`, whereas a task return value is owned by
the submitter and Ray can
rebuild it by re-running the task. Doing the read inside each rewrite
task instead would turn one
broadcast into (candidate files x new files) reads, hence the single
collect task.

The coarse filter that comes out is identical to before: `pc.min_max`
skips nulls, and the distinct
that used to run on the driver cannot move either bound.

## Related issues

Related to #66145.

This covers steps 1 and 2 of that issue plus the key-broadcast half of
step 3. It deliberately does
not change where the rewrite tasks build their anti-join, which is the
remaining scaling limit
discussed in [my design comment on the
issue](#66145 (comment))
and is why the issue stays open -- see "What this does not fix" below.

Not a duplicate: no issue or PR mentioned the UPSERT key aggregation
before #66145, and no open PR
touches `iceberg_datasink.py`'s upsert path (#62749 adds CREATE mode,
#61753 adds checkpointing,
#63337 is a Delta/write-abstraction stack). #66144 is mine and changes
the OVERWRITE commit path in
the same file; this PR is independent of it -- the two touch different
functions, the branches merge
cleanly and the merged tree passes the Iceberg tests, so they can land
in either order and whichever
is second just needs a rebase.

## Additional information

### Driver memory

Measured with the script from #66145: the Iceberg table stays at 1000
rows and only the upsert dataset
grows. The dataset is generated with `ray.data.range().map_batches()` so
its blocks live in the object
store; what the driver holds is the key aggregation under measurement.
Peak driver RSS delta over the
`write_iceberg` call, sampled every 20ms. Both arms run back to back on
the same machine, `num_cpus=8`.

`overlap` means the existing rows' ids fall inside the upsert range, so
the coarse filter makes the
existing file a rewrite candidate and the keys really are needed; `no
overlap` is the pure-insert case
where they are not.

| upserted rows | before (no overlap) | after | before (overlap) | after
|
|---|---|---|---|---|
| 1,000,000 | 191 MiB | 10 MiB | 207 MiB | 11 MiB |
| 2,000,000 | 368 MiB | 10 MiB | 377 MiB | 11 MiB |
| 4,000,000 | 786 MiB | 11 MiB | 793 MiB | 11 MiB |
| 8,000,000 | 1,457 MiB | 12 MiB | 1,534 MiB | 12 MiB |

Linear before, flat after. Row counts after the write are identical in
every pair (1,001,000 /
2,001,000 / 4,001,000 / 8,001,000 for the insert case, 1M / 2M / 4M / 8M
for the overlap case, where
the upsert replaces the 1000 existing rows). Wall time is not what this
change is about and is within
noise of before (for example 8M: 5.9s -> 5.5s no overlap, 5.5s -> 6.6s
overlap); the failure mode
being removed is a driver OOM, not a slowdown.

### What this does not fix

Memory per rewrite task is unchanged: each rewrite task still builds an
anti-join table over the whole
key set, ~19 MiB per million rows, so a 100M-row upsert would still ask
about 1.9 GiB of each rewrite
task. Fixing that needs the broadcast anti-join replaced by a
distributed join, which is a design
change with its own trade-offs (for a small upsert against a large table
the broadcast is cheaper than
a shuffle, so it likely wants a broadcast-or-shuffle choice rather than
a replacement). That is step 3
of #66145 and I have asked there whether maintainers want it in the same
effort.

### Tests

Two cases added to `TestUpsertScanMerge` in
`python/ray/data/tests/datasource/test_iceberg.py`:

- `test_upsert_write_tasks_do_not_ship_keys_to_the_driver` -- wraps
`on_write_complete` to inspect what
the write tasks actually returned, and asserts none of them carried a
key table, that the bounds
cover the written keys, and that the resulting rows are correct. This is
the regression test for the
  behaviour being changed.
- `test_upsert_row_with_null_join_key_is_inserted` -- a row whose join
key is NULL must be inserted
rather than matched against anything. NULL filtering moved from the
driver into the write tasks, so
this guards that the move did not change the semantics; it passes before
and after by design.

Commands and results:

```
# with master's iceberg_datasink.py, new tests only
pytest python/ray/data/tests/datasource/test_iceberg.py \
  -k "upsert_write_tasks_do_not_ship_keys or upsert_row_with_null_join_key"
  -> 1 failed, 1 passed
     AssertionError: write tasks returned upsert key tables to the driver ([2] rows);
     the driver only needs the per-column bounds

# with this change
pytest python/ray/data/tests/datasource/test_iceberg.py \
  -k "upsert_write_tasks_do_not_ship_keys or upsert_row_with_null_join_key"
  -> 2 passed
pytest python/ray/data/tests/datasource/test_iceberg.py
  -> 85 passed
pre-commit run --files python/ray/data/_internal/datasource/iceberg_datasink.py \
  python/ray/data/tests/datasource/test_iceberg.py
  -> all hooks pass
```

Run on Python 3.12.11, macOS arm64, with the pinned `pyiceberg==0.11.0`
/ `pyarrow==23.0.1`.

### AI assistance

AI assistance was used for this change: the investigation, the
implementation, this description and the
measurements above were produced with Claude Code, and every number
quoted here comes from the runs
listed. I have reviewed every changed line and run the tests locally.

---------

Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
Signed-off-by: Ayush Kumar <ayushk7102@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ayush Kumar <ayushk7102@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Comment on lines +1947 to +1950
assert files_read_on_driver == 0, (
"a filter that covers whole files must stay on the metadata-only path, "
f"but {files_read_on_driver} data file(s) were read"
)

@ayushk7102 ayushk7102 Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This test asserts that no file was read on the driver. Is it misleading to say that we are asserting that whole file deletes must stay on the metadata-only path?

@ayushk7102 ayushk7102 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with minor nit

@goutamvenkat-anyscale goutamvenkat-anyscale added the go add ONLY when ready to merge, run all tests label Oct 14, 2026
@goutamvenkat-anyscale
goutamvenkat-anyscale merged commit b575a6d into ray-project:master Oct 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Data] write_iceberg OVERWRITE with an overwrite_filter rewrites every candidate file on the driver

5 participants