Repository navigation
[Data] Cut Iceberg UPSERT driver memory from O(upserted rows) to O(tasks) - #66146
Conversation
…sks) Every write task returned the join key columns of every row it wrote, the driver concatenated all of them, filtered NULL keys and ran a distinct over the result. Driver memory therefore grew with the size of the upsert: measured at ~185 MiB per million rows with a two-column key, 1.4 GiB for 8M rows, and paid even when the coarse filter matched no file and no rewrite was needed. What the driver actually needs is the per-column min/max used to build the coarse range filter, so the write tasks now fold their block's keys into running bounds (excluding rows with a NULL join key, which cannot match anything) and return only those bounds. When the filter then plans no candidate file, the keys are never needed at all. When there are candidate files, a single task reads the join key columns back out of the data files this write just produced, deduplicates them, and every rewrite task shares that one result; the driver passes the reference along without resolving it. The keys come from a task return value rather than a ray.put, so Ray can rebuild them by re-running the task if the copy is lost. Peak driver RSS during the write, measured with the script in ray-project#66145: rows no overlap overlap before after before after 1,000,000 191MiB 10MiB 207MiB 11MiB 2,000,000 368MiB 10MiB 377MiB 11MiB 4,000,000 786MiB 11MiB 793MiB 11MiB 8,000,000 1,457MiB 12MiB 1,534MiB 12MiB Memory per rewrite task is unchanged: each still builds an anti-join table over the whole key set. That is the remaining scaling limit and needs a distributed join, tracked in ray-project#66145. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
There was a problem hiding this comment.
Code Review
This pull request optimizes the Iceberg UPSERT path by aggregating join key bounds on the write tasks and sending only these bounds to the driver, preventing driver memory from scaling with the upsert size. Tests are also added to verify this behavior and the handling of NULL join keys. Feedback suggests parallelizing Parquet file reads during key collection using a ThreadPoolExecutor to reduce network latency, and strengthening test assertions to verify that the returned key bounds and row counts are correctly populated.
| tables = [] | ||
| for path in data_file_paths: | ||
| with io.new_input(path).open() as input_stream: | ||
| tables.append(pq.read_table(input_stream, columns=list(upsert_cols))) |
There was a problem hiding this comment.
Reading the Parquet files sequentially can be slow due to network latency when fetching from cloud storage (e.g., S3, GCS). We can parallelize the file reads using a ThreadPoolExecutor to significantly speed up the key collection phase.
from concurrent.futures import ThreadPoolExecutor
def read_table(path):
with io.new_input(path).open() as input_stream:
return pq.read_table(input_stream, columns=list(upsert_cols))
with ThreadPoolExecutor() as executor:
tables = list(executor.map(read_table, data_file_paths))There was a problem hiding this comment.
Good suggestion, done in 012f1df -- though through PyIceberg's shared executor
(ExecutorFactory.get_or_create(), the same one ArrowScan uses) rather than a fresh
ThreadPoolExecutor, so the pool is shared with the rest of PyIceberg's reads in that worker and
PYICEBERG_MAX_WORKERS applies as a knob.
Measured against S3 (us-west-2, 8 data files / 2.22M rows, reading the two key columns, three rounds
alternating sequential and parallel):
| round | sequential | parallel |
|---|---|---|
| 1 | 6.67s | 2.66s |
| 2 | 7.50s | 1.61s |
| 3 | 4.88s | 1.38s |
Peak memory in the task is unchanged: every table was already held until the concat at the end.
| assert captured, "no write results reached the driver" | ||
| with_keys = [r for r in captured if getattr(r, "upsert_keys", None) is not None] |
There was a problem hiding this comment.
The test currently only asserts that upsert_keys is not present, but it does not verify that the new upsert_key_bounds, upsert_key_rows, and upsert_null_key_rows attributes are correctly populated with the expected values. Adding these assertions will make the test much more robust.
assert captured, "no write results reached the driver"
for r in captured:
assert r.upsert_key_bounds == {"col_a": (2, 4)}
assert r.upsert_key_rows == 2
assert r.upsert_null_key_rows == 0
with_keys = [r for r in captured if getattr(r, "upsert_keys", None) is not None]There was a problem hiding this comment.
Agreed -- the test only checked that no key table came back, not that the bounds returned instead were
right. Added in 012f1df: the merged bounds, the row count and the NULL count.
I aggregated across the captured write results rather than asserting per result, because Ray is free to
split even this two-row write across tasks, and a per-task upsert_key_bounds == {"col_a": (2, 4)}
would then fail on a task that only saw col_a = 2:
merged = {}
for write_return in captured:
for name, (col_min, col_max) in (write_return.upsert_key_bounds or {}).items():
previous = merged.get(name)
merged[name] = (
(min(previous[0], col_min), max(previous[1], col_max))
if previous
else (col_min, col_max)
)
assert merged == {"col_a": (2, 4)}, merged
assert sum(r.upsert_key_rows for r in captured) == 2
assert sum(r.upsert_null_key_rows for r in captured) == 0… test Review feedback on ray-project#66146. _collect_upsert_keys read the new data files one at a time, so on object storage the per-file latency was serial. The reads now go through PyIceberg's shared executor, the one its own scans use, so they overlap and PYICEBERG_MAX_WORKERS applies. Measured against S3 (us-west-2, 8 data files, 2.22M rows, reading the two key columns, three rounds alternating): 6.67s / 7.50s / 4.88s sequential against 2.66s / 1.61s / 1.38s parallel. The test asserted only that no write result carried a key table, not that the bounds it returned instead were right. It now also checks the bounds, the row count and the NULL count. The check aggregates across the captured write results rather than asserting per result, because Ray is free to split even a two-row write across tasks. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
|
@ayushk7102, gentle ping for review when you have a chance. The branch still merges cleanly onto current master, and CI was green on the last run. It is independent of #66144, so the two can be reviewed in either order. |
ayushk7102
left a comment
There was a problem hiding this comment.
Thanks for your contribution @moomindani! This will be a good improvement moving memory intensive work off the driver. A couple of comments, mainly about the implementation details. Also, I'm going to see if we can bump up the pyiceberg version to see if it makes the implementation easier.
|
hi @moomindani any updates here? |
_collect_upsert_keys now takes the DataFiles this write produced and reads the join columns with ArrowScan, the reader the rewrite tasks already use, instead of reading Parquet paths on PyIceberg's internal ExecutorFactory. ArrowScan needs a spec_id that DataFile.from_args drops on PyIceberg 0.11, so restore the default spec when it is missing. Reserve memory for the key task from the on-disk sizes of the join columns, falling back to the whole file when a file has no column metrics, so it does not oversubscribe a worker. Drop the docstring claim that Ray rebuilds the keys through lineage, which stops holding once Ray Data owns fault tolerance, and apply the diagram and test-name suggestions from review. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Noritaka Sekiyama <noritaka.sekiyama@databricks.com>
|
tests failing still? |
Signed-off-by: Ayush Kumar <ayushk7102@gmail.com>
There was a problem hiding this comment.
Thank you for the contribution, @moomindani. LGTM
…Value] for key_bounds in iceberg upsert Signed-off-by: Ayush Kumar <ayushk7102@gmail.com>
| import functools | ||
|
|
||
| import pyarrow.compute as pc | ||
|
|
||
| masks = (pc.is_valid(keys_table[col]) for col in upsert_cols) | ||
| kept = keys_table.filter(functools.reduce(pc.and_, masks)) | ||
| dropped = len(keys_table) - len(kept) | ||
| if len(kept) == 0: | ||
| return 0, dropped | ||
|
|
||
| for col in upsert_cols: | ||
| min_max = pc.min_max(kept[col]) | ||
| col_min, col_max = min_max["min"].as_py(), min_max["max"].as_py() | ||
| if col_min is None: | ||
| continue | ||
| if col in bounds: | ||
| prev_min, prev_max = bounds[col] | ||
| bounds[col] = (min(prev_min, col_min), max(prev_max, col_max)) | ||
| else: | ||
| bounds[col] = (col_min, col_max) | ||
| return len(kept), dropped |
There was a problem hiding this comment.
def _merge_key_bounds(
bounds: Dict[str, Tuple["pa.Scalar", "pa.Scalar"]],
keys_table: "pa.Table",
upsert_cols: List[str],
) -> Tuple[int, int]:
import pyarrow.compute as pc
kept = keys_table.select(upsert_cols).drop_null() # any-null row filter, one kernel
dropped = len(keys_table) - len(kept)
if len(kept) == 0:
return 0, dropped
for col in upsert_cols:
mm = pc.min_max(kept[col]) # no nulls left, so never null here
col_min, col_max = mm["min"], mm["max"]
if col in bounds:
prev_min, prev_max = bounds[col]
bounds[col] = (
pc.min_element_wise(prev_min, col_min),
pc.max_element_wise(prev_max, col_max),
)
else:
bounds[col] = (col_min, col_max)
return len(kept), dropped
There was a problem hiding this comment.
Want to minimize the python conversion here. @moomindani
There was a problem hiding this comment.
Kept the .select(upsert_columns).drop_null(), but pc.min_element_wise on mm["min"] would return a pa.Scalar rather than a Python object (which pyiceberg's filter expressions need), else we error out with a TypeError. I tried deferring the .py() function to once per task like this
def _collect_key_bounds(acc, keys_table, upsert_cols): # per block
kept = keys_table.drop_null()
...
for col in upsert_cols:
mm = pc.min_max(kept[col])
mins, maxs = acc.setdefault(col, ([], []))
mins.append(mm["min"]); maxs.append(mm["max"])
def _finalize_key_bounds(acc): # once per task, before IcebergWriteResult
return {c: (pc.min(pa.array(mins)).as_py(), pc.max(pa.array(maxs)).as_py())
for c, (mins, maxs) in acc.items()}
But this version does not meaningfully improve performance (within noise). Also, using pc.max(), pc.min() results in worse performance due to the conversion back to python required
Description
Dataset.write_iceberg(mode=SaveMode.UPSERT, ...)aggregated the join keys of every upserted row onthe driver: each write task returned the key columns of every row it wrote,
on_write_completeconcatenated all of them into one table, and the driver then filtered NULL keys and ran a distinct
over the result. Driver memory therefore grew with the size of the upsert -- ~185 MiB per million rows
for a two-column key, 1.4 GiB at 8M rows -- and it was paid even when the coarse filter matched no
file and no rewrite was needed at all.
What the driver actually needs is only the per-column min/max used to build the coarse range filter.
So:
(min, max)bounds and return onlythose bounds. Rows with a NULL in any join column are left out of the bounds, because
NULL != NULLmeans they can never match an existing row -- that is where the driver used to filter them.
before. When the bounds are empty -- every row had a NULL join key -- the commit short-circuits to
a plain append, and when
plan_files()then finds no candidate file the keys are never needed.this write just produced and deduplicates them; every rewrite task shares that single result. The
driver passes the reference along without ever resolving it.
On step 3: the keys are columns of the dataset that was just written, so they are already in the new
data files and a projected read gets them back without the driver touching them. This is a task return
value rather than a
ray.put, deliberately:ray.putobjects are owned by the worker that createdthem and are not lineage-reconstructable, so losing that worker after the write phase would fail the
commit with
ObjectLostError, whereas a task return value is owned by the submitter and Ray canrebuild it by re-running the task. Doing the read inside each rewrite task instead would turn one
broadcast into (candidate files x new files) reads, hence the single collect task.
The coarse filter that comes out is identical to before:
pc.min_maxskips nulls, and the distinctthat used to run on the driver cannot move either bound.
Related issues
Related to #66145.
This covers steps 1 and 2 of that issue plus the key-broadcast half of step 3. It deliberately does
not change where the rewrite tasks build their anti-join, which is the remaining scaling limit
discussed in my design comment on the issue
and is why the issue stays open -- see "What this does not fix" below.
Not a duplicate: no issue or PR mentioned the UPSERT key aggregation before #66145, and no open PR
touches
iceberg_datasink.py's upsert path (#62749 adds CREATE mode, #61753 adds checkpointing,#63337 is a Delta/write-abstraction stack). #66144 is mine and changes the OVERWRITE commit path in
the same file; this PR is independent of it -- the two touch different functions, the branches merge
cleanly and the merged tree passes the Iceberg tests, so they can land in either order and whichever
is second just needs a rebase.
Additional information
Driver memory
Measured with the script from #66145: the Iceberg table stays at 1000 rows and only the upsert dataset
grows. The dataset is generated with
ray.data.range().map_batches()so its blocks live in the objectstore; what the driver holds is the key aggregation under measurement. Peak driver RSS delta over the
write_icebergcall, sampled every 20ms. Both arms run back to back on the same machine,num_cpus=8.overlapmeans the existing rows' ids fall inside the upsert range, so the coarse filter makes theexisting file a rewrite candidate and the keys really are needed;
no overlapis the pure-insert casewhere they are not.
Linear before, flat after. Row counts after the write are identical in every pair (1,001,000 /
2,001,000 / 4,001,000 / 8,001,000 for the insert case, 1M / 2M / 4M / 8M for the overlap case, where
the upsert replaces the 1000 existing rows). Wall time is not what this change is about and is within
noise of before (for example 8M: 5.9s -> 5.5s no overlap, 5.5s -> 6.6s overlap); the failure mode
being removed is a driver OOM, not a slowdown.
What this does not fix
Memory per rewrite task is unchanged: each rewrite task still builds an anti-join table over the whole
key set, ~19 MiB per million rows, so a 100M-row upsert would still ask about 1.9 GiB of each rewrite
task. Fixing that needs the broadcast anti-join replaced by a distributed join, which is a design
change with its own trade-offs (for a small upsert against a large table the broadcast is cheaper than
a shuffle, so it likely wants a broadcast-or-shuffle choice rather than a replacement). That is step 3
of #66145 and I have asked there whether maintainers want it in the same effort.
Tests
Two cases added to
TestUpsertScanMergeinpython/ray/data/tests/datasource/test_iceberg.py:test_upsert_write_tasks_do_not_ship_keys_to_the_driver-- wrapson_write_completeto inspect whatthe write tasks actually returned, and asserts none of them carried a key table, that the bounds
cover the written keys, and that the resulting rows are correct. This is the regression test for the
behaviour being changed.
test_upsert_row_with_null_join_key_is_inserted-- a row whose join key is NULL must be insertedrather than matched against anything. NULL filtering moved from the driver into the write tasks, so
this guards that the move did not change the semantics; it passes before and after by design.
Commands and results:
Run on Python 3.12.11, macOS arm64, with the pinned
pyiceberg==0.11.0/pyarrow==23.0.1.AI assistance
AI assistance was used for this change: the investigation, the implementation, this description and the
measurements above were produced with Claude Code, and every number quoted here comes from the runs
listed. I have reviewed every changed line and run the tests locally.