Repository navigation
[Data] read_iceberg on Datasource V2 — end-to-end preview (draft, do not merge) - #66182
AarryaSaraf wants to merge 5 commits into
Conversation
First of three PRs that port read_iceberg to Datasource V2. This one is the data contract the other two are written against, and nothing calls it yet: the encoder in [2b/6], the read path in [2c/6]. It is split out because it is the piece with the most detail per line and the least context needed to review it -- pure encode/decode over table metadata, no Ray operators involved. V2 splits a read into a ListFiles task that decides what to read and ReadFiles tasks that read it. For Iceberg that boundary is awkward: PyIceberg's unit of work is a FileScanTask, which holds a DataFile, its delete files and a residual expression, and neither a Table nor a FileIO can cross a task boundary (issue ray-project#49107 is the V1 version of the same problem). So the listing task encodes each FileScanTask into manifest columns and the read task decodes it back. Ten columns, all prefixed __ib_: - path and file size, which the base FileManifest already defines; - __ib_file_format and __ib_record_count, off the DataFile; - __ib_spec_id and __ib_partition, which together let the reader rebuild the partition tuple. The partition rides as a typed Arrow struct built from the union of every partition spec the table has ever had, so a table that evolved its spec still round-trips, and each field carries Iceberg's own internal representation (a date is days-since-epoch, a UUID is 16 bytes) rather than a string; - __ib_delete_paths and __ib_delete_formats, one list per data file; - __ib_residual_is_true, the one derived column: whether plan_files() found the file's partition values already satisfy the whole filter. Nothing consumes it in this stack until the metadata-only count() in [3/6], where it is exactly the condition that makes a row count exact under a filter. FileManifest allows extra columns (file_manifest.py), which is what makes this work without a new plumbing type: a manifest block stays a block, and the partitioner in [2b/6] can slice and concatenate it like any other. Tests: six, all round trips against the SQLite-catalog fixture -- every field the reader will use survives, delete files survive, rebuilt scan tasks decode to the same rows, and the partition struct is typed and lossless across an identity partition, an evolved spec and the unpartitioned case. Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
…berg Second of three. The four components that do the work, built against the manifest from [2a/6]; the datasource that wires them into read_iceberg, and the flag that turns any of it on, arrive in [2c/6]. Nothing in this PR changes behaviour -- no existing code path reaches these classes. Kept together because they are one round trip and each is small: the indexer produces manifest rows, the partitioner groups them, the scanner is what a task carries, and the reader turns a group back into blocks. The scanner also cannot be separated from the indexer -- the indexer calls combine_row_filter to translate a pushed Ray predicate, and there is only one such translator. listing/iceberg_file_indexer.py -- plan_files() inside the listing task, one manifest row per data file, emitted in blocks of RAY_DATA_ICEBERG_LISTING_BATCH_FILES files (default 1024) purely to amortize Arrow array construction. Listing does no data IO at all: an Iceberg manifest already records each file's path, size, row count and partition values, so unlike Parquet there is no footer to fetch. It ignores shuffle_config, since shuffling exists to spread contention over a directory listing and there is no directory listing here. partitioners/iceberg_file_partitioner.py -- groups those rows into read units against a byte budget (RAY_DATA_ICEBERG_BIN_PACKING_BYTES, default 128 MiB). requires_global_input = True, which is also what makes listing a single task. It only ever slices and concatenates the blocks the indexer handed it, so every __ib_ column survives to the reader; OnlineBinPacker cannot be reused because it rebuilds each bin from paths, sizes and chunk metadata alone, and for Iceberg the other columns are the contract. Bins are sized by estimate_decoded_size(), which is compressed bytes times RAY_DATA_ICEBERG_DECODE_RATIO (default 4). That multiplier is a deliberate placeholder: Iceberg metadata records compressed bytes and row counts, nothing anywhere records decoded size, and measured expansion spans 1x to 26x, so there is no honest constant. The seam matters more than the number -- packing is written against decoded bytes, which is the quantity that governs downstream memory, and the sharpener (column_sizes over value_counts, both spec-optional) can land later without moving it. scanners/iceberg_scanner.py -- frozen dataclass pickled to every task; filter, column and limit pushdown. combine_row_filter does the Ray-to-Iceberg translation and reports what it could not translate, so the residual is enforced above the read rather than dropped, which is where V1 raises instead. readers/iceberg_file_reader.py -- one ArrowScan per file, which is what makes positional deletes, field-ID schema resolution, type promotion and partition backfill correct without reimplementing any of them. One file at a time because ArrowScan materializes every batch of every task it is handed, so a whole read unit at once would be held at once. Each batch is cast to the projected schema, because to_record_batches types a batch from the file it came out of and two files can disagree on string vs large_string -- the same normalization V1 does. An empty projection needed a fix no design anticipated: PyIceberg returns zero-row batches for it, and zero-column Arrow tables lose their length on concat, so a count() over no columns would report 0 rows on a 101-row table. The reader decodes the narrowest column instead and swaps in the same __bsp_stub column the Parquet V2 reader uses. Tests: four, driving the components directly since nothing wires them up yet -- position deletes actually being applied (built by hand: PyIceberg 0.11.1 only deletes copy-on-write, so no table it writes exercises merge-on-read), the byte budget deciding how many read units there are, packing preserving every manifest column, and a pushed predicate pruning files during listing rather than only filtering rows. Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
Third of three, and the one that turns [2a/6] and [2b/6] into a working reader. read_iceberg goes through DataSourceV2 (ListFiles -> ReadFiles) when DataContext.use_iceberg_datasource_v2 is set -- default False, env RAY_DATA_USE_ICEBERG_DATASOURCE_V2. V1 is untouched: outside the tests this PR deletes no line of existing code. The flag is separate from use_datasource_v2 (already default True) so that Parquet's default-on state and Iceberg's opt-in state are independent knobs. iceberg_datasource_v2.py is the driver side. Two things it does that are worth review: - The schema comes from table metadata, not from a file. Iceberg records the schema in the table, so schema_to_pyarrow(scan.projection()) is exact and free; the V2 sampling path (open up to 16 files at plan time and infer) is declined via schema_needs_file_sample = False. - The snapshot is resolved once on the driver and latched. Listing now runs in a task at execution time, so without this two executions of one Dataset could resolve different "current" snapshots, and a task retry could straddle a commit. None is a legitimate latched value -- a table with no snapshot yet -- so "not resolved" is tracked by the latch, not by the id being None. parallelism and override_num_blocks warn and are ignored, since listing and packing decide task granularity. That is a deliberate difference from V1, which bin-packs files into exactly `parallelism` chunks. Tests. The eleven read-path tests in test_iceberg.py now run against both readers through the both_iceberg_readers fixture, so every assertion in them is a parity assertion; seven already existed and four are new. Two of the four branch on the reader instead of asserting the same thing, pinning a V1 limitation V2 does not have: a predicate Iceberg cannot express (V1 raises at read time, V2 filters above the read) and a declared-but-empty table (V1 reports no schema at all). The other two are new because V2 needs them and V1 never could fail them: that two executions of one Dataset see the same snapshot, and that count() with an empty projection is right. Three more cover the hooks the shared-seams PR [1/6] added but could not test there, since Iceberg is their first real implementation: that planning opens no file at all, not even to sample one (sample_files is patched to raise); that infer_schema and resolve_partitioning accept a None sample, which is the whole point of schema_needs_file_sample; and that a source which declines the metadata-only count() gets an ordinary read instead. Deliberately not optimized here, each its own follow-up in this stack: count() decodes one column per file rather than summing record_count from metadata; a limit does not stop the plan_files() walk early, only the per-unit read; every file goes through ArrowScan even when it has no delete files, an AlwaysTrue residual and no schema drift, so none of the native Parquet reader's benefits apply; and an oversized file becomes one oversized task, because an Iceberg scan task names a file and never a range within one. Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
The port in [2/6] made count() correct but not cheap: it decodes the narrowest column of every file, because an empty projection makes PyIceberg return zero-row batches. This makes it free for the common table, and answers three hooks [1/6] added. Iceberg writes each data file's record_count into its own manifests, so the whole answer is already in the listing task's output -- not even a footer has to be fetched, which the equivalent Parquet path still does per file. The manifest has carried __ib_record_count since [2/6] for exactly this, so nothing about the carrier changes. - IcebergFileReader implements SupportsMetadata. read_metadata reads __ib_record_count off the manifest. That count includes rows a delete file removes, so a file carrying delete files is counted by decoding it instead -- the same branch PyIceberg's own count() takes, and the only one that touches data. Falling back per file rather than declining outright means a merge-on-read table still gets the cheap path for every clean file in it. - IcebergScanner.metadata_row_count_is_exact() declines only a limit. A row filter is fine, and this is the case the hook was reworded in [1/6] to allow: Iceberg plans a residual per file -- the part of the filter the file's partition values do not already satisfy -- so where the residual is AlwaysTrue the recorded record_count IS the post-filter count. A filter on a partition column leaves every listed file in that state, which makes the common `WHERE partition = x` count free. A filter on a data column leaves none of them and every file is decoded, which is the same work the ordinary read path would have done. A limit has no such metadata: nothing says which file it stops at. Column pruning changes a block's width, not its height. - IcebergFileIndexer.as_whole_file_indexer() returns itself. The base default is None because an indexer may emit one manifest row per chunk of a file, which would over-count it: that is the footer-based Parquet indexer, where one file's row groups become several rows. An Iceberg scan task names a whole file and is never split, so one file is always one row here -- grouping those rows into read units happens later, in IcebergFilePartitioner. The other half of the default -- no per-file IO while listing -- holds too, since planning reads table metadata only. available_metadata() repeats the limit guard on the reader side, so a caller that reaches read_metadata without going through the rule still cannot be handed a partial count. It does not refuse a row filter, since read_metadata accounts for one per file. One shared-code fix comes with this, because answering the hook under a filter is what exposes it. PushdownCountFiles deletes the ReadFiles and puts a MapBatches in its place, and DeriveListFilesPushdown clears a ListFiles' constraints for any consumer that is not a ReadFiles -- so the listing ran unfiltered while the count trusted per-file residuals computed as if it had been filtered, and an unfiltered count came back. Parquet never saw it: its scanner declines the hook under any predicate. The rule now pins the predicate onto the listing it rebuilds and marks it ListFiles.pushdown_is_final, which the derive rule leaves alone. That is sound rather than an exemption -- count_rows applies the filter through the same reader the deleted ReadFiles would have used, so the listing prunes by exactly what the consumer applies. Left to [4/6]: the snapshot summary's total-records, which answers an unfiltered whole-table count() on the driver with no tasks at all. Tests: seven Iceberg ones, none with a V1 counterpart, plus one unit test for the derive rule's new opt-out. The optimized count plan is ListFiles -> MapBatches(count_rows) with no ReadFiles and the Iceberg indexer surviving the rule's rebuild. Three filters keep the rewrite and still count correctly -- on the partition column, on a data column, and on a data column the count projection does not include, which is the one that would break an implementation assuming the filter binds only to projected columns. Each also asserts the mechanism rather than just the answer: whether every listed file came back with an AlwaysTrue residual is exactly whether the count was free, and the rebuilt listing carries the predicate. Those three tests compared take_all() against a count() called after it, which a materialized Dataset answers from the block metadata it just produced rather than by re-optimizing -- so they passed against the unfiltered count above. They count first now, and fail against the unfixed rule. A limit declines. read_metadata subtracts hand-built position deletes while the manifest still carries the pre-delete number, and available_metadata declines a limited reader but not a filtered one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
[3/6] made an unfiltered count() free of data reads but not free of
tasks: it still lists the table and sums per-file record_counts. Iceberg
already published that sum. Every snapshot carries a summary, and
total-records in it counts the rows of every data file in the snapshot
-- and the summary sits in table metadata this read has already loaded,
for the schema. So the whole answer is a dict lookup on the driver, and
count() returns without submitting anything.
This also restores parity with V1, which has always answered this
count() from Read.infer_metadata; the V2 port lost it because
ReadFiles.infer_metadata reports nothing.
- Scanner.exact_row_count() is a new hook, defaulting to None ("not
known"). It reports the rows the scan returns *after* every pushdown
the scanner holds, so it must decline whenever one of them can reduce
the count -- the same fail-closed reasoning as
metadata_row_count_is_exact, since the number is handed to the user as
the answer and not as an estimate.
- ReadFiles.infer_metadata() reports it as num_rows, which is what
reaches Dataset.count() -- through _meta_count, before the plan
count() would otherwise build. Everything else in that BlockMetadata
stays None as before, and a scanner that answers None leaves the
operator exactly as it was, so Parquet is unaffected.
- IcebergDatasourceV2._snapshot_row_count() reads the pinned snapshot's
summary, not the current one's, so a concurrent commit cannot make the
count disagree with the files the tasks would read. Every summary
field is optional by spec and writer-populated, so a missing key means
decline rather than assume. total-records counts rows as the data
files hold them, which is the table's row count only if no delete file
removes any -- hence total-position-deletes and total-equality-deletes,
which must both be present and zero. A table with no snapshot at all
has no data files, which is a count of exactly zero.
- IcebergScanner.exact_row_count() declines a limit and any filter.
Unlike metadata_row_count_is_exact, this cannot survive a filter even
one Iceberg fully satisfies per file: the summary describes the whole
snapshot and is blind to which files a filter leaves. row_filter is
the single gate because it is where every filter lands --
read_iceberg(row_filter=...) sets it directly, and push_filters only
records a predicate in the same step that ANDs it into row_filter.
Column pruning changes a block's width, not its height.
Declining costs a slower count(), not a wrong one: [3/6]'s per-file
counts are the fallback, and reading the data is the fallback to that.
So a filtered count() is answered by the rule as before.
Tests: five. The count comes back without a plan, and survives column
pruning. Both filter spellings decline, and they arrive by different
routes -- filter() through push_filters, read_iceberg(row_filter=...)
baked in before the optimizer sees it -- and both still count correctly
via the fallback. A pushed limit makes the scanner decline. Each way the
summary can fail to answer is exercised by mutating the fixture table's
own summary: total-records absent, either delete total absent, either
delete total non-zero, and a non-numeric value. A table with no snapshot
counts zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarrya <aarrya.saraf@anyscale.com>
| remaining = None if self._limit is None else self._limit - rows_read | ||
| if remaining is not None and remaining <= 0: | ||
| return | ||
| for batch in self._scan(remaining).to_record_batches([task]): |
There was a problem hiding this comment.
This carries over the delete-file behaviour in #66147. to_record_batches deduplicates delete files only
within the tasks it is handed, so one task per call means a delete file shared by k data files in this
input split is fetched k times; _read_deletes has no cache, and the bin-packing partitioner does not
group by delete file.
Measured on your branch rather than inferred: one input split with 8 data files sharing one positional
delete file opens that delete file 8 times, and each data file once. On the existing datasource, which has
the same shape, the cost at a realistic size was 24 data files sharing one 19.7 MiB delete file read from
S3 -- 473 MiB fetched instead of 19.7 MiB, and 74-96s of a 94-113s read spent inside _read_deletes.
#66148 fixes it by wrapping the FileIO so a shared file is fetched once. That keeps one data file per
scan call, so the memory bound in your docstring above is unaffected, and it ports here as a call at the
top of read() since you already keep self._io.
Worth noting for Iceberg format version 3: with deletion vectors each data file gets its own DataFile
entry into one shared Puffin file, and PuffinFile parses every blob on each read, so PyIceberg's
DataFile-level deduplication cannot collapse them even if a call were handed every task. Deduplicating
by path does.
| """Rebuild one PyIceberg scan task per manifest row. | ||
|
|
||
| The rebuilt ``DataFile``\\ s carry only the fields the read path reads; a | ||
| delete file's ``record_count`` and ``file_size_in_bytes`` are 0 because |
There was a problem hiding this comment.
file_size_in_bytes on a delete file does get consulted if the cache in #66148 is ported to this reader:
admission is decided on the declared size, so a file too large for the budget is never wrapped. An earlier
version of that PR wrapped it anyway, read the whole file on every reference and kept none of it; Bugbot
caught that.
Confirmed on your branch: after manifest_from_scan_tasks / scan_tasks_from_manifest, the delete file's
file_size_in_bytes comes back as 0, so nothing would ever be admitted and the cache would be inert.
Could the manifest carry a __ib_delete_sizes list column next to __ib_delete_paths and
__ib_delete_formats? It is one more list_(int64) from f.file_size_in_bytes, and it is the only thing
the port needs from the listing side.
There was a problem hiding this comment.
You are right. Please add __ib_delete_sizes in your PR once this is merged.
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
|
This pull request has been automatically closed because there has been no more activity in the 14 days Please feel free to reopen or open a new pull request if you'd still like this to be addressed. Again, you can always ask for help on our discussion forum or Ray's public slack channel. Thanks again for your contribution! |
…, row_id (#66654) ## Description Part 1 of 3 of the `read_mcap` move to Datasource V2, based on master. #67166 is stacked on it, and #67167 on #67166. The rest of the `read_mcap` stack (#67189 onward) sits on top. **Problem.** One task reads one whole file: - Parallelism is capped by the file count: ten 30 GB recordings are ten tasks. - Every file is opened, even when its index shows it holds nothing selected. - Rows have no stable id, so a checkpointed pipeline cannot resume. **Change.** `read_mcap` moves to Datasource V2, on by default as for `read_parquet` (`DataContext.use_datasource_v2`). The legacy datasource stays as the fallback. ``` files │ listing: read each file's index (a few ranged reads) │ one row per chunk, weighted by the memory reading it takes │ drop chunks that cannot match topics / time_range ▼ chunk rows │ packing: about 128 MiB per read task │ one big file ──► many tasks; small files ──► one task │ with preserve_order: one task per file, in listing order ▼ read tasks: seek to their chunks, merge them by log time ``` - New code lives in `datasource_v2/formats/mcap/`. The V2 interfaces do not change. - Columns and types match the legacy path, with two changes. `schema_data` is dictionary-encoded. A new `channel_metadata` column holds each channel's metadata map as JSON text, dictionary-encoded, so a block stores it once per channel. - `data` holds decoded JSON only when every selected channel is JSON. Otherwise it holds bytes. Planning types it from the selected messages of the first chunk that holds one, as a read task types the messages it holds. - `row_id` is `<path>#c=<chunk offset>:<index>`, stable however the read is split. It is on with `include_row_id=True`, or automatically under `CheckpointConfig(id_column="row_id")`. - New `log_time_order` argument, default `True`. Each task returns each of its files in `log_time` order. With `preserve_order`, each file is read by one task, so it comes out in order from start to end, and files come out in the order they were listed. #67166 splits a large file there too, only where its order holds. - A file whose summary cannot be read gets a read task of its own, so `max_errored_blocks` can skip it alone. - The task size is `RAY_DATA_MCAP_BIN_PACKING_BYTES` (128 MiB). **Changed after the first review round.** Each is its own commit: - Summaries and chunks are read in a few ranged requests, not a few bytes per GET. - An unreadable file gets a read task of its own. - Tables are built column by column, cut counting every column, and chunks are weighted by the memory reading them takes. - The read stops at the row limit. - A file without a chunk index is sorted by log time. - `channel_metadata` is JSON text, stored once per block. - With `preserve_order`, each file is kept in one read task, and files keep their listing order. - Planning skips a chunk it cannot decode, and types `data` from the first chunk's messages. **Measured** on an 845 MB recording with 1,052 chunks: reading its 125 KB summary took about 20,000 GETs and now takes 3. **Decisions to confirm:** - Splitting is on by default. Block boundaries change; rows do not. - With `preserve_order`, this PR reads each file in one task, as the legacy reader does. #67166 lets it split a large file where its log-time order holds. - `ds.schema()` comes from the first chunk that holds a selected message. A field that first appears later is missing from it; the docstring says so. ## Related issues - Related to #65640 and #65641. - No open PR moves MCAP to Datasource V2. #66524 (ORC) and #66182 (Iceberg) are other formats moving to V2. - The generated-ID stack (#65886 to #65888) is orthogonal. - `log_time_order` comes from my #66653, now closed. My legacy-path PRs #65912, #65913 and #65916 are closed in favour of this one. ## Additional information ``` PYTHONPATH=python/ray/data/tests pytest python/ray/data/tests/datasource/test_mcap.py \ python/ray/data/tests/unit/datasource_v2/test_mcap_*.py \ python/ray/data/tests/unit/datasource_v2/test_online_bin_packer.py ``` 178 passed, 3 skipped. pre-commit and pyrefly clean. AI assistance (Claude Code) was used to draft this change; I reviewed every line and ran the tests above locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Description
Draft for visibility — do not merge. This is the whole
read_iceberg→Datasource V2 port in one diff so the design can be seen end to end. It lands as
a stack of small reviewable PRs instead; this branch exists so reviewers do not
have to reconstruct the shape from five partial diffs, and it may lag them.
Datasource V2 is itself under active change, so the seams this uses are moving
and the branch is already behind master. It is here for visibility, not review —
the reviewable PRs will be cut against whatever V2 looks like when they land.
Motivation and the measured numbers are in #66181 — please leave design comments
there.
Initially behind an opt-in flag:
DataContext.use_iceberg_datasource_v2(default
False, envRAY_DATA_USE_ICEBERG_DATASOURCE_V2). Nothing changes onthe default path and the public
read_icebergsignature does not change.The driver keeps only plan-time work: schema and snapshot come from table
metadata, so no file is sampled.
plan_files()moves into the listing task withthe pushed-down filter attached, and its scan tasks are encoded as typed manifest
columns — the only thing that crosses from listing to reading. Read units are
bin packed on estimated decoded bytes rather than compressed bytes, and the
reader streams one block per record batch instead of materializing a chunk.
Two shortcuts fall out of the manifest. Where Iceberg's per-file residual is
AlwaysTrue, the recorded record count is the post-filter count, socount()needs no data — which also required a small shared-code fix, since the count
rule rewrites the plan in a way that was discarding the listing filter and
counting the whole table. Where the snapshot summary is trustworthy,
count()returns on the driver with no tasks at all; both paths fail closed to the
ordinary read.
The existing Iceberg suite runs parametrized over both readers, so every existing
assertion is a parity assertion. New tests cover what has no V1 counterpart:
the manifest round trip including value types, delete files carried and
applied, awkward partition types and an evolved spec, bin-packing granularity,
listing-time file pruning, and the count paths.
Not implemented yet, and what we intend after this
AlwaysTrue,with no delete files and a schema matching the table's, is just a Parquet
file — route it through Ray's own Parquet reader and send only the dirty
minority through PyIceberg. Needs a schema-parity suite between the two
readers first.
split_offsetsgives the split points with no footer reads, but PyIceberg'stask type has no offset field, so this is gated on the native path above.
files and each is read once per task that needs it (the cost in [Data] read_iceberg re-reads a shared delete file once per data file instead of once per read task #66147).
Out of scope, each blocked outside this reader: incremental scans (#64464),
equality deletes (PyIceberg raises at plan time), and plan-time physical
properties (uniqueness / clustering / sortedness / NDV).
Known gaps
correctly, but nothing yet proves PyIceberg attaches catalog-planned delete
manifests to the right tasks.
SqlCatalogonfile://so far, so no catalog credential hashad to survive pickling into a listing task on another machine.
Related issues
Related to #66181 and #66147. Not a duplicate: no open issue or PR covers moving
read_icebergto Datasource V2.Additional information
Tests run locally —
pytestontests/datasource/test_iceberg.py,tests/unit/datasource_v2/,_internal/logical/operators/tests/test_read_files_logical.pyandtests/datasource/test_read_parquet_v2.py: 284 passed. Lint clean under Ray'spinned
ruff 0.8.4/black 22.10.0;pyreflyclean on the changed files.AI assistance (Claude Code) was used. I have reviewed every changed line, can
defend the design end to end, and ran the tests above locally.
Signed-off-by: Aarrya aarrya.saraf@anyscale.com
🤖 Generated with Claude Code