Repository navigation
Description
Feature Request / Improvement
Motivation
Iceberg's custom row-based Parquet reader currently performs pruning at row-group granularity but does not use Parquet ColumnIndex/OffsetIndex metadata to skip individual pages within retained row groups.
This can result in substantially more I/O when a selective predicate matches only a small portion of a large row group, particularly when data is physically clustered by the predicate column.
Current POC
I built a POC against current main using parquet-java 1.17.1.
The POC integrates Parquet Page Index pruning into Iceberg's custom row-based reader using:
Iceberg Expression
-> Parquet predicate
-> ColumnIndex filtering
-> readFilteredRowGroup(rowGroupIndex)
-> filtered PageReadStore
-> Iceberg ParquetValueReader
This remains conservative page-level pruning. It does not replace Iceberg's residual predicate evaluation.
Correctness findings
The POC currently verifies the following:
- ColumnIndex and OffsetIndex generated by Iceberg writers can be consumed by parquet-java for page-level pruning.
- A narrow predicate can produce non-contiguous candidate row ranges.
PageReadStore.getRowIndexes()exposes row-group-relative row indexes.PageReadStore.getRowIndexOffset()provides the file-level row-group start.- Physical positions can therefore be reconstructed as:
file position = rowIndexOffset + relative row index
- Iceberg
_posremains the original physical file position after page pruning. - fallback
_row_idvalues derived fromfirst_row_id + _posremain correct. - multiple row groups are handled correctly.
- row groups for which the ColumnIndex eliminates every page are skipped correctly.
- files without a usable ColumnIndex conservatively fall back to reading the retained row group.
These position semantics are important because candidate pages cannot be renumbered without breaking metadata-column correctness and row-level delete semantics.
File schema / filter conversion
One issue exposed by the POC is that ParquetFilters.convert requires file-specific physical aliases.
A plain logical Iceberg read schema is insufficient because the Parquet predicate needs to resolve the current Iceberg field ID to the physical path used in the particular Parquet file.
The current POC therefore:
opens the Parquet file
-> discovers the physical schema and aliases
-> constructs the Parquet predicate
-> reopens the reader with the final ParquetReadOptions
This extra open is acceptable for demonstrating feasibility but may not intended as the final reader architecture.
This is also closely related to the schema/alias concerns discussed in #1566.
Initial I/O measurements
I also ran a small local POC microbenchmark using:
- 500,000 rows
- one row group
- approximately 1,000 rows per page
- uncompressed Parquet
- projection:
id,payload - predicate:
id >= 250000 AND id < 250100
The current double-open overhead is included in the byte measurements.
| Dataset | Page Index | Candidate Rows | Bytes Read | Median Time |
|---|---|---|---|---|
| Sorted | OFF | 500,000 | 131.054 MiB | 67.79 ms |
| Sorted | ON | 1,000 | 0.288 MiB | 8.16 ms |
| Random | OFF | 500,000 | 131.054 MiB | 66.56 ms |
| Random | ON | 500,000 | 131.079 MiB | 69.72 ms |
| Sorted / no index | OFF | 500,000 | 131.054 MiB | 66.02 ms |
| Sorted / no index | ON | 500,000 | 131.070 MiB | 69.15 ms |
This is not a formal JMH benchmark. The timing numbers are directional; the byte-count results are the main signal.
For sorted data, page pruning reduced candidate rows by 99.8% and measured bytes read by approximately 99.78%.
For randomly distributed data, the ColumnIndex could not eliminate pages.
A file without a predicate-column ColumnIndex also fell back conservatively.
Proposed initial scope
I suggest keeping the first implementation intentionally narrow:
- custom row-based Parquet reader only;
- Page Index used only as conservative I/O pruning;
- residual predicate evaluation remains unchanged;
- simple supported Parquet predicates first;
- missing/unsupported Page Index information falls back conservatively;
- no public Spark/Flink/table configuration in the first implementation.
Vectorized-reader support, broader predicate coverage, and configuration/default policy can be handled separately once the row-based path is well defined.
Query engine
None
Willingness to contribute
- I can contribute this improvement/feature independently
- I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- I cannot contribute this improvement/feature at this time
Activity
[-]Parquet: Page Index pruning in the custom Parquet reader[/-][+]Parquet: POC Page Index pruning in the custom Parquet reader[/+][-]Parquet: POC Page Index pruning in the custom Parquet reader[/-][+]Parquet: Page Index pruning in the custom Parquet reader[/+]ipatriciahonorato commentedon Oct 30, 2026
Hi, we tested the #17597 draft on a real workload and it helps a lot. On our wide scans, Iceberg went from 2.0-2.7x slower than reading the same Parquet files directly with Spark to 2.5-3x faster.
Setup: one day of time-series data (about 1 TB of Parquet, rows with wide structs of arrays), Spark 3.5.8, Java 17, HDFS, default 128 MiB row groups. We built Iceberg main (commit 6938d21) with the #17597 draft, the fix suggested in its review, and parquet-java 1.17.1 with #3748. We also added a switch to turn pruning on and off, and support for IN predicates. Most of our queries filter on a list of ids, so for us IN support would be needed in the first version, not later.
The queries read all columns for a set of ids in a 1 minute window. Two runs with pruning off and two with it on:
A query that reads array columns read 44% less data and got 20% faster. Results matched direct Parquet in all our queries, but we only checked our own scenarios.
Two things we noticed:
We'd really like to see this merged and are happy to run more tests or help review.
yangshangqing95 commentedon Oct 30, 2026
Hi @ipatriciahonorato thanks for sharing your test result, that's exactly what motivated me to put forward this proposal.
I’m planning to clean up and finalize the formal version before submitting it sometime soon. I previously submitted a few prerequisite PRs, but I haven’t received any feedback from the community yet.