Skip to content

Parquet: Page Index pruning in the custom Parquet reader #17596

Open

Description

@yangshangqing95

Feature Request / Improvement

Motivation

Iceberg's custom row-based Parquet reader currently performs pruning at row-group granularity but does not use Parquet ColumnIndex/OffsetIndex metadata to skip individual pages within retained row groups.

This can result in substantially more I/O when a selective predicate matches only a small portion of a large row group, particularly when data is physically clustered by the predicate column.

Current POC

I built a POC against current main using parquet-java 1.17.1.

The POC integrates Parquet Page Index pruning into Iceberg's custom row-based reader using:

Iceberg Expression
    -> Parquet predicate
    -> ColumnIndex filtering
    -> readFilteredRowGroup(rowGroupIndex)
    -> filtered PageReadStore
    -> Iceberg ParquetValueReader

This remains conservative page-level pruning. It does not replace Iceberg's residual predicate evaluation.

Correctness findings

The POC currently verifies the following:

  • ColumnIndex and OffsetIndex generated by Iceberg writers can be consumed by parquet-java for page-level pruning.
  • A narrow predicate can produce non-contiguous candidate row ranges.
  • PageReadStore.getRowIndexes() exposes row-group-relative row indexes.
  • PageReadStore.getRowIndexOffset() provides the file-level row-group start.
  • Physical positions can therefore be reconstructed as:
file position = rowIndexOffset + relative row index
  • Iceberg _pos remains the original physical file position after page pruning.
  • fallback _row_id values derived from first_row_id + _pos remain correct.
  • multiple row groups are handled correctly.
  • row groups for which the ColumnIndex eliminates every page are skipped correctly.
  • files without a usable ColumnIndex conservatively fall back to reading the retained row group.

These position semantics are important because candidate pages cannot be renumbered without breaking metadata-column correctness and row-level delete semantics.

File schema / filter conversion

One issue exposed by the POC is that ParquetFilters.convert requires file-specific physical aliases.

A plain logical Iceberg read schema is insufficient because the Parquet predicate needs to resolve the current Iceberg field ID to the physical path used in the particular Parquet file.

The current POC therefore:

opens the Parquet file
-> discovers the physical schema and aliases
-> constructs the Parquet predicate
-> reopens the reader with the final ParquetReadOptions

This extra open is acceptable for demonstrating feasibility but may not intended as the final reader architecture.

This is also closely related to the schema/alias concerns discussed in #1566.

Initial I/O measurements

I also ran a small local POC microbenchmark using:

  • 500,000 rows
  • one row group
  • approximately 1,000 rows per page
  • uncompressed Parquet
  • projection: id, payload
  • predicate: id >= 250000 AND id < 250100

The current double-open overhead is included in the byte measurements.

Dataset Page Index Candidate Rows Bytes Read Median Time
Sorted OFF 500,000 131.054 MiB 67.79 ms
Sorted ON 1,000 0.288 MiB 8.16 ms
Random OFF 500,000 131.054 MiB 66.56 ms
Random ON 500,000 131.079 MiB 69.72 ms
Sorted / no index OFF 500,000 131.054 MiB 66.02 ms
Sorted / no index ON 500,000 131.070 MiB 69.15 ms

This is not a formal JMH benchmark. The timing numbers are directional; the byte-count results are the main signal.

For sorted data, page pruning reduced candidate rows by 99.8% and measured bytes read by approximately 99.78%.

For randomly distributed data, the ColumnIndex could not eliminate pages.

A file without a predicate-column ColumnIndex also fell back conservatively.

Proposed initial scope

I suggest keeping the first implementation intentionally narrow:

  • custom row-based Parquet reader only;
  • Page Index used only as conservative I/O pruning;
  • residual predicate evaluation remains unchanged;
  • simple supported Parquet predicates first;
  • missing/unsupported Page Index information falls back conservatively;
  • no public Spark/Flink/table configuration in the first implementation.

Vectorized-reader support, broader predicate coverage, and configuration/default policy can be handled separately once the row-based path is well defined.

Query engine

None

Willingness to contribute

  • I can contribute this improvement/feature independently
  • I would be willing to contribute this improvement/feature with guidance from the Iceberg community
  • I cannot contribute this improvement/feature at this time

Activity

  1. changed the title [-]Parquet: Page Index pruning in the custom Parquet reader[/-] [+]Parquet: POC Page Index pruning in the custom Parquet reader[/+] on Aug 10, 2026
  2. changed the title [-]Parquet: POC Page Index pruning in the custom Parquet reader[/-] [+]Parquet: Page Index pruning in the custom Parquet reader[/+] on Aug 10, 2026
  3. ipatriciahonorato commented on Oct 30, 2026

    @ipatriciahonorato

    Hi, we tested the #17597 draft on a real workload and it helps a lot. On our wide scans, Iceberg went from 2.0-2.7x slower than reading the same Parquet files directly with Spark to 2.5-3x faster.

    Setup: one day of time-series data (about 1 TB of Parquet, rows with wide structs of arrays), Spark 3.5.8, Java 17, HDFS, default 128 MiB row groups. We built Iceberg main (commit 6938d21) with the #17597 draft, the fix suggested in its review, and parquet-java 1.17.1 with #3748. We also added a switch to turn pruning on and off, and support for IN predicates. Most of our queries filter on a list of ids, so for us IN support would be needed in the first version, not later.

    The queries read all columns for a set of ids in a 1 minute window. Two runs with pruning off and two with it on:

    pruning off pruning on
    Iceberg time / Parquet time (3 wide-scan queries) 2.0-2.7 0.33-0.40
    Iceberg data read (one query) 1593 MB 107 MB
    Iceberg rows read (same query) 28.5 M 1.88 M

    A query that reads array columns read 44% less data and got 20% faster. Results matched direct Parquet in all our queries, but we only checked our own scenarios.

    Two things we noticed:

    • Queries that only read primitive columns go through the vectorized reader and don't change, so vectorized support (Parquet: Preserve non-contiguous row positions in vectorized reads #17663) would matter for us too.
    • In our table with the widest rows, a row group has only about 3,500 rows, and the filter columns have one page per row group, so nothing gets pruned there. A smaller page row limit should help, but we haven't tested it yet.

    We'd really like to see this merged and are happy to run more tests or help review.

  4. yangshangqing95 commented on Oct 30, 2026

    @yangshangqing95
    ContributorAuthor

    Hi @ipatriciahonorato thanks for sharing your test result, that's exactly what motivated me to put forward this proposal.
    I’m planning to clean up and finalize the formal version before submitting it sometime soon. I previously submitted a few prerequisite PRs, but I haven’t received any feedback from the community yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementPR that improves existing functionality

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions