Skip to content

ParquetMetaDataWriter silently omits page indexes from custom PageIndexProviders #11030

Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We added the PageIndexProvider trait so users can supply custom page index representations. However, when a ParquetMetaData holds a custom PageIndexProvider implementation, ParquetMetaDataWriter::finish silently omits the page indexes (ColumnIndex / OffsetIndex) from the output.

This means a user who:

  1. Reads parquet metadata using a custom PageIndexProvider, and
  2. Writes that metadata back out with ParquetMetaDataWriter

gets a file with no page indexes — with no error or warning. Readers of that file can no longer do page-level pruning, silently degrading query performance.

The cause is that the writer serializes the indexes by downcasting to the concrete built-in PageIndex type, which fails for custom providers:

// Downcast to PageIndex to access raw index structures for serialization
// TODO: rework the encoder to work with the PageIndexProvider API to
// remove the need for this cast. Because the page index is a dyn trait,
// it can't be cloned. An implementation would have to create a new
// `PageIndex` from the current `PageIndexProvider` and pass that to
// the encoder.
if let Some(page_index_arc) = self.metadata.page_index.as_ref()
&& let Some(page_index) = page_index_arc
.as_any()
.downcast_ref::<crate::file::metadata::PageIndex>()
{

// Downcast to PageIndex to access raw index structures for serialization
// TODO: rework the encoder to work with the PageIndexProvider API to
// remove the need for this cast. Because the page index is a dyn trait,
// it can't be cloned. An implementation would have to create a new
// `PageIndex` from the current `PageIndexProvider` and pass that to
// the encoder.
if let Some(page_index_arc) = self.metadata.page_index.as_ref()
    && let Some(page_index) = page_index_arc
        .as_any()
        .downcast_ref::<crate::file::metadata::PageIndex>()

Describe the solution you'd like

Rework the metadata thrift encoder to work with the PageIndexProvider API directly, so page indexes are written regardless of the concrete provider implementation.

Describe alternatives you've considered

  • Materialize a built-in PageIndex from any custom provider before serialization (simpler, but requires an extra copy of the index data).
  • Keep the limitation but make it visible: document it on ParquetMetaDataWriter and/or return an error instead of silently omitting the indexes.

Additional context

Activity

  1. etseidl commented on Sep 8, 2026

    @etseidl
    Contributor

    😮

  2. self-assigned this
    on Sep 8, 2026
  3. etseidl commented on Sep 8, 2026

    @etseidl
    Contributor

    I have a PR pretty much ready to go to serialize indexes from custom providers...just waiting on #10842

  4. alamb commented on Sep 9, 2026

    @alamb
    ContributorAuthor

    I have a PR pretty much ready to go to serialize indexes from custom providers...just waiting on #10842

    LOL -- now I just need to review it!

  5. etseidl commented on Sep 9, 2026

    @etseidl
    Contributor

    LOL -- now I just need to review it!

    Probably not worth holding up 60.0.0 for this. 😅

  6. added a commit that references this issue on Sep 24, 2026
    34060c4
  7. added a commit that references this issue on Sep 25, 2026
    cfc331c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions