Repository navigation
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
- Follow on to Add
PageIndexBuilderandPageIndexProviderfor Parquet page indexes #10842 from @etseidl
We added the PageIndexProvider trait so users can supply custom page index representations. However, when a ParquetMetaData holds a custom PageIndexProvider implementation, ParquetMetaDataWriter::finish silently omits the page indexes (ColumnIndex / OffsetIndex) from the output.
This means a user who:
- Reads parquet metadata using a custom
PageIndexProvider, and - Writes that metadata back out with
ParquetMetaDataWriter
gets a file with no page indexes — with no error or warning. Readers of that file can no longer do page-level pruning, silently degrading query performance.
The cause is that the writer serializes the indexes by downcasting to the concrete built-in PageIndex type, which fails for custom providers:
arrow-rs/parquet/src/file/metadata/writer.rs
Lines 464 to 474 in 79dbf5a
| // Downcast to PageIndex to access raw index structures for serialization | |
| // TODO: rework the encoder to work with the PageIndexProvider API to | |
| // remove the need for this cast. Because the page index is a dyn trait, | |
| // it can't be cloned. An implementation would have to create a new | |
| // `PageIndex` from the current `PageIndexProvider` and pass that to | |
| // the encoder. | |
| if let Some(page_index_arc) = self.metadata.page_index.as_ref() | |
| && let Some(page_index) = page_index_arc | |
| .as_any() | |
| .downcast_ref::<crate::file::metadata::PageIndex>() | |
| { |
// Downcast to PageIndex to access raw index structures for serialization
// TODO: rework the encoder to work with the PageIndexProvider API to
// remove the need for this cast. Because the page index is a dyn trait,
// it can't be cloned. An implementation would have to create a new
// `PageIndex` from the current `PageIndexProvider` and pass that to
// the encoder.
if let Some(page_index_arc) = self.metadata.page_index.as_ref()
&& let Some(page_index) = page_index_arc
.as_any()
.downcast_ref::<crate::file::metadata::PageIndex>()Describe the solution you'd like
Rework the metadata thrift encoder to work with the PageIndexProvider API directly, so page indexes are written regardless of the concrete provider implementation.
Describe alternatives you've considered
- Materialize a built-in
PageIndexfrom any custom provider before serialization (simpler, but requires an extra copy of the index data). - Keep the limitation but make it visible: document it on
ParquetMetaDataWriterand/or return an error instead of silently omitting the indexes.
Additional context
- Original review discussion: Add
PageIndexBuilderandPageIndexProviderfor Parquet page indexes #10842 (comment)
Activity
etseidl commentedon Sep 8, 2026
😮
etseidl commentedon Sep 8, 2026
I have a PR pretty much ready to go to serialize indexes from custom providers...just waiting on #10842
alamb commentedon Sep 9, 2026
LOL -- now I just need to review it!
etseidl commentedon Sep 9, 2026
Probably not worth holding up 60.0.0 for this. 😅
Allow for serialization of Parquet Page Index via custom `PageIndexPr…
Allow for serialization of Parquet Page Index via custom `PageIndexPr…