Skip to content

feat: Add separate Parquet PageIndex decoder to ParquetMetaDataPushDecoder and ParquetMetaDataReader - #11438

Merged
alamb merged 13 commits into
apache:mainfrom
etseidl:decode_page_index
Oct 15, 2026
Merged

alamb merged 13 commits into
apache:mainfrom
etseidl:decode_page_index

Conversation

@etseidl

@etseidl etseidl commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The current APIs for loading the page indexes all modify the ParquetMetaData, installing newly parsed page indexes via set_page_index. Users wishing to create a page index cache can currently fetch and decode indexes themselves, but it would be nice to be able to simply parse and return the requested page indexes without installing them in the metadata for return.

What changes are included in this PR?

Adds a new try_decode_page_index function to ParquetMetaDataPushDecoder, as well as helper wrapper functions to ParquetMetaDataReader.

Note that the required ParquetMetaData is consumed in this process, but #11297 should make this more palatable.

Are these changes tested?

Yes, includes new tests of the added APIs

Are there any significant user-facing changes?

No changes to existing public APIs, adds new functions to the public API.

AI Disclaimer

Codex was used in the creation of this PR, but I am responsible for the changes.

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Oct 7, 2026
Comment thread parquet/src/file/metadata/reader.rs Outdated
///
/// This operation consumes `self` and the stored [`ParquetMetaData`]. Use
/// [`Self::new_with_metadata`] to construct the reader.
pub fn read_page_index<R: ChunkReader>(self, reader: &R) -> Result<Option<PageIndex>> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to better (or at least less similar) names here and below.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about read_page_index_and_finish and load_page_index_and_finish to mirror existing APIs (suggested by Claude)?

It is also kind of strange that this consumes self -- which has an embedded ParquetMetaData -- maybe it could return the (ParquetMetaData, Option<PageIndex>) instead 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a deliberate decision by me to add justification for the easy to clone metadata 😄. I also think it's weird for a get_page_index kind of call to return a tuple. But I guess we could go that way if people don't like the clone.

@etseidl etseidl changed the title feat: Add separate Parquet PageIndex decoder to ParquetMetaDataPushDecoder feat: Add separate Parquet PageIndex decoder to ParquetMetaDataPushDecoder and ParquetMetaDataReader Oct 8, 2026

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @etseidl -- this looks very cool to me.

I had some questions on API design that I posted. Let me know what you think

/// Try to decode the page indexes from the pushed data, returning the
/// decoded [`PageIndex`] or an error if not enough data is available.
///
/// This method can only be called after [`Self::try_new_with_metadata`],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could add an example here of how to use this function -- this text explains it in words, but I am still not quite sure how it would work

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you know I had one, but then deleted it thinking I could later point to https://github.com/etseidl/arrow-rs/blob/58ae3172712f0a43ac785325630b77379464cb8d/parquet/examples/page_index_cache.rs#L156 which has the full example. I can add it back here if you'd prefer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here's where the example was 7e6518a

@etseidl etseidl Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added an example to the docs for ParquetMetaDataReader::read_page_index_and_finish which I think is the better place to demonstrate the two stage page index reading. be1ea0f

page_index_range.start,
)?;

self.state = DecodeState::Finished;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does setting the state to Finished mean that we can only try and decode page metadata once and that subsequent calls to try_decode_page_index will error? I suppose that makes sense

But I do think an end to end example showing how to use these APIs to add PageIndex to an existing ParquetMetadata would be super helpful

Like is it read the metadata without a page index and then create a new reader with that metadata configured to just read the relevant parts of the page index 🤔

@etseidl etseidl Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think the model would be create a new decoder to get the state back to ReadingPageIndex.

In my mind it's start with metadata and no page index. When a query comes in, use this new API to grab the bits you need, store them somewhere, and then create a new PageIndex (or custom PageIndexProvider), stick that into a cloned metadata and execute query with that metadata.

New query comes in, figure out the needed indexes, compare that with what's already been fetched, use this new API to get the new bits, merge those into the cache, then create new PageIndex and cloned metadata to satisfy query 2.

Rinse repeat.

Again, I think the example in #11475 is my current thinking.

Intermediate,
}

enum PageIndexDecodeResult {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty similar to DecodeResult<(Box<ParquetMetaData>,Option<PageIndex>) 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tru...let me work on that

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So claude points out reusing DecodeResult means having to account for the Finished variant which is an impossible state. I'll switch if you hate the new enum.

assert!(expect_data(decoder.try_decode_page_index()).is_none());
expect_finished(decoder.try_decode_page_index());

let metadata = test_metadata_without_page_index();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think another interesting test would be to make the second ParquetMetaDataPushDecoder with a page_index_mask to read it as a subsequent step

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally looping back to this one. The preceding test (test_decode_page_index_after_metadata) does what I think you're requesting here. It first reads the metadata without a page index, then does a subsequent call to the new try_decode_page_index with a mask set. Or am I misunderstanding your intent?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I was thinking more of something slightly different:

Read the metadata (but no PageIndex) via

let mut decoder =
            ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
                .unwrap()
                .with_page_index_policy(PageIndexPolicy::Skip);
...

And then make a new one like

   let mut decoder =
            ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
                .unwrap()
                .with_page_index_mask(ColumnChunkMask::all());
  ... // decode here and ensure the metadata has the column index 

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is all basically the same as me asking for an example of how to use this / a demo of incrementally decoding the PageIndex

Comment thread parquet/src/file/metadata/reader.rs Outdated
///
/// This operation consumes `self` and the stored [`ParquetMetaData`]. Use
/// [`Self::new_with_metadata`] to construct the reader.
pub fn read_page_index<R: ChunkReader>(self, reader: &R) -> Result<Option<PageIndex>> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about read_page_index_and_finish and load_page_index_and_finish to mirror existing APIs (suggested by Claude)?

It is also kind of strange that this consumes self -- which has an embedded ParquetMetaData -- maybe it could return the (ParquetMetaData, Option<PageIndex>) instead 🤔

@etseidl

etseidl commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review @alamb. I'll work on fixing this up Monday tonight.

I wound up changing the new functions to not consume the metadata and eliminated the new enum. I also refactored the reader code some, and in the process discovered that the reader's metadata is not restored on a needs-more-data error return. I'll file an issue for that. Still working on your test suggestion.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @etseidl -- I went through this again and I think it makes a lot of sense. I took the liberty of pushing a commit that fixes a small error hole (make it so we can decode even if there is an error) and updates some comments

It is unfortunately that we have load_..., read_..., load_and_finish_... but I don't really see any ways around it without deprecating a bunch of things (like maybe the async methods 🤔 )

let page_index = builder.build();
if page_index.has_column_indexes() || page_index.has_offset_indexes() {
metadata.set_page_index(Some(Arc::new(page_index)));
Ok(Some(page_index))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a much nicer API 👍

assert!(expect_data(decoder.try_decode_page_index()).is_none());
expect_finished(decoder.try_decode_page_index());

let metadata = test_metadata_without_page_index();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I was thinking more of something slightly different:

Read the metadata (but no PageIndex) via

let mut decoder =
            ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
                .unwrap()
                .with_page_index_policy(PageIndexPolicy::Skip);
...

And then make a new one like

   let mut decoder =
            ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
                .unwrap()
                .with_page_index_mask(ColumnChunkMask::all());
  ... // decode here and ensure the metadata has the column index 

assert!(expect_data(decoder.try_decode_page_index()).is_none());
expect_finished(decoder.try_decode_page_index());

let metadata = test_metadata_without_page_index();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is all basically the same as me asking for an example of how to use this / a demo of incrementally decoding the PageIndex

pub fn read_page_indexes_sized<R: ChunkReader>(
/// Unlike [`Self::read_page_indexes`], this returns the newly decoded [`PageIndex`] separately;
/// any page index already attached to the returned metadata is unchanged. This is useful for
/// populating an external page index cache. The configured page index policies and masks

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

@etseidl

etseidl commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @alamb!

It is unfortunately that we have load_..., read_..., load_and_finish_... but I don't really see any ways around it without deprecating a bunch of things (like maybe the async methods 🤔 )

Yes, I find this unfortunate as well. FWIW I've embarked on a Bloom filter provider akin to the page index provider, and I'm trying to not repeat past mistakes. Hopefully when that's complete, we can apply lessons learned to the page indexes (when we can make breaking changes ofc).

@etseidl

etseidl commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

@alamb I added one more correctness tweak in 7bb72d5. If that looks good to you then I think this is ready to merge. Thanks again for the reviews!

@alamb

alamb commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Looks good to me -- let's 🚢 🇮🇹

@alamb
alamb added this pull request to the merge queue Oct 15, 2026
Merged via the queue into apache:main with commit 152f0d3 Oct 15, 2026
25 checks passed
@alamb

alamb commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

woho!

@Jefffrey Jefffrey added the enhancement New feature/addition label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature/addition parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants