Repository navigation
feat: Add separate Parquet PageIndex decoder to ParquetMetaDataPushDecoder and ParquetMetaDataReader - #11438
Conversation
| /// | ||
| /// This operation consumes `self` and the stored [`ParquetMetaData`]. Use | ||
| /// [`Self::new_with_metadata`] to construct the reader. | ||
| pub fn read_page_index<R: ChunkReader>(self, reader: &R) -> Result<Option<PageIndex>> { |
There was a problem hiding this comment.
I'm open to better (or at least less similar) names here and below.
There was a problem hiding this comment.
How about read_page_index_and_finish and load_page_index_and_finish to mirror existing APIs (suggested by Claude)?
It is also kind of strange that this consumes self -- which has an embedded ParquetMetaData -- maybe it could return the (ParquetMetaData, Option<PageIndex>) instead 🤔
There was a problem hiding this comment.
That was a deliberate decision by me to add justification for the easy to clone metadata 😄. I also think it's weird for a get_page_index kind of call to return a tuple. But I guess we could go that way if people don't like the clone.
PageIndex decoder to ParquetMetaDataPushDecoderPageIndex decoder to ParquetMetaDataPushDecoder and ParquetMetaDataReader
| /// Try to decode the page indexes from the pushed data, returning the | ||
| /// decoded [`PageIndex`] or an error if not enough data is available. | ||
| /// | ||
| /// This method can only be called after [`Self::try_new_with_metadata`], |
There was a problem hiding this comment.
I wonder if we could add an example here of how to use this function -- this text explains it in words, but I am still not quite sure how it would work
There was a problem hiding this comment.
you know I had one, but then deleted it thinking I could later point to https://github.com/etseidl/arrow-rs/blob/58ae3172712f0a43ac785325630b77379464cb8d/parquet/examples/page_index_cache.rs#L156 which has the full example. I can add it back here if you'd prefer.
There was a problem hiding this comment.
I've added an example to the docs for ParquetMetaDataReader::read_page_index_and_finish which I think is the better place to demonstrate the two stage page index reading. be1ea0f
| page_index_range.start, | ||
| )?; | ||
|
|
||
| self.state = DecodeState::Finished; |
There was a problem hiding this comment.
Does setting the state to Finished mean that we can only try and decode page metadata once and that subsequent calls to try_decode_page_index will error? I suppose that makes sense
But I do think an end to end example showing how to use these APIs to add PageIndex to an existing ParquetMetadata would be super helpful
Like is it read the metadata without a page index and then create a new reader with that metadata configured to just read the relevant parts of the page index 🤔
There was a problem hiding this comment.
Yeah, I think the model would be create a new decoder to get the state back to ReadingPageIndex.
In my mind it's start with metadata and no page index. When a query comes in, use this new API to grab the bits you need, store them somewhere, and then create a new PageIndex (or custom PageIndexProvider), stick that into a cloned metadata and execute query with that metadata.
New query comes in, figure out the needed indexes, compare that with what's already been fetched, use this new API to get the new bits, merge those into the cache, then create new PageIndex and cloned metadata to satisfy query 2.
Rinse repeat.
Again, I think the example in #11475 is my current thinking.
| Intermediate, | ||
| } | ||
|
|
||
| enum PageIndexDecodeResult { |
There was a problem hiding this comment.
This looks pretty similar to DecodeResult<(Box<ParquetMetaData>,Option<PageIndex>) 🤔
There was a problem hiding this comment.
tru...let me work on that
There was a problem hiding this comment.
So claude points out reusing DecodeResult means having to account for the Finished variant which is an impossible state. I'll switch if you hate the new enum.
| assert!(expect_data(decoder.try_decode_page_index()).is_none()); | ||
| expect_finished(decoder.try_decode_page_index()); | ||
|
|
||
| let metadata = test_metadata_without_page_index(); |
There was a problem hiding this comment.
I think another interesting test would be to make the second ParquetMetaDataPushDecoder with a page_index_mask to read it as a subsequent step
There was a problem hiding this comment.
Finally looping back to this one. The preceding test (test_decode_page_index_after_metadata) does what I think you're requesting here. It first reads the metadata without a page index, then does a subsequent call to the new try_decode_page_index with a mask set. Or am I misunderstanding your intent?
There was a problem hiding this comment.
I guess I was thinking more of something slightly different:
Read the metadata (but no PageIndex) via
let mut decoder =
ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
.unwrap()
.with_page_index_policy(PageIndexPolicy::Skip);
...And then make a new one like
let mut decoder =
ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
.unwrap()
.with_page_index_mask(ColumnChunkMask::all());
... // decode here and ensure the metadata has the column index There was a problem hiding this comment.
I think this is all basically the same as me asking for an example of how to use this / a demo of incrementally decoding the PageIndex
| /// | ||
| /// This operation consumes `self` and the stored [`ParquetMetaData`]. Use | ||
| /// [`Self::new_with_metadata`] to construct the reader. | ||
| pub fn read_page_index<R: ChunkReader>(self, reader: &R) -> Result<Option<PageIndex>> { |
There was a problem hiding this comment.
How about read_page_index_and_finish and load_page_index_and_finish to mirror existing APIs (suggested by Claude)?
It is also kind of strange that this consumes self -- which has an embedded ParquetMetaData -- maybe it could return the (ParquetMetaData, Option<PageIndex>) instead 🤔
|
Thanks for the review @alamb. I'll work on fixing this up I wound up changing the new functions to not consume the metadata and eliminated the new enum. I also refactored the reader code some, and in the process discovered that the reader's metadata is not restored on a needs-more-data error return. I'll file an issue for that. Still working on your test suggestion. |
alamb
left a comment
There was a problem hiding this comment.
Thanks @etseidl -- I went through this again and I think it makes a lot of sense. I took the liberty of pushing a commit that fixes a small error hole (make it so we can decode even if there is an error) and updates some comments
It is unfortunately that we have load_..., read_..., load_and_finish_... but I don't really see any ways around it without deprecating a bunch of things (like maybe the async methods 🤔 )
| let page_index = builder.build(); | ||
| if page_index.has_column_indexes() || page_index.has_offset_indexes() { | ||
| metadata.set_page_index(Some(Arc::new(page_index))); | ||
| Ok(Some(page_index)) |
There was a problem hiding this comment.
I think this is a much nicer API 👍
| assert!(expect_data(decoder.try_decode_page_index()).is_none()); | ||
| expect_finished(decoder.try_decode_page_index()); | ||
|
|
||
| let metadata = test_metadata_without_page_index(); |
There was a problem hiding this comment.
I guess I was thinking more of something slightly different:
Read the metadata (but no PageIndex) via
let mut decoder =
ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
.unwrap()
.with_page_index_policy(PageIndexPolicy::Skip);
...And then make a new one like
let mut decoder =
ParquetMetaDataPushDecoder::try_new_with_metadata(test_file_len(), metadata)
.unwrap()
.with_page_index_mask(ColumnChunkMask::all());
... // decode here and ensure the metadata has the column index | assert!(expect_data(decoder.try_decode_page_index()).is_none()); | ||
| expect_finished(decoder.try_decode_page_index()); | ||
|
|
||
| let metadata = test_metadata_without_page_index(); |
There was a problem hiding this comment.
I think this is all basically the same as me asking for an example of how to use this / a demo of incrementally decoding the PageIndex
| pub fn read_page_indexes_sized<R: ChunkReader>( | ||
| /// Unlike [`Self::read_page_indexes`], this returns the newly decoded [`PageIndex`] separately; | ||
| /// any page index already attached to the returned metadata is unchanged. This is useful for | ||
| /// populating an external page index cache. The configured page index policies and masks |
|
Thanks @alamb!
Yes, I find this unfortunate as well. FWIW I've embarked on a Bloom filter provider akin to the page index provider, and I'm trying to not repeat past mistakes. Hopefully when that's complete, we can apply lessons learned to the page indexes (when we can make breaking changes ofc). |
|
Looks good to me -- let's 🚢 🇮🇹 |
|
woho! |
Which issue does this PR close?
Rationale for this change
The current APIs for loading the page indexes all modify the
ParquetMetaData, installing newly parsed page indexes viaset_page_index. Users wishing to create a page index cache can currently fetch and decode indexes themselves, but it would be nice to be able to simply parse and return the requested page indexes without installing them in the metadata for return.What changes are included in this PR?
Adds a new
try_decode_page_indexfunction toParquetMetaDataPushDecoder, as well as helper wrapper functions toParquetMetaDataReader.Note that the required
ParquetMetaDatais consumed in this process, but #11297 should make this more palatable.Are these changes tested?
Yes, includes new tests of the added APIs
Are there any significant user-facing changes?
No changes to existing public APIs, adds new functions to the public API.
AI Disclaimer
Codex was used in the creation of this PR, but I am responsible for the changes.