Repository navigation
feat(parquet): Improved back-end storage for PageIndex - #11475
Conversation
|
It's also worth pointing out the 32-bit concerns that Codex noticed and documented at the head of |
| assert_eq!(metadata.memory_size(), 8310); | ||
| #[cfg(feature = "encryption")] | ||
| assert_eq!(metadata.memory_size(), 11750); | ||
| assert_eq!(metadata.memory_size(), 8734); |
There was a problem hiding this comment.
The added Arcs lead to some bloat for a fully populated index, but down here for a small subset the reduced storage overhead wins big.
|
run benchmark metadata |
|
🤖 Benchmark starting (GKE) | trigger Target: metadata Sharding factor: 1 (1 workers). Comparing page_index_cache (58ae317) to e189ba8 (merge-base) diff Run configurationrun benchmark metadata
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: metadata Comparing page_index_cache (58ae317) to e189ba8 (merge-base) diff Run configurationrun benchmark metadata
shards: 1
DetailsPer-runner informationmetadata — shard 1/1Node: gk3-benchmark-cluster-nap-1s1srbox-e6d80935-rrt4 Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench metadataCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
performance looks good! |
yeah, I'm mildly surprised the page index parsing didn't slow down more 😅 |
a456f2c to
35b7b7f
Compare
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| //! Sparse storage keyed by row-group and column coordinates. |
There was a problem hiding this comment.
My current thinking is this can be reused for the Bloom filters, and perhaps ColumnChunkMetaData at some point.
| /// # Panics | ||
| /// | ||
| /// Panics if either dimension exceeds `i32::MAX`, the maximum collection size representable | ||
| /// by Thrift. | ||
| /// | ||
| /// Use [`try_new`](Self::try_new) for a fallible alternative. |
There was a problem hiding this comment.
This is the biggest behavior change (along with the changes to the infallible allocate_ functions below). This used to take any valid usize dimension, but now panics if a dimension exceeds what's possible for Parquet.
That said, it may have panicked before anyway if it couldn't allocate a huge matrix.
There was a problem hiding this comment.
allocating more than i32::MAX entries does sound like a bad idea to me and not a case that is important
PageIndex and PageIndexBuilderPageIndex
PageIndexPageIndex
cdff542 to
976bb35
Compare
|
run benchmark metadata |
|
🤖 Benchmark starting (GKE) | trigger Target: metadata Sharding factor: 1 (1 workers). Comparing page_index_cache (976bb35) to bad29dd (merge-base) diff Run configurationrun benchmark metadata
shards: 1
Results will be posted when all workers finish. File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Target: metadata Comparing page_index_cache (976bb35) to bad29dd (merge-base) diff Run configurationrun benchmark metadata
shards: 1
DetailsPer-runner informationmetadata — shard 1/1Node: gk3-benchmark-cluster-nap-8drekkdo-ff4449e6-kd9s Instance: c4a-highmem-16 (12 vCPU / 65 GiB) uname: BENCH_COMMAND: cargo bench --features=arrow,async,test_common,experimental,object_store --bench metadataCPU Details (lscpu)Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
Looks like perf is flat (which is good). I guess we don't have a direct measure of memory consumption per benchmark. |
Thanks for bringing this up @adriangb! While trying to answer I found a pretty big issue with the PR. Everything is roses for sparse indexes. I had my agent whip up a quick test where the populated index entries were kept fixed (10 entries), but the declared dimensions of the table increase.
Pretty good stuff. But then I asked the obvious question, what about fully populated indexes? A comparison of fully populated 1000 x 1000 showed a significant slowdown for the new form (memory use was nearly flat, but total lifecycle time goes from 500ms to 1740ms), with most of the extra time going to populating the hash map in the builder, and then sorting the entries in the freeze stage. That got me thinking that for the normal case where we're building an index from scratch, we're already doing the traversal in row-major order, and we know the exact bounds (mod missing indexes) up front. I had my agent whip up a test of going straight to the boxed slice form (skipping the builder), and was able to almost reach parity.
I'll polish that work up and push it soon. |
|
Is this something that I should put on my review queue @etseidl ? |
If you would be so kind. 🙏 otherwise I'll just keep making changes 😂 |
|
Seriously, though, it is fairly large at this point. If you'd rather concentrate on getting 60.2.0 shipped then I'm ok with waiting. |
|
Thanks -- I'll try and give it a review. BTW I highlighted the work to make metadata work better for metadata in a talk I am preparing for next weeks meetup if you are interested: https://docs.google.com/presentation/d/1Mlt4Ri9AjzwbJTB_xpfw3WU8nlDrh5i5FfF-U3cICkQ |
|
Today I need to focus on getting DataFusion 56 release blocking issues ready |
|
Sorry -- checking this one out again |
| validate_dimensions(self.num_row_groups, self.num_columns)?; | ||
| let num_entries = self.entries.len(); | ||
| let mut entries = Vec::new(); | ||
| entries.try_reserve_exact(num_entries).map_err(|err| { |
There was a problem hiding this comment.
I don't understand the rationale for checking for an allocation failure -- we don't do that in the rest of the crate, do we? I think it is fine, I just found it kind of strange
There was a problem hiding this comment.
Yeah, it's a mania my agent has picked up regarding fallibility. I'm inclined to keep it as a defense against future issues claiming some malformed input can overflow blah blah blah, but if you'd prefer we keep allocations consistent I can remove it (and the one right above it).
| } | ||
|
|
||
| #[derive(Debug, Clone, PartialEq)] | ||
| pub(crate) struct ColumnChunkEntries<T> { |
There was a problem hiding this comment.
This seems like it is very similar to the Map above, except that it has a single contiguous allocation -- what is the ratinale for the difference? Is it to reduce the in memory / heap size? If so perhaps we can document that rationale
| /// # Panics | ||
| /// | ||
| /// Panics if either dimension exceeds `i32::MAX`, the maximum collection size representable | ||
| /// by Thrift. | ||
| /// | ||
| /// Use [`try_new`](Self::try_new) for a fallible alternative. |
There was a problem hiding this comment.
allocating more than i32::MAX entries does sound like a bad idea to me and not a case that is important
| .parse_and_finish(&file) | ||
| .unwrap(); | ||
| let mut builder = metadata.into_builder(); | ||
| let row_groups = builder |
There was a problem hiding this comment.
I found this test hard to follow as the behavior to be validated (the row / columns) is interleaved with the mechnics of setting up the test
I would personally find it easier to read if there was some sort of human readable string that showed what the contents of the input row groups and the output row groups
| assert!(metadata.page_index().is_some()); | ||
| #[cfg(not(feature = "encryption"))] | ||
| assert_eq!(metadata.memory_size(), 12088); | ||
| assert_eq!(metadata.memory_size(), 9656); |
There was a problem hiding this comment.
that certainly looks a lot nicer 🎉
| assert!(metadata.page_index().is_some()); | ||
| #[cfg(not(feature = "encryption"))] | ||
| assert_eq!(metadata.memory_size(), 13929); | ||
| assert_eq!(metadata.memory_size(), 13993); |
There was a problem hiding this comment.
this is slightly larger because there is some extra overhead per entry, right?
There was a problem hiding this comment.
Yeah, there's a tradeoff here. We get better sparse memory usage, at the cost of a bit of overhead on storing all the keys inline. Thankfully that's offset some by dropping the Option wrapper on everything as well.
Down the road we could look at different storage for a fully populated index if this becomes and issue.
|
Thanks for the review @alamb. Please take one more quick look-see and let me know if you want any other changes before merging. 🙏 |
|
Thanks @alamb! On to Bloom filters! 🚀 😄
Next, on "Unsung Heroes of Parquet"... 😂 |
Well, we could make a Bay Area Parquet Meetup to tell the stories :) I am not sure I would necessarily watch that on youtube, but I would watch in real life |
Which issue does this PR close?
Rationale for this change
The current PageIndex uses a nested vector of vectors to hold the page indexes. This wastes a great deal of memory when only a few columns or row groups are desired.
What changes are included in this PR?
PageIndexBuilderfor arbitrary insertion and replacement.Are these changes tested?
Yes there are new tests added, but existing tests provide coverage as well.
Are there any significant user-facing changes?
There are no changes to the public API, but memory use for sparse indexes should drop.
Deprecates
PageIndexBuilder::put_column_indexandPageIndexBuilder::put_offset_indexwhich silently fail in certain instances.AI disclosure
Implementation, tests, and local benchmarks used coding-agent assistance, but I understand and own all the changes made.