You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support reporting heap size of custom PageIndexProvider #11028
We added the PageIndexProvider trait so users can supply custom page index representations. However there is no way for PageIndexProvider to report their memory usage in ParquetMetaData::memory_size
Since one of the main motivations for the provider trait is more efficient caching of parquet metadata, callers need accurate memory accounting for custom providers as well.
See here for some prior attempts at exposing HeapSize publicly:
Add a way for the PageIndexProvider trait to report its heap size so ParquetMetaData::memory_size can include it. Ideally implementors could re-use the HeapSize machinery the parquet crate already implements.
Describe alternatives you've considered
Add a required (or defaulted-to-zero) heap_size(&self) -> usize method directly on PageIndexProvider, without exposing the HeapSize trait publicly.
Make the HeapSize trait public (per the prior attempts above) and require PageIndexProvider: HeapSize.
I'd land Option A on its own. PageIndexProvider already defaults is_complete, so a defaulted heap_size(&self) -> usize { 0 } is additive for existing implementors. memory_size can replace downcast_ref::<PageIndex>() at parquet/src/file/metadata/mod.rs:242 with the trait call. Built-in PageIndex would report the same number using its existing private HeapSize impl.
Both prior attempts died on scope rather than merit: #9138 was a 2k-line new crate that never got reviewed, and #9842 stalled on where the trait should live. Public HeapSize still seems worth doing, but it doesn't have to gate accounting for custom providers. Does leaving public HeapSize for a follow-up fit what you had in mind?
The issue is that calculating the heap size is quite complex and easy to do wrong. We have a number of implementations of HeapSize in the parquet crate for various containers the would be of great use to authors of custom providers. How to get them access to these is the real issue. Forcing each provider implementation to reinvent this wheel is not tenable. So I'm personally not a fan of simply adding heap_size to the provider trait.
@etseidl, would you be open to making the existing HeapSize trait and its container implementations public within the parquet crate? That would let custom PageIndexProvider implementations reuse the existing accounting code for their fields. I’d check the API with a small example outside the crate before updating #11057. Would that be a reasonable scope for a first patch?
@zhangxinyao88 thanks for holding off on this for so long. I think the page index APIs have settled enough to revive this discussion. I'll admit to not having given this a whole lot of thought, but I'd probably prefer the approach in #9842.
One nuance I think we'd want to consider, however, is how to treat cached index entries? Do we count the cost of the entire cache in total_memory, or do we ignore cached values and only count the extra heap to contain the cached copies? I don't work in Datafusion much, so I'm not in a position really to say. @alamb@adriangb any opinions here?
Activity
bharadwaj-pendyala commentedon Sep 10, 2026
I'd land Option A on its own.
PageIndexProvideralready defaultsis_complete, so a defaultedheap_size(&self) -> usize { 0 }is additive for existing implementors.memory_sizecan replacedowncast_ref::<PageIndex>()atparquet/src/file/metadata/mod.rs:242with the trait call. Built-inPageIndexwould report the same number using its existing privateHeapSizeimpl.Both prior attempts died on scope rather than merit: #9138 was a 2k-line new crate that never got reviewed, and #9842 stalled on where the trait should live. Public
HeapSizestill seems worth doing, but it doesn't have to gate accounting for custom providers. Does leaving publicHeapSizefor a follow-up fit what you had in mind?etseidl commentedon Sep 11, 2026
The issue is that calculating the heap size is quite complex and easy to do wrong. We have a number of implementations of
HeapSizein the parquet crate for various containers the would be of great use to authors of custom providers. How to get them access to these is the real issue. Forcing each provider implementation to reinvent this wheel is not tenable. So I'm personally not a fan of simply addingheap_sizeto the provider trait.zhangxinyao88 commentedon Nov 17, 2026
@etseidl, would you be open to making the existing
HeapSizetrait and its container implementations public within theparquetcrate? That would let customPageIndexProviderimplementations reuse the existing accounting code for their fields. I’d check the API with a small example outside the crate before updating #11057. Would that be a reasonable scope for a first patch?etseidl commentedon Nov 17, 2026
@zhangxinyao88 thanks for holding off on this for so long. I think the page index APIs have settled enough to revive this discussion. I'll admit to not having given this a whole lot of thought, but I'd probably prefer the approach in #9842.
One nuance I think we'd want to consider, however, is how to treat cached index entries? Do we count the cost of the entire cache in
total_memory, or do we ignore cached values and only count the extra heap to contain the cached copies? I don't work in Datafusion much, so I'm not in a position really to say. @alamb @adriangb any opinions here?alamb commentedon Nov 17, 2026
I agree making the HeapSize trait public is probably my preference too
To make things more complicated, the Rust language just stabalized the allocation API, and if we worked that through all the low level Arrow APIs https://doc.rust-lang.org/std/alloc/trait.Allocator.html
And then Vecs etc get that allocator https://doc.rust-lang.org/std/vec/struct.Vec.html
So it may be better to think about how to plumb that through 🤔