Skip to content

Support reporting heap size of custom PageIndexProvider #11028

Open

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We added the PageIndexProvider trait so users can supply custom page index representations. However there is no way for PageIndexProvider to report their memory usage in ParquetMetaData::memory_size

Since one of the main motivations for the provider trait is more efficient caching of parquet metadata, callers need accurate memory accounting for custom providers as well.

See here for some prior attempts at exposing HeapSize publicly:

Describe the solution you'd like

Add a way for the PageIndexProvider trait to report its heap size so ParquetMetaData::memory_size can include it. Ideally implementors could re-use the HeapSize machinery the parquet crate already implements.

Describe alternatives you've considered

  • Add a required (or defaulted-to-zero) heap_size(&self) -> usize method directly on PageIndexProvider, without exposing the HeapSize trait publicly.
  • Make the HeapSize trait public (per the prior attempts above) and require PageIndexProvider: HeapSize.

Additional context

Activity

  1. bharadwaj-pendyala commented on Sep 10, 2026

    @bharadwaj-pendyala
    Contributor

    I'd land Option A on its own. PageIndexProvider already defaults is_complete, so a defaulted heap_size(&self) -> usize { 0 } is additive for existing implementors. memory_size can replace downcast_ref::<PageIndex>() at parquet/src/file/metadata/mod.rs:242 with the trait call. Built-in PageIndex would report the same number using its existing private HeapSize impl.

    Both prior attempts died on scope rather than merit: #9138 was a 2k-line new crate that never got reviewed, and #9842 stalled on where the trait should live. Public HeapSize still seems worth doing, but it doesn't have to gate accounting for custom providers. Does leaving public HeapSize for a follow-up fit what you had in mind?

  2. etseidl commented on Sep 11, 2026

    @etseidl
    Contributor

    The issue is that calculating the heap size is quite complex and easy to do wrong. We have a number of implementations of HeapSize in the parquet crate for various containers the would be of great use to authors of custom providers. How to get them access to these is the real issue. Forcing each provider implementation to reinvent this wheel is not tenable. So I'm personally not a fan of simply adding heap_size to the provider trait.

  3. zhangxinyao88 commented on Nov 17, 2026

    @zhangxinyao88
    Contributor

    @etseidl, would you be open to making the existing HeapSize trait and its container implementations public within the parquet crate? That would let custom PageIndexProvider implementations reuse the existing accounting code for their fields. I’d check the API with a small example outside the crate before updating #11057. Would that be a reasonable scope for a first patch?

  4. etseidl commented on Nov 17, 2026

    @etseidl
    Contributor

    @zhangxinyao88 thanks for holding off on this for so long. I think the page index APIs have settled enough to revive this discussion. I'll admit to not having given this a whole lot of thought, but I'd probably prefer the approach in #9842.

    One nuance I think we'd want to consider, however, is how to treat cached index entries? Do we count the cost of the entire cache in total_memory, or do we ignore cached values and only count the extra heap to contain the cached copies? I don't work in Datafusion much, so I'm not in a position really to say. @alamb @adriangb any opinions here?

  5. alamb commented on Nov 17, 2026

    @alamb
    ContributorAuthor

    I agree making the HeapSize trait public is probably my preference too

    To make things more complicated, the Rust language just stabalized the allocation API, and if we worked that through all the low level Arrow APIs https://doc.rust-lang.org/std/alloc/trait.Allocator.html

    And then Vecs etc get that allocator https://doc.rust-lang.org/std/vec/struct.Vec.html

    So it may be better to think about how to plumb that through 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions