You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While updating DataFusion to arrow 60 I found the new Metadata struct introduced in #10075 does not expose its underlying reference-counted map. This hurts in two places:
Memory accounting: DataFusion tracks heap usage of plans/schemas and dedupes allocations shared via Arc by pointer. Since Metadata hides its allocation, DataFusion can only approximate its size and double-counts clones that share the same map:
implDFHeapSizeforMetadata{fnheap_size(&self,ctx:&mutDFHeapSizeCtx) -> usize{// `Metadata` does not expose its underlying reference-counted map, so// this approximates the `BTreeMap` entries' sizes and cannot dedupe// instances that share the same allocation.self.iter().map(|(k, v)| {size_of::<(String,String)>() + k.heap_size(ctx) + v.heap_size(ctx)}).sum()}}
Zero-copy conversion to wrapper types: DataFusion's FieldMetadata also stores an Arc<BTreeMap<String, String>>. The inbound direction is cheap (Metadata: From<Arc<BTreeMap<String, String>>>), but the outbound direction has to go through BTreeMap, which clones the map whenever the Arc is shared:
implFrom<Metadata>forFieldMetadata{fnfrom(value:Metadata) -> Self{// From<Metadata> for BTreeMap clones the map when the Arc is sharedSelf::new(value.into())}}
Describe the solution you'd like
Accessors that expose the shared map, for example Metadata::as_arc(&self) and Metadata::into_arc(self). Then the conversion is always cheap:
Activity
[-]Expose `Metadata`'s inner `Arc<BTreeMap<String, String>>` (e.g. `as_arc` / `into_arc`)[/-][+]Expose `Metadata`'s inner `Arc` (e.g. `as_arc` / `into_arc`)[/+]Add `Metadata` `Arc` helper methods (apache#10693)
Add `Metadata` `Arc` helper methods (apache#10693)
Add `Metadata` `Arc` helper methods (apache#10693)
Add `Metadata` `Arc` helper methods (apache#10693)
Add `Metadata` `Arc` helper methods (apache#10693)
Add `Metadata` `Arc` helper methods (apache#10693)
Add `Metadata` `Arc` helper methods (apache#10693)
Add `Metadata` `Arc` helper methods (apache#10693)