What is the problem the feature request solves?
Comet's tracing counters cover native allocations (native_allocated, jemalloc_allocated), memory pool reservations (thread_NNN_comet_memory_reserved) and the JVM heap (jvm_heap_used). Arrow buffers allocated on the JVM are off-heap, so none of these counters see them. When a traced run shows native memory growing faster than the pools account for, there is currently no way to tell how much Arrow memory the JVM itself is holding.
A single gauge over CometArrowAllocator would not answer the question either. Comet imports batches from native over the Arrow C Data Interface, and Arrow charges an imported buffer to whichever allocator wraps it (BaseAllocator.wrapForeignAllocation calls allocateBytes). So the root allocator's total mixes memory the JVM allocated with native memory that native_allocated already counts, and the two cannot be separated after the fact.
Describe the potential solution
Hold buffers imported over the C Data Interface in a dedicated child allocator, and report two counters:
jvm_arrow_allocated: everything the root allocator accounts forjvm_arrow_imported: the imported portion, which is native memory and so is also counted by native_allocated
Because the child reserves nothing, every byte still escalates to the parent and the root keeps reporting the total, which makes the difference between the two counters the Arrow memory the JVM allocated itself.
Describe alternatives you've considered
Reporting only the root allocator's total. That is simpler, but it double counts imported native buffers against native_allocated with no way to tell how much of the overlap there is, which defeats the purpose of comparing the two.
Additional context
Comes out of an investigation into what a traced TPC-H run can and cannot account for.
What is the problem the feature request solves?
Comet's tracing counters cover native allocations (
native_allocated,jemalloc_allocated), memory pool reservations (thread_NNN_comet_memory_reserved) and the JVM heap (jvm_heap_used). Arrow buffers allocated on the JVM are off-heap, so none of these counters see them. When a traced run shows native memory growing faster than the pools account for, there is currently no way to tell how much Arrow memory the JVM itself is holding.A single gauge over
CometArrowAllocatorwould not answer the question either. Comet imports batches from native over the Arrow C Data Interface, and Arrow charges an imported buffer to whichever allocator wraps it (BaseAllocator.wrapForeignAllocationcallsallocateBytes). So the root allocator's total mixes memory the JVM allocated with native memory thatnative_allocatedalready counts, and the two cannot be separated after the fact.Describe the potential solution
Hold buffers imported over the C Data Interface in a dedicated child allocator, and report two counters:
jvm_arrow_allocated: everything the root allocator accounts forjvm_arrow_imported: the imported portion, which is native memory and so is also counted bynative_allocatedBecause the child reserves nothing, every byte still escalates to the parent and the root keeps reporting the total, which makes the difference between the two counters the Arrow memory the JVM allocated itself.
Describe alternatives you've considered
Reporting only the root allocator's total. That is simpler, but it double counts imported native buffers against
native_allocatedwith no way to tell how much of the overlap there is, which defeats the purpose of comparing the two.Additional context
Comes out of an investigation into what a traced TPC-H run can and cannot account for.