Repository navigation
Proposal: Pluggable allocator API #509
Replies: 2 comments
|
To say it explicitly and without agentic intervention, my goal was to fix some performance and correctness issues and improve expressivity while preserving existing APIs and behaviors for backwards-compatibility. PR #504 is neither complete nor would it be submitted as a single giant PR; it's more-or-less an existence proof that the proposed API can work. @kronic @teo-tsirpanis @adamreeve @cmettler and everyone else, I'd love to hear your feedback. |
|
Thanks for writing this up. I tried PR #504 (head Context. The host is a DuckDB extension that runs CoreCLR in process and exchanges Arrow with DuckDB over the C stream interface in both directions: DuckDB hands us batches (scan arguments, filter constants, Default path: neutralWith no allocator passed, our tests give identical results on the PR and on 23.0.0, with the same assertion count per suite: 101 self-contained suites (mostly Delta/Parquet, no external services) and 92 that run against SQL Server, about 150k assertions in total. That needed one change on our side, and it was for #483 rather than #504 (see below); We also ran the self-contained suites on both versions, and the SQL Server suites on 23.0.0, with a background thread that forced a blocking compacting gen2 plus Opt-in: what it buysFunctionally, the SQL Server suites pass with identical counts with each of For the numbers, the test was a full scan of 1M rows and 7 types, with the row count of each batch known up front, so the builders could be created with that capacity.
We passed the allocator explicitly to every column builder: all seven types, including string, decimal and boolean, which we sized with The wall-time difference is noise: on our path the database client and per-value boxing dominate. Our reader also appends one value at a time through I also ran 8 concurrent scans in one process (8M rows):
Wall time did not move in either case. So for us Q1 and Q3: disposal is load-bearing with
|
| live allocations at the end | maximum | |
|---|---|---|
| with #483 | 11 (the last scan's columns, still in flight) | 62 |
| #483 reverted | 1,514, growing with every scan | 1,514 |
| #483 reverted, producer disposes each batch after export | 11 | 62 |
The public single-batch ExportRecordBatch deliberately leaves the batch to the caller, as #483 says, and that is right: callers can dispose right after the call, and code that keeps using a batch after exporting it would break otherwise. With CallerOwned, though, its docs should say so explicitly, because a forgotten Dispose there is a permanent leak rather than a late free.
Smaller points
PoisonDisposedBuffersfound no use-after-dispose on our reader path, but the pooled allocator only covered the batches we build ourselves, which is a small part of what our code reads.- Mixing allocators within one batch works, because ownership is per buffer. We have batches that combine library-decoded columns with a column we built ourselves; they export and release cleanly in our tests.
Uh oh!
There was an error while loading. Please reload this page.
Opt-in pluggable allocators: proposed public API changes
This summarizes the public API changes in the draft PR #504, to get feedback on the API before the work is split into smaller PRs. Implementation details are left out.
The change is purely additive. I compared the public and protected API of
Apache.Arrow,Apache.Arrow.Flight,Apache.Arrow.Flight.SqlandApache.Arrow.Operationswithmain(2fc782e). Nothing is removed or changed, and 127 members are added. Existing code compiles and behaves as before. Passing no allocator, ornull, gives today's behaviour, and the new behaviour applies only when you pass one of the new allocators.Goals
Allocators (
Apache.Arrow.Memory)AllocationLifetime(new enum)This enum states what happens to a buffer that is never disposed. Whatever the lifetime, a span must never be used after its buffer is disposed.
GarbageCollectedCallerOwnedFinalizedMemoryAllocator(new virtual members)Lifetime: defaults toFinalized, so existing third-party allocators describe themselves conservatively without any change.NativeMemoryAllocatoroverrides it to returnFinalized.Resize: returns a buffer of the new length with the old contents, and releases the old one. Builders grow through it. The default implementation allocates, copies and releases; an allocator can override it to grow in place.MemoryAllocator.Defaultis unchanged and can't be replaced. To use a different allocator, you pass it to whatever allocates.New allocators
ManagedMemoryAllocator()GarbageCollectedbyte[]with no pooling, and disposing a buffer releases nothing. This is the safest choice: spans stay valid even after disposal. Aligned only to the pointer size.PooledManagedAllocator()/(PooledManagedAllocatorOptions)GarbageCollectedResize.PooledManagedAllocatorOptionsMaxPooledLength(default 1 MiB),MaxArraysPerSize(default 16, about 32 MiB worst case),MaxRetainedBytes, andPoisonDisposedBuffers, which fills disposed buffers with0xDEto expose use-after-dispose bugs.CallerOwnedNativeAllocator(int alignment = 64)CallerOwnedBuilders
Allocator constructors
Every array builder gains a constructor that takes the allocator for its working memory. Constructors that already shipped are kept as they are, for binary compatibility.
(…, int capacity, MemoryAllocator allocator = null). This coversInt8throughUInt64,HalfFloat,Float,Double,Date32,Date64,Bool8, the three interval types,Duration(type, …),Time32(type, …),Time64(type, …)andTimestamp(type, …). The protected base constructorPrimitiveArrayBuilder(int capacity, MemoryAllocator allocator = null)is new too.Boolean,String,LargeString,StringView,GuidandVariantwith(MemoryAllocator).Binary,LargeBinaryandBinaryViewget(IArrowType, MemoryAllocator), and the decimal builders get(DecimalNNType, MemoryAllocator).List,LargeList,ListViewandLargeListViewget(Field or IArrowType, MemoryAllocator),FixedSizeListgets(Field or IArrowType, int listSize, MemoryAllocator),Mapgets(MapType, MemoryAllocator)andTimestampWithOffsetgets(TimestampWithOffsetType, MemoryAllocator). Protected equivalents are added on theBuilderBaseclasses.ArrowBuffer.Builder<T>andArrowBuffer.BitmapBuilderget(int capacity, MemoryAllocator allocator).RecordBatch.Builder's column helpers build in the batch's allocator.The design rule is that builders take a
MemoryAllocator, and operations such as readers, writers and Flight take anArrowContext.What passing an allocator changes
null(or an existing constructor): the builder grows in managed memory, andBuildcopies the result once into the target allocator, or intoMemoryAllocator.Defaultif none is given. This is what builders do today.Build()gets no allocator or the same one, the working memory usually becomes the built buffer, with no allocation or copy. Passing a different allocator toBuildcopies instead. The builder can still be used afterwards: the next write copies its contents back into memory of its own, andClear()discards them without copying.Builders implement
IDisposableEach builder class now has
Dispose()andprotected virtual Dispose(bool):ArrowBuffer.Builder<T>andArrowBuffer.BitmapBuilderPrimitiveArrayBuilder<…>, in both its forms, andDelegatingArrayBuilder<…>BuilderBaseclasses ofBinaryArray,LargeBinaryArray,BinaryViewArrayandFixedSizeBinaryArrayBooleanArray,ListArray,LargeListArray,ListViewArray,LargeListViewArray,FixedSizeListArray,MapArray,TimestampWithOffsetArrayandVariantArrayA parent builder disposes its child builders. Arrays that have already been built aren't affected.
CallerOwnedallocator. Otherwise disposing just releases memory sooner.IArrowArrayBuilderis unchanged: addingIDisposableto the interface would break third-party implementers.Writing values directly:
GetSpan/AdvanceThese are new on
ArrowBuffer.Builder<T>andPrimitiveArrayBuilder<T, TArray, TBuilder>:The span is invalidated by any other call on the builder. This is similar to
IBufferWriter<T>, butAdvancereturns the builder so calls can be chained.IPC and Flight:
ArrowContextArrowStreamWriter(ArrowContext, Stream, Schema, bool leaveOpen = false, IpcOptions options = null)andArrowFileWriterconstructors in the same shape. The context supplies the allocator for temporary buffers. It also supplies the compression codec factory whenIpcOptionsdoesn't specify one.FlightClient(CallInvoker, ArrowContext context = null). It is marked[ActivatorUtilitiesConstructor]so that clients created byGrpc.Net.ClientFactoryget theArrowContextregistered for dependency injection. That needs a new package dependency onMicrosoft.Extensions.DependencyInjection.Abstractions(6.0.0, matching theLogging.Abstractionsversion Flight already uses).protected FlightServer(ArrowContext context)andpublic ArrowContext Context { get; }. A server hosted through ASP.NET Core dependency injection gets the context by taking it in its own constructor.protected FlightSqlServer(ArrowContext? context, ILoggerFactory? factory).FlightServerRecordBatchStreamWriter(IServerStreamWriter<FlightData>, ArrowContext).mainbut not yet released: the readers' context support came from feat: add ICompressionCodecFactory support to Flight record batch readers #285, which isn't in v23.0.0. That coversFlightClient(ChannelBase, ArrowContext),FlightClient.Create(CallInvoker, ArrowContext)andFlightServerRecordBatchStreamReader(…, ArrowContext). Because they're unreleased, they can still be changed.Behaviour changes with the default allocator
These come without any opt-in. Most are bug fixes.
Buildfails partway.ArrowBuffer.CloneandEqualskeep the buffer alive while reading it.NativeMemoryAllocatorwith a non-default alignment now removes exactly the memory pressure it added (NativeMemoryAllocator doesn't remove all memory pressure #50).Behaviour on the opt-in path that deserves a look
GarbageCollectedandCallerOwnedlifetimes. Once that buffer is disposed, using the builder throwsObjectDisposedExceptionuntilClear()is called. The alternative was for the builder to keep a reference to the buffer, but then the commonnew Builder(allocator).Append(x).Build()would leak whenever the builder isn't disposed. WithFinalizedlifetimes the builder still keeps its contents, as it does today.Capacitycan be larger than requested, because working memory is rounded up to 64 bytes.Questions for discussion
IDisposableacceptable? Code analysis users with CA2000 enabled may start seeing warnings on builders they never disposed. ShouldIArrowArrayBuilderstay as it is?ManagedMemoryAllocator, or arrays on the pinned object heap, which are rooted by the GC, aligned and never move. Or should the default stay native, with fixes like this one targeted at specific cases?ArrowContext?AllocationLifetimeand its values,CallerOwnedNativeAllocator,PooledManagedAllocatorandManagedMemoryAllocatorthe right names?FlightClient.Create: this unreleased method is now redundant with the new constructor. Should it be removed before it ships?Microsoft.Extensions.DependencyInjection.Abstractionsacceptable?GetSpanon the converting buildersAll reactions