Repository navigation
Conversation
…functions These functions take a MemoryPool but built their IpcWriteOptions with the default pool, so the flatbuffer metadata was allocated from default_memory_pool(): - SerializeSchema(schema, pool): the pool was passed only to the output stream. - GetTensorMessage(tensor, pool): the pool was used only to make a non-contiguous tensor contiguous. - GetSparseTensorPayload(sparse_tensor, pool, out), and so GetSparseTensorMessage(sparse_tensor, pool): the pool was not used. Set IpcWriteOptions::memory_pool to the given pool in each, and add a test per function that checks that the default pool is not used.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Some IPC serialization functions take a
MemoryPool* poolbut build theirIpcWriteOptionswith the default pool, so the flatbuffer metadata is allocated fromdefault_memory_pool()instead of the pool the caller gave. This defeats pool accounting and limits. Parquet hits it throughSerializeSchema(), whichGetSchemaMetadata()calls inFileWriter::Open()whenArrowWriterProperties::store_schema()is set.The affected functions are:
ipc::SerializeSchema(schema, pool):poolwas passed only to the outputBufferOutputStream; the writer usedIpcWriteOptions::Defaults().ipc::GetTensorMessage(tensor, pool):poolwas used only to make a non-contiguous tensor contiguous; the tensor message used a default-constructedIpcWriteOptions.ipc::GetSparseTensorPayload(sparse_tensor, pool, out), and soipc::GetSparseTensorMessage(sparse_tensor, pool):poolwas not used;SparseTensorSerializerusedIpcWriteOptions::Defaults().What changes are included in this PR?
IpcWriteOptions::memory_poolto the given pool inSerializeSchema()andGetTensorMessage().SparseTensorSerializera pool argument.GetSparseTensorPayload()passes itspool, andWriteSparseTensor(), which takes no pool, passesdefault_memory_pool(), as before.Are these changes tested?
Yes, with one new test per function:
TestSchemaMetadata.SerializeSchemaUsesGivenMemoryPoolincpp/src/arrow/ipc/read_write_test.ccTestTensor.GetTensorMessageUsesGivenMemoryPoolincpp/src/arrow/ipc/tensor_test.cc, for a contiguous and a non-contiguous tensorTestSparseTensor.GetSparseTensorPayloadUsesGivenMemoryPoolincpp/src/arrow/ipc/tensor_test.cc, forGetSparseTensorPayload()andGetSparseTensorMessage()Each test serializes with a pool from
MemoryPool::CreateDefault(), which has its own statistics and so is distinct fromdefault_memory_pool()whatever allocator backs both. It checks thatdefault_memory_pool()->total_bytes_allocated()is unchanged, that the given pool was used, and that the result reads back equal (ReadSchema(),ReadTensor(),internal::ReadSparseTensorPayload()). ForGetSparseTensorMessage(), the test compares the message metadata with the payload's. Each test fails without its fix and passes with it.Are there any user-facing changes?
No API changes; these functions now allocate only from the pool they are given.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: