Skip to content

GH-55623: [C++] Use the given memory pool in IPC serialization functions - #55635

Open
davidnich wants to merge 1 commit into
apache:mainfrom
davidnich:GH-55623-serialize-schema-pool
Open

davidnich wants to merge 1 commit into
apache:mainfrom
davidnich:GH-55623-serialize-schema-pool

Conversation

@davidnich

Copy link
Copy Markdown

Rationale for this change

Some IPC serialization functions take a MemoryPool* pool but build their IpcWriteOptions with the default pool, so the flatbuffer metadata is allocated from default_memory_pool() instead of the pool the caller gave. This defeats pool accounting and limits. Parquet hits it through SerializeSchema(), which GetSchemaMetadata() calls in FileWriter::Open() when ArrowWriterProperties::store_schema() is set.

The affected functions are:

  • ipc::SerializeSchema(schema, pool): pool was passed only to the output BufferOutputStream; the writer used IpcWriteOptions::Defaults().
  • ipc::GetTensorMessage(tensor, pool): pool was used only to make a non-contiguous tensor contiguous; the tensor message used a default-constructed IpcWriteOptions.
  • ipc::GetSparseTensorPayload(sparse_tensor, pool, out), and so ipc::GetSparseTensorMessage(sparse_tensor, pool): pool was not used; SparseTensorSerializer used IpcWriteOptions::Defaults().

What changes are included in this PR?

  • Set IpcWriteOptions::memory_pool to the given pool in SerializeSchema() and GetTensorMessage().
  • Give the internal SparseTensorSerializer a pool argument. GetSparseTensorPayload() passes its pool, and WriteSparseTensor(), which takes no pool, passes default_memory_pool(), as before.

Are these changes tested?

Yes, with one new test per function:

  • TestSchemaMetadata.SerializeSchemaUsesGivenMemoryPool in cpp/src/arrow/ipc/read_write_test.cc
  • TestTensor.GetTensorMessageUsesGivenMemoryPool in cpp/src/arrow/ipc/tensor_test.cc, for a contiguous and a non-contiguous tensor
  • TestSparseTensor.GetSparseTensorPayloadUsesGivenMemoryPool in cpp/src/arrow/ipc/tensor_test.cc, for GetSparseTensorPayload() and GetSparseTensorMessage()

Each test serializes with a pool from MemoryPool::CreateDefault(), which has its own statistics and so is distinct from default_memory_pool() whatever allocator backs both. It checks that default_memory_pool()->total_bytes_allocated() is unchanged, that the given pool was used, and that the result reads back equal (ReadSchema(), ReadTensor(), internal::ReadSparseTensorPayload()). For GetSparseTensorMessage(), the test compares the message metadata with the payload's. Each test fails without its fix and passes with it.

Are there any user-facing changes?

No API changes; these functions now allocate only from the pool they are given.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

…functions

These functions take a MemoryPool but built their IpcWriteOptions with
the default pool, so the flatbuffer metadata was allocated from
default_memory_pool():

- SerializeSchema(schema, pool): the pool was passed only to the output
  stream.
- GetTensorMessage(tensor, pool): the pool was used only to make a
  non-contiguous tensor contiguous.
- GetSparseTensorPayload(sparse_tensor, pool, out), and so
  GetSparseTensorMessage(sparse_tensor, pool): the pool was not used.

Set IpcWriteOptions::memory_pool to the given pool in each, and add a
test per function that checks that the default pool is not used.
Copilot AI balanced review requested due to automatic review settings November 16, 2026 13:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55623 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants