Skip to content

[Parquet][C++] ipc::SerializeSchema() allocates from the default memory pool instead of the pool it is given #55623

Open

Description

@davidnich

Describe the bug, including details regarding any error messages, version, and platform.

arrow::ipc::SerializeSchema(const Schema& schema, MemoryPool* pool) allocates from default_memory_pool() even though a pool is passed explicitly.

pool is given only to the output BufferOutputStream. The writer is built with IpcWriteOptions::Defaults(), whose memory_pool is default_memory_pool(), so the flatbuffer schema message (WriteFBMessage → WriteFlatbufferBuilder → AllocateBuffer) is allocated from the default pool:

Result<std::shared_ptr<Buffer>> SerializeSchema(const Schema& schema, MemoryPool* pool) {
  ARROW_ASSIGN_OR_RAISE(auto stream, io::BufferOutputStream::Create(1024, pool));

  auto options = IpcWriteOptions::Defaults();   // memory_pool = default_memory_pool()
  ...

(cpp/src/arrow/ipc/writer.cc, unchanged on main at ab5b422)

Parquet calls SerializeSchema() from GetSchemaMetadata() (cpp/src/parquet/arrow/writer.cc) in FileWriter::Open() when ArrowWriterProperties::store_schema() is set, so a Parquet writer opened with an explicit pool also allocates from the default pool.

Found while using a custom memory pool: allocations escape the pool, which defeats pool accounting and limits. Where the default pool is mimalloc and the program runs under valgrind, this also shows up as Invalid free() reports, since valgrind intercepts libmimalloc's free but not mi_malloc_aligned.

Reproducer (Arrow 23.0.1, Ubuntu packages, x86_64; default pool mimalloc):

// g++ -std=c++17 ser_pool.cc $(pkg-config --cflags --libs arrow)
#include <arrow/api.h>
#include <arrow/ipc/api.h>
#include <iostream>

int main() {
    auto schema = arrow::schema({arrow::field("a", arrow::int64()), arrow::field("b", arrow::utf8())});
    arrow::ProxyMemoryPool pool(arrow::system_memory_pool());
    arrow::MemoryPool* def = arrow::default_memory_pool();
    int64_t before = def->total_bytes_allocated();
    auto result = arrow::ipc::SerializeSchema(*schema, &pool);
    if (!result.ok()) { std::cerr << result.status().ToString() << std::endl; return 2; }
    int64_t def_delta = def->total_bytes_allocated() - before;
    std::cout << "passed pool allocated " << pool.total_bytes_allocated()
              << " bytes; default pool allocated " << def_delta << " bytes" << std::endl;
    return def_delta ? 1 : 0;
}

Output:

passed pool allocated 1024 bytes; default pool allocated 192 bytes

Expected: the default pool allocates 0 bytes.

Suggested fix: set options.memory_pool = pool; in SerializeSchema().

Component(s)

C++, Parquet

Activity

  1. changed the title [-][C++] ipc::SerializeSchema() allocates from the default memory pool instead of the pool it is given[/-] [+][Parquet][C++] ipc::SerializeSchema() allocates from the default memory pool instead of the pool it is given[/+] on Nov 15, 2026
  2. davidnich commented on Nov 16, 2026

    @davidnich
    Author

    The same pattern exists in two sibling functions in cpp/src/arrow/ipc/writer.cc, and the PR fixes them too:

    • GetTensorMessage(tensor, pool) uses pool only to make a non-contiguous tensor contiguous; the tensor message's flatbuffer is allocated from the default pool.
    • GetSparseTensorPayload(sparse_tensor, pool, out) (and so GetSparseTensorMessage(sparse_tensor, pool)) never uses pool.

    No other IPC function that takes a MemoryPool* allocates from the default pool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions