Skip to content

[Discussion] Adding BufferManager and ThreadPool utilities #50

Closed

Description

@zhjwpku
  • BufferManager: A centralized mechanism to manage memory buffers efficiently, particularly in scenarios involving high-frequency data processing, streaming, or caching, also control the memory usage and deal with user provided memory.
  • ThreadPool: A reusable pool of threads to manage task execution and improve concurrency without overwhelming system resources.

Activity

  1. lidavidm commented on Mar 25, 2025

    @lidavidm
    Member

    For ThreadPool, is there an opportunity to align with future C++ APIs (std::execution) and play well with the ecosystem/avoid having to design our own abstractions?

  2. wgtmac commented on Mar 25, 2025

    @wgtmac
    Member

    If we have libiceberg-bundle, we can simply reuse Arrow for all of these. We don't need this in the libiceberg, right?

  3. lidavidm commented on Mar 25, 2025

    @lidavidm
    Member

    Well, Arrow's Executor/ThreadPool is technically an internal API so that may not be ideal

  4. zhjwpku commented on Oct 23, 2026

    @zhjwpku
    MemberAuthor

    Closed as #687 added executor pool support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions