Skip to content

core/memory: downstream allocator fast path improvements - #325

Merged
travisdowns merged 7 commits into
redpanda-data:v26.3.xfrom
travisdowns:td-memory-avi-v26.3.x
Oct 30, 2026
Merged

travisdowns merged 7 commits into
redpanda-data:v26.3.xfrom
travisdowns:td-memory-avi-v26.3.x

Conversation

@travisdowns

@travisdowns travisdowns commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Cherry-pick of upstream scylladb#3660 (merge 7ebfaa1, all six commits) and scylladb#3661 (78f64e2), applied cleanly on v26.3.x.

These make the allocator fast path cheaper:

  • memory: modernize cpu_mem initialization scylladb/seastar#3660 makes cpu_mem (the per-shard cpu_pages), its small pool arrays and the heap profiling sampler constant-initializable and trivially destructible. An access to cpu_mem then needs no TLS guard check, so the cpu_mem_ptr indirection (an extra TLS load and a dereference on every allocation and free) and get_cpu_mem() go away. Per-thread TLS usage of memory.cc also drops.
  • core/memory: don't check is_reactor_thread on the allocation fast path scylladb/seastar#3661 removes the is_reactor_thread check from the start of every allocation. The fast path only pops the small pool free list; a non-reactor thread has empty (constant-initialized) free lists, so it falls into the slow path, which checks is_reactor_thread and hands the allocation to the system allocator as before.

The fork's scoped_system_alloc_fallback hooks are in allocate_slowpath() / allocate_aligned_slowpath(), which these commits do not change, so they apply as-is.

Testing: alloc_test, allocator_test, cpu_profiler_alloc_test and thread_test pass in dev mode (with heap profiling).

allocator_perf

@seastar//:allocator_perf built in redpanda with --config=release (seastar compiled with heap profiling, the redpanda default) against v26.3.x tip (ce8497e) and against this branch. One shard pinned to one CPU, address randomization off (setarch -R), 4 alternating rounds per build of 5 x 1s runs; median per op:

                                   sampling off                      sampling on (--heapprof=3000037, redpanda's rate)
                              instructions    cycles               instructions    cycles
                             before  after  before after          before  after  before after
malloc_only                    49.3   43.3    10.6   9.3 (-12%)     49.4   43.4    10.7   9.4 (-12%)
free_only                      38.3   37.3     9.3   8.3 (-11%)     38.3   37.3     9.4   9.1  (-3%)
malloc_free                    87.7   80.7    20.0  17.9 (-10%)     87.8   80.8    20.1  18.9  (-6%)
op_new_only                    50.3   47.3    10.7  10.2  (-5%)     50.4   47.4    10.7  10.2  (-5%)
op_delete_only                 38.3   37.3     9.5   8.4 (-12%)     38.3   37.3     9.6   8.3 (-13%)
op_new_delete                  88.7   84.7    20.1  18.5  (-8%)     88.8   84.8    20.3  18.5  (-9%)
array_new_delete               91.7   87.7    20.0  18.7  (-6%)     91.8   87.8    20.0  19.2  (-4%)
alloc_free_large              436.0  433.0    95.0  92.7  (-2%)    825.6  818.8   196.1 189.9  (-3%)
small_many                    826.0  756.0   173.3 161.9  (-7%)    827.2  757.2   175.0 160.3  (-8%)
small_many_cross_page         416.0  381.0    84.1  78.1  (-7%)    479.2  443.4   100.4  94.3  (-6%)
small_many_cross_page_more  15303.0 14713.0 3950.5 3651.5 (-8%) 16656.0 15975.9 4714.0 4563.2 (-3%)

(small_many* are the single_alloc_and_free_small_many* tests.) The new build is faster in every round of every test.

With address randomization on, cycles for the new build were bimodal in the single-op tests: in some process runs they were 3-8 cycles slower than the baseline (e.g. malloc_free 26-27 vs 20), while instruction counts were unchanged. That goes away with a fixed layout, so it is a per-process memory layout effect in this microbenchmark.

avikivity and others added 7 commits October 29, 2026 20:09
test_allocation_function() measures foreign malloc/free counters around a
single allocation in a fresh thread, and requires the two to balance.

The first allocation in a thread is not just one allocation, though: it
lazily initializes thread_local allocator state, and registering the
destructor of such an object itself allocates - a foreign malloc which is
counted, but whose matching free only happens at thread exit, long after
the measurement window.

Today this happens to be hidden, because some earlier lazy initialization
already pulled those registrations in before the window opened. That is
not something the test should depend on, so allocate and free once up
front, before the "before" snapshot is taken.

The pointer is passed through a volatile variable, as gcc otherwise
elides the malloc()/free() pair entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f1f4fab)
sampler's constructor seeded a std::mt19937_64 from a std::random_device,
which made every object embedding a sampler require dynamic
initialization - including the thread_local cpu_pages in the allocator,
where a TLS initialization guard is not affordable.

Give sampler a constexpr default constructor which just spells out the
"sampling disabled" state it used to compute at run time, and create the
random engine on first use instead, which for most processes is never.
This also makes sampler trivially destructible, and drops ~2.5KB from
the TLS block of every thread when sampling is off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 91d23f7)
Pure code motion: move small_pool's constructor next to the definitions
of idx_to_size()/size_to_idx(), before small_pool_array. No functional
change.

The next patch makes the constructor constexpr, and small_pool_array
constant-evaluates it, which requires the definition to precede it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 9f35a2b)
small_pool_array constructed its pools with a loop of placement new calls
in a union constructor, so the thread_local cpu_pages holding two of them
needed dynamic initialization - and therefore a TLS initialization guard
on every access.

The pool parameters are derived from the pool index alone, so make the
constructor constexpr and build the array with a pack expansion over the
indices instead. The array is now baked into the TLS image at compile
time, and no longer needs a union to suppress destruction: drop
~small_pool(), which was only ever reached for pools outside these arrays
- of which there are none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 574e146)
The thread_local cpu_mem needed dynamic initialization, so every access
to it had to be preceded by a check of a TLS guard variable. The
allocator fast path cannot pay for that, so it went through cpu_mem_ptr
instead - a second thread_local, set by init_cpu_mem(), costing an extra
TLS load and a dereference on every allocation and free.

Make cpu_pages constant-initializable and trivially destructible instead,
so that get_cpu_mem() can return cpu_mem directly and the compiler emits
neither a guard check nor a __cxa_thread_atexit() registration:

 - the members which cannot be constant-initialized (the reclaim hook,
   the reclaimer list and the allocation site set) move into
   manual_lifetime, and are constructed by initialize() once the shard
   can allocate. Like the union which held the allocation sites before,
   manual_lifetime never destroys what it holds.

 - ~cpu_pages() only cleared the shard's live_cpus flag; that moves to a
   separate thread_local whose destructor does the same, and which
   initialize() touches to arm.

 - the remaining members get explicit initializers.

TLS usage of memory.cc drops from 12836 to 7804 bytes per thread: the
small pool arrays move from .tbss to .tdata, but sampler no longer drags
in a mt19937_64.

Verified with gcc 16 and clang 22, PIC and non-PIC, with and without
SEASTAR_HEAPPROF, that no TLS initialization guard is referenced from the
allocation and free paths any more.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e8d7b5c)
Now that cpu_mem is constant-initialized, get_cpu_mem() is just a way to
spell cpu_mem which asks the reader to go look up why the indirection
exists. Use cpu_mem directly and move the note about initialization to
its definition.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 329cf92)
Every allocation started by loading the thread_local is_reactor_thread
and branching on it, so that threads which don't use the seastar
allocator can be routed to the system one. That check is only interesting
once we are about to touch the shard's memory, and it is paid for by the
overwhelmingly common case of a reactor thread allocating from a small
pool free list.

Take mimalloc's approach instead: let the fast path speak only to the
small pool free list, and let an empty free list stand for "this thread
has nothing prepared for it". A thread which is not a reactor thread has
never allocated from the seastar allocator, so its cpu_mem - which is
constant-initialized - has empty free lists, the pop fails, and we land
in the slow path, which checks is_reactor_thread and hands the allocation
to the system allocator as before.

small_pool::try_allocate() is small_pool::allocate() without the
add_more_objects() refill, so the fast path can fail instead of touching
uninitialized shard state, and allocate()/allocate_aligned() are split
into an always-inlined fast path and a shared out-of-line slow path.

The fast path now calls maybe_sample() unconditionally. Foreign threads
never enable heap profiling, so their sampler is constant-initialized to
disabled and always answers false.

Verified with gcc that malloc(), aligned_alloc() and friends no longer
reference the is_reactor_thread TLS, with and without SEASTAR_HEAPPROF.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 78f64e2)
Copilot AI balanced review requested due to automatic review settings October 30, 2026 12:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@travisdowns
travisdowns merged commit bc1eb32 into redpanda-data:v26.3.x Oct 30, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants