Repository navigation
core/memory: downstream allocator fast path improvements - #325
Merged
travisdowns merged 7 commits intoOct 30, 2026
Merged
Conversation
test_allocation_function() measures foreign malloc/free counters around a single allocation in a fresh thread, and requires the two to balance. The first allocation in a thread is not just one allocation, though: it lazily initializes thread_local allocator state, and registering the destructor of such an object itself allocates - a foreign malloc which is counted, but whose matching free only happens at thread exit, long after the measurement window. Today this happens to be hidden, because some earlier lazy initialization already pulled those registrations in before the window opened. That is not something the test should depend on, so allocate and free once up front, before the "before" snapshot is taken. The pointer is passed through a volatile variable, as gcc otherwise elides the malloc()/free() pair entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit f1f4fab)
sampler's constructor seeded a std::mt19937_64 from a std::random_device, which made every object embedding a sampler require dynamic initialization - including the thread_local cpu_pages in the allocator, where a TLS initialization guard is not affordable. Give sampler a constexpr default constructor which just spells out the "sampling disabled" state it used to compute at run time, and create the random engine on first use instead, which for most processes is never. This also makes sampler trivially destructible, and drops ~2.5KB from the TLS block of every thread when sampling is off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 91d23f7)
Pure code motion: move small_pool's constructor next to the definitions of idx_to_size()/size_to_idx(), before small_pool_array. No functional change. The next patch makes the constructor constexpr, and small_pool_array constant-evaluates it, which requires the definition to precede it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 9f35a2b)
small_pool_array constructed its pools with a loop of placement new calls in a union constructor, so the thread_local cpu_pages holding two of them needed dynamic initialization - and therefore a TLS initialization guard on every access. The pool parameters are derived from the pool index alone, so make the constructor constexpr and build the array with a pack expansion over the indices instead. The array is now baked into the TLS image at compile time, and no longer needs a union to suppress destruction: drop ~small_pool(), which was only ever reached for pools outside these arrays - of which there are none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 574e146)
The thread_local cpu_mem needed dynamic initialization, so every access to it had to be preceded by a check of a TLS guard variable. The allocator fast path cannot pay for that, so it went through cpu_mem_ptr instead - a second thread_local, set by init_cpu_mem(), costing an extra TLS load and a dereference on every allocation and free. Make cpu_pages constant-initializable and trivially destructible instead, so that get_cpu_mem() can return cpu_mem directly and the compiler emits neither a guard check nor a __cxa_thread_atexit() registration: - the members which cannot be constant-initialized (the reclaim hook, the reclaimer list and the allocation site set) move into manual_lifetime, and are constructed by initialize() once the shard can allocate. Like the union which held the allocation sites before, manual_lifetime never destroys what it holds. - ~cpu_pages() only cleared the shard's live_cpus flag; that moves to a separate thread_local whose destructor does the same, and which initialize() touches to arm. - the remaining members get explicit initializers. TLS usage of memory.cc drops from 12836 to 7804 bytes per thread: the small pool arrays move from .tbss to .tdata, but sampler no longer drags in a mt19937_64. Verified with gcc 16 and clang 22, PIC and non-PIC, with and without SEASTAR_HEAPPROF, that no TLS initialization guard is referenced from the allocation and free paths any more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit e8d7b5c)
Now that cpu_mem is constant-initialized, get_cpu_mem() is just a way to spell cpu_mem which asks the reader to go look up why the indirection exists. Use cpu_mem directly and move the note about initialization to its definition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 329cf92)
Every allocation started by loading the thread_local is_reactor_thread and branching on it, so that threads which don't use the seastar allocator can be routed to the system one. That check is only interesting once we are about to touch the shard's memory, and it is paid for by the overwhelmingly common case of a reactor thread allocating from a small pool free list. Take mimalloc's approach instead: let the fast path speak only to the small pool free list, and let an empty free list stand for "this thread has nothing prepared for it". A thread which is not a reactor thread has never allocated from the seastar allocator, so its cpu_mem - which is constant-initialized - has empty free lists, the pop fails, and we land in the slow path, which checks is_reactor_thread and hands the allocation to the system allocator as before. small_pool::try_allocate() is small_pool::allocate() without the add_more_objects() refill, so the fast path can fail instead of touching uninitialized shard state, and allocate()/allocate_aligned() are split into an always-inlined fast path and a shared out-of-line slow path. The fast path now calls maybe_sample() unconditionally. Foreign threads never enable heap profiling, so their sampler is constant-initialized to disabled and always answers false. Verified with gcc that malloc(), aligned_alloc() and friends no longer reference the is_reactor_thread TLS, with and without SEASTAR_HEAPPROF. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 78f64e2)
StephanDollberg
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of upstream scylladb#3660 (merge 7ebfaa1, all six commits) and scylladb#3661 (78f64e2), applied cleanly on v26.3.x.
These make the allocator fast path cheaper:
cpu_mem(the per-shardcpu_pages), its small pool arrays and the heap profilingsamplerconstant-initializable and trivially destructible. An access tocpu_memthen needs no TLS guard check, so thecpu_mem_ptrindirection (an extra TLS load and a dereference on every allocation and free) andget_cpu_mem()go away. Per-thread TLS usage of memory.cc also drops.is_reactor_threadcheck from the start of every allocation. The fast path only pops the small pool free list; a non-reactor thread has empty (constant-initialized) free lists, so it falls into the slow path, which checksis_reactor_threadand hands the allocation to the system allocator as before.The fork's
scoped_system_alloc_fallbackhooks are inallocate_slowpath()/allocate_aligned_slowpath(), which these commits do not change, so they apply as-is.Testing: alloc_test, allocator_test, cpu_profiler_alloc_test and thread_test pass in dev mode (with heap profiling).
allocator_perf
@seastar//:allocator_perfbuilt in redpanda with--config=release(seastar compiled with heap profiling, the redpanda default) against v26.3.x tip (ce8497e) and against this branch. One shard pinned to one CPU, address randomization off (setarch -R), 4 alternating rounds per build of 5 x 1s runs; median per op:(
small_many*are thesingle_alloc_and_free_small_many*tests.) The new build is faster in every round of every test.With address randomization on, cycles for the new build were bimodal in the single-op tests: in some process runs they were 3-8 cycles slower than the baseline (e.g. malloc_free 26-27 vs 20), while instruction counts were unchanged. That goes away with a fixed layout, so it is a per-process memory layout effect in this microbenchmark.