LRU optimization: sharding for the LRU list and batched processing of make-young promotions - #6121
Closed
polchawa-percona wants to merge 14 commits into
Closed
LRU optimization: sharding for the LRU list and batched processing of make-young promotions#6121polchawa-percona wants to merge 14 commits into
polchawa-percona wants to merge 14 commits into
Conversation
https://perconadev.atlassian.net/browse/PS-11141 Optimization for contention on LRU list mutex: defers "make young" LRU promotions and applies them in batch. The deferred promotions are gathered in a per-buffer-pool lock-free queue, and drained in batches under a single LRU_list_mutex acquisition instead of taking the mutex on every promotion.
…f concurrent user and background drainings
… merged with LRU threads PR but let's have it for now)
Lays the compile-verified groundwork for reworking buf_pool->LRU into a list of groups of up to BUF_LRU_GROUP_SIZE pages, so that most page-level LRU operations (starting with make-young) can move off the single per-pool LRU_list_mutex onto a per-group mutex. - buf_lru_group_t: the group type (mutex, page slots, count, old flag). - New SYNC_BUF_LRU_GROUP latch level, slotted between LRU_list_mutex and the block mutex, fully wired into LatchDebug and PFS. - buf_page_t group back-pointer fields (lru_group, lru_slot), unused so far. - Buf_LRU_pages: an ordered, group-aware page iterator that currently just wraps the existing flat buf_pool->LRU, so the ~10 consumer sites outside buf0lru.cc can migrate later without learning group internals. This phase is deliberately additive only: buf_pool->LRU itself is not yet rewired to use groups. That rewrite (add/remove/make-young/eviction/ promotion-drain operating on groups) is substantial, correctness-critical work scoped as its own follow-on phase rather than rushed here. Verified: standalone compiles at each step against a Debug+WITH_DEBUG=ON build; a real mysqld+mysql smoke test (data load, updates, SHOW ENGINE INNODB STATUS, clean shutdown with no assertion/crash markers); the four LRU-relevant MTR suites plus all_persisted_variables all pass at the test-body level (only the environment's known InnoDB thread-nice/priority warnings trigger MTR's blanket warning-based fail marker). Full design and phased plan: .requirements/20260720T221500Z_lru_groups_redesign/REQUIREMENTS.md Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rewrites buf_pool_t::LRU from a flat, page-granular intrusive list into a list of buf_lru_group_t groups of up to BUF_LRU_GROUP_SIZE pages, so that group contents are protected by a per-group mutex and LRU_list_mutex only protects the links between groups. Covers add/remove, the young/old fill-group append primitives, LRU_old boundary maintenance at group granularity, make-young/make-old, the promotion-queue drain, eviction and flush batch scans, and every other page-granular LRU walk site (buf0buf.cc, buf0flu.cc, buf0buddy.cc, buf0dump.cc, btr0sea.cc, i_s.cc) via nested group/page loops. Fixes two mutex-assertion crashes found via runtime testing, caused by a hazard-pointer .get() call reachable as a for-loop increment even after a successful free had already released LRU_list_mutex. Fixes a buf_pool->LRU_old_len bookkeeping drift where a group's "old" flag could flip out from under a still-in-use fill-group pointer (via buf_LRU_old_adjust_len()/buf_LRU_old_init()/buf_LRU_remove_block()'s boundary shift), letting a page get appended on the wrong side; the append helpers now abandon a fill-group pointer whose old-ness no longer matches. Adds a UNIV_DEBUG-only oracle that recomputes LRU_old_len from scratch after every add/remove to catch any future drift immediately at the primitive that introduces it, rather than via the sampled validator. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
buf_page_t still carried its pre-grouping intrusive LRU list node, unused by any group-based code but still read by buf_LRU_insert_zip_clean() to find the LRU-order successor for its position in buf_pool->zip_clean. Since nothing ever links pages via this field anymore, that lookup always came back empty, silently always falling back to inserting at the front regardless of true position -- a missed migration site (P5) surfaced by review, not a crash (the field happens to read back zeroed). Removes the dead field and its copy-constructor entry, and simplifies buf_LRU_insert_zip_clean() to insert at the head unconditionally: with groups not tracking intra-group page order, the old LRU-order-preserving search can no longer be answered precisely, and both of zip_clean's consumers (buf_pool_validate_instance(), buf_all_freed()) iterate the whole list regardless of order, so exact position was never load-bearing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pure refactor, no behavior change: splits buf_LRU_remove_block() into buf_LRU_detach_from_group() (group-mutex-only slot vacate) and buf_LRU_remove_block_finish() (the buf_pool->LRU_list_mutex-held remainder -- byte/page counters, unzip_LRU unlink, empty-group reclaim, the "list too short" fallback, and the LRU_old adjustment), with the empty-group reclaim logic further extracted into buf_LRU_reclaim_empty_group() so it can be reused once per drain batch instead of once per page. buf_LRU_remove_block() itself now just calls detach() followed by finish(..., defer_reclaim=false), identical to before. Adds a defer_old_adjust parameter to buf_LRU_add_block_low(), defaulted to false everywhere it's currently called. This is the first of a five-phase follow-up (P7-P11) that lets the promotion drain's per-page group eviction run under only that page's group mutex instead of buf_pool->LRU_list_mutex, so unrelated eviction/ insert/old_adjust_len traffic isn't blocked for the whole drain batch -- the structural contention-reduction goal the grouped-LRU redesign was for, which P2-P6 correctly implemented but did not yet deliver (every group mutation there still takes LRU_list_mutex before ever touching a group mutex). None of the new parameters are exercised with a non-default value yet; P8-P10 build on this split to actually remove LRU_list_mutex from the drain's hot loop. Also fixes two latent bugs surfaced by review while preparing for the fast path, both real independent of it: buf_LRU_old_adjust_len()'s grow/ shrink branches rejected crossing a zero-page group as a no-op tie (a zero-page group can only exist mid-drain in the coming phases, but the bug itself is in already-shipped P2 code and is fixed unconditionally); buf_LRU_old_init() derived LRU_old_len from buf_pool->LRU_n_pages instead of the per-group sum it already computes while marking every group old. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…aders
Critical fix, found by finally running --innodb-sync-debug against a live
server (initialize-insecure alone reproduced it in seconds): the original
P1 design ordered buf_lru_group_t::mutex (SYNC_BUF_LRU_GROUP) ABOVE the
per-page block mutex (SYNC_BUF_BLOCK), on the assumption that group
operations would only ever be reached while already holding
LRU_list_mutex, before any block mutex. That's false for the most basic
call path there is: buf_LRU_block_remove_hashed() asserts the block mutex
is held on entry and calls buf_LRU_remove_block() (which takes the page's
group mutex) while still holding it -- this is InnoDB's long-standing
convention of transitioning a page's state under its own block mutex
before unlinking it from any list, unrelated to this redesign, and every
real eviction hits it. Re-levels SYNC_BUF_LRU_GROUP below SYNC_BUF_BLOCK
in sync0types.h (now the lowest of the three: LRU_list_mutex > block/zip
mutex > group mutex), updates the level's doc comment, buf_lru_group_t's
comment in buf0buf.h, and the REQUIREMENTS.md lock-ordering section to
match. buf_LRU_reclaim_empty_group()'s and buf_LRU_old_adjust_len()'s
buf_page_set_old() calls made inside a group-mutex critical section remain
correct: that function only asserts LRU_list_mutex ownership and takes no
mutex of its own.
Also lands the actual P8 work: hardens the ~5 hot-path (Tier A) readers of
group->pages[]/n_pages that currently rely on LRU_list_mutex alone to make
that array read safe -- buf_LRU_free_from_common_LRU_list(),
buf_flush_LRU_list_batch(), buf_flush_single_page_from_LRU(), the two
fill-group peek pre-checks in buf_LRU_append_to_{young,old}_fill_group(),
and buf_pool_withdraw_blocks()'s relocation loop -- so each now reads a
group's pages/n_pages under that specific group's own mutex (a brief,
separate critical section per read, released before any further per-page
processing that might recurse into the same or another group's mutex).
LRU_list_mutex is still held everywhere by everyone (no fast path exists
yet), so this is purely defensive, zero-functional-change hardening ahead
of the drain fast path landing in a later phase -- but it's what let
--innodb-sync-debug exercise every new nested acquisition and catch the
ordering bug above before anything actually depended on it.
Verified: --initialize-insecure --innodb-sync-debug (previously crashed
within seconds on the first ordinary page eviction, now clean); a live
server run under --innodb-sync-debug with a mixed compressed-table
INSERT/UPDATE/DELETE workload exercising eviction and the zip-reinsert
path; the five LRU-relevant MTR suites, both with and without
--innodb-sync-debug.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eting
Adds buf_pool_t::LRU_drain_mutex (new SYNC_BUF_LRU_DRAIN latch level,
acquired before LRU_list_mutex) and LRU_drain_active (a lock-free atomic
flag), wired through sync0types.h/sync0debug.cc/sync0sync.{h,cc}/
ha_innodb.cc the same way SYNC_BUF_LRU_GROUP was.
buf_LRU_drain_promote_queue() now acquires LRU_drain_mutex and sets
LRU_drain_active for the whole of its (still unchanged) LRU_list_mutex-held
loop; buf_LRU_validate_instance() acquires LRU_drain_mutex for its entire
check. No fast path exists yet -- the drain still does exactly what it did
before -- so this doesn't change any behavior today, but it lands the one
mutual-exclusion point a future group-mutex-only fast path will need: that
fast path will mutate group contents without LRU_list_mutex, and
LRU_drain_mutex is what will let the validator prove it never observes
that in-flight rather than merely usually not observing it.
buf_LRU_old_len_validate() cannot acquire LRU_drain_mutex -- it runs from
inside LRU_list_mutex-held code, and acquiring a higher-ordered latch there
would invert the drain's own acquisition order -- so it checks
LRU_drain_active instead and skips its recompute-vs-counter assertion
while a drain is active, a lock-free best-effort substitute.
Also brackets the administrative/rare LRU_list_mutex-only scans that can
afford to fully wait out a future in-flight drain rather than adding
per-group mutex scoping: buf_dump(), btr_drop_next_batch() (AHI drop),
i_s_innodb_fill_buffer_lru(), buf_LRU_drop_page_hash_for_tablespace(),
buf_LRU_remove_all_pages() (careful here: its own
buf_LRU_drain_promote_queue() call stays outside the bracket, since that
function acquires the same mutex itself and it isn't recursive),
buf_LRU_count_space_references(), and buf_LRU_print_instance().
One site initially planned for this same bracket turned out to need the
opposite fix instead, found while implementing it: buf_buddy_relocate()'s
LRU fallback scan (only reached from buf_pool_withdraw_blocks() during
buffer-pool resize) runs with LRU_list_mutex already held, so acquiring
LRU_drain_mutex there would acquire it *after* LRU_list_mutex -- the same
class of inversion the previous commit fixed for the group/block mutex
order. Gave it the fine-grained per-group-mutex treatment instead, matching
buf_pool_withdraw_blocks()'s own relocation loop it's nested inside of.
Verified: --initialize-insecure --innodb-sync-debug; a live server run
under --innodb-sync-debug exercising DROP TABLE, ALTER TABLE ... DISCARD
TABLESPACE, an information_schema.INNODB_BUFFER_PAGE_LRU query, a manual
buffer pool dump, and dump-at-shutdown -- the administrative paths this
phase touches; the five LRU-relevant MTR suites, with --innodb-sync-debug.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Restructures buf_LRU_drain_promote_queue() so a page's group-eviction step runs under only that page's group mutex, deferring all LRU_list_mutex-held bookkeeping (counters, empty-group reclaim, LRU_old boundary shift, re-appending drained pages into fresh young groups) to one batched pass per drain call, instead of taking LRU_list_mutex for the whole batch. Found and fixed two concurrency bugs during stress testing under --innodb-sync-debug: - buf_LRU_drain_promote_queue() published LRU_drain_active via a bare relaxed atomic store with no lock connecting it to the threads that check it, so a concurrent LRU_list_mutex holder could observe the flag as still false during a drain's real, in-flight fast pass and trip a false assertion on the deliberately transient LRU_old_len staleness. Fixed by publishing the true-transition while briefly holding LRU_list_mutex, so ordinary mutex acquire/release semantics guarantee visibility to any later LRU_list_mutex holder. - buf_flush_LRU_list_batch() (buf0flu.cc) continued its per-slot loop with a stale `group` pointer after buf_flush_page_and_try_neighbors() released and re-acquired LRU_list_mutex internally for a flush dispatch (not just an eviction) -- only a successful eviction was treated as invalidating the pointer. A concurrent thread emptying and freeing that group during the flush-dispatch release window caused a use-after-free on group->mutex. Fixed by treating flush-dispatch the same as eviction, so the loop always breaks and re-fetches the next group via the hazard pointer. Verified with a 24MB buffer pool, 28 concurrent clients (point selects, scans, updates), innodb_lru_make_young_drain_threshold=8 to force frequent drains: 20 and 90 minute clean runs with both fixes, plus a final run after removing temporary debug instrumentation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
polchawa-percona
force-pushed
the
PS-11141-8.4-lru-groups
branch
from
August 5, 2026 16:45
d4b14fd to
fc2aa6c
Compare
…er A scans buf_LRU_free_from_common_LRU_list(), buf_flush_LRU_list_batch(), and buf_flush_single_page_from_LRU() each read a group's pages[] array under that group's own mutex, one slot at a time (32 separate lock/unlock pairs per group scanned), to stay safe against the promotion drain's group-mutex-only fast path concurrently vacating a slot. That protection only requires a consistent snapshot of the array, not a separate acquisition per slot. Copy the whole array into a local std::array under one critical section instead: a page a concurrent detach removes from the snapshot afterwards is still a live buf_page_t descriptor (descriptors are never freed, only unlinked), and every use below re-validates it under its own block mutex before acting -- the eventual buf_LRU_free_page()/detach reads bpage->lru_group fresh, so it is correct regardless of which group (if any) the page has since moved to. All three call sites already break out of their slot loop on the first outcome that could invalidate the rest of the snapshot (a successful evict/flush-dispatch, or a successful free), so a stale later entry is never dereferenced. Reduces 32 mutex acquisitions per group scanned to 1 in these hot eviction/flush paths, without changing LRU_list_mutex semantics. Suspected contributor to a throughput regression (14k->10k TPS) observed under a real sysbench oltp_read_write benchmark (64 threads, 80 CPUs, 2 buffer pool instances) -- that benchmark's crash is still separately under investigation and not addressed by this change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d callers A page enqueued on the deferred promotion queue (buf_LRU_enqueue_promote()) is buffer-fixed and its group membership is owned by the drain's pipeline from enqueue until the deferred pass finishes re-appending it (buf_LRU_drain_promote_queue()). buf_LRU_make_block_young() and buf_LRU_make_block_old() did not know about this and would act on the same page directly whenever an unrelated caller held its own, separate buf-fix on it -- e.g. purge's ibuf_update_free_bits_if_full() calls buf_page_make_young() directly, bypassing the threshold/queue routing that buf_page_make_young_if_needed() applies. This produced two distinct crashes depending on which side won the race: - buf_LRU_remove_block()'s ut_a(group) firing when the drain's fast pass had already detached the page (bpage->lru_group == nullptr). - buf_LRU_detach_from_group()'s slot-consistency assert firing when this call's own detach won instead, moving the page to a different group before the drain -- already holding a pointer to the old group -- got that group's mutex. Both were reliably reproduced within 1-2 minutes under concurrent purge-generating (DELETE+INSERT) and promotion-generating (point-select) load against a small buffer pool with a low drain threshold. Fix: check bpage->LRU_in_promote_queue at the top of both functions and skip entirely if set, instead of only checking for a null group -- this stops either race from starting at all, rather than only catching one of its two possible outcomes. The drain's own re-append already satisfies make-young's intent; for make-old, skipping silently drops one demotion attempt, which normal aging repeats on a later pass. Verified clean for ~70 minutes under the same reproducer that crashed the unfixed code in 1-2 minutes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ator Addresses the three concrete, verified findings from an external review of this branch's ~3x sysbench OLTP RW regression. 1. buf_LRU_drain_promote_queue()'s deferred pass walked every group in the pool -- not just the batch -- under LRU_list_mutex whenever the batch had emptied any group, to safely reclaim without dereferencing a group pointer captured without that mutex. Under real eviction pressure the gate is true most of the time, so this was O(pool size / 32) work in the global critical section on the common path, not the assumed O(batch). Group pointers are now retained in Drained_page_info and the deferred pass reclaims exactly the groups the batch emptied, guarded by a fresh in_LRU_list && n_pages == 0 re-check under LRU_list_mutex -- safe regardless of pointer staleness, since reclaiming any linked, empty group is always legitimate. 2. That fix requires group pointers to never dangle. buf_lru_group_alloc/ free previously did a heap allocation + mutex_create (with PFS registration) on every group creation and mutex_free + heap free on every emptying -- both in the LRU hot path, under LRU_list_mutex. Groups are now recycled through a small per-pool free-list (LRU_group_cache, bounded by BUF_LRU_GROUP_CACHE_MAX) instead, which both removes that allocation churn and gives (1) its safety property. A new buf_lru_group_t::in_LRU_list flag distinguishes a linked group from a cached one for the re-check in (1). 3. buf_LRU_old_len_validate() recomputed old_len from every group and asserted it against buf_pool->LRU_old_len on every LRU add and remove. The sampled buf_LRU_validate_instance() already asserts the identical invariant, so this added no coverage while making every mutation O(number of groups) under UNIV_DEBUG -- likely dominant in any debug-build benchmark. Removed, along with the now-dead LRU_drain_active flag it existed to let skip itself (that flag also cost two extra LRU_list_mutex acquisitions per drain call to publish safely; both are gone). Verified: clean build, git-clang-format clean, and a 7-minute run of the purge/promote-queue concurrency reproducer that caught the last two bugs in this area within 1-2 minutes (32 clients: point-select, delete+insert to drive purge, range scan; drain threshold 8; --innodb-sync-debug) completed with mysqld still alive and zero assertions. Still open per the review: the eviction-side group-only fast path (not requiring LRU_list_mutex during tail eviction, the design's original intended win) is not yet implemented, and buf_flush_LRU_list_batch() still abandons the rest of a group after one action per group. Both are next.
buf_lru_group_release() previously heap-freed a group once LRU_group_cache_len hit BUF_LRU_GROUP_CACHE_MAX (64). The promotion drain's fast pass can capture a buf_lru_group_t* without holding LRU_list_mutex and only dereference it later, under LRU_list_mutex, in its deferred pass -- so an unrelated thread's release of that exact group, if the cache happened to be full at that moment, could heap-free it out from under the drain first. Make the cache unconditionally unbounded during normal operation; the memory it pins is bounded by the pool's peak historical group count, a small, firm fraction of the pool's own budget rather than a real leak. Groups are only actually freed at buffer-pool teardown, via buf_LRU_free_group_cache().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.