Skip to content

memtable: reduce range-deletion fragment rebuild allocations - #6207

Open
tdeebswihart wants to merge 3 commits into
cockroachdb:masterfrom
tdeebswihart:tim/keyspanfrash-cheaper-rebuild-clean
Open

tdeebswihart wants to merge 3 commits into
cockroachdb:masterfrom
tdeebswihart:tim/keyspanfrash-cheaper-rebuild-clean

Conversation

@tdeebswihart

@tdeebswihart tdeebswihart commented Oct 2, 2026 •

Copy link
Copy Markdown

Each batch containing a range deletion invalidates the memtable's cached fragments, causing the next reader to rebuild them. While the cache is being built all readers are blocked, which is causing nasty spikes in the 99th percentile tail latency of our critical workloads.

Rebuilding previously allocated keys for each tombstone and fragment, even when the tombstones were disjoint. This PR keeps the lazy cache design and reduces its rebuild allocations:

  1. Add Metrics.MemTableRangeDelCache to understand the dynamics of the cache.
  2. Add pebble bench rangedel to approximate our workload and measure the impact of this optimization.
  3. Preallocate decode keys and output spans using the tombstone count as a capacity hint. Emit isolated tombstones directly and pass overlapping spans through the Fragmenter. Range-key rebuilding stays unchanged.

Early testing on our production workload shows 99th percentile read latencies at around one third of their previous level.

Earlier rebuild microbenchmarks on an AMD EPYC 9R45, with 1,000 and 10,000 tombstones, showed disjoint rebuilds taking 57-73% less time and mostly-disjoint rebuilds taking 44-69% less time. Disjoint rebuild allocations fell from roughly 2,000-20,000 to 5-6 per operation. Mostly-disjoint allocations fell by about 90%, and allocations fell by about 33% when range deletes overlap.

Future Work

Given how much we rely on range deletion, as a follow-up I'm prototyping an incrementally-maintained rangedel cache. That's a much more invasive change and I've a lot more work to do on it before I waste y'all time on it, but is that something you'd be interested in in the first place?

@cockroach-teamcity

Copy link
Copy Markdown
Member

This change is Reviewable

@tdeebswihart
tdeebswihart force-pushed the tim/keyspanfrash-cheaper-rebuild-clean branch from 029fbf7 to 89efad0 Compare October 2, 2026 18:55
@tdeebswihart
tdeebswihart marked this pull request as ready for review October 4, 2026 17:44
@tdeebswihart
tdeebswihart requested a review from a team as a code owner October 4, 2026 17:44
@tdeebswihart
tdeebswihart requested a review from fqazi October 4, 2026 17:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants