[austin-barrington]

What happened. n3's shared tenant pool (hyperbytedb-shared/shared-pool-1) was OOM-killed about 216 times in 19 hours on 2026-10-24/25, at its 4 GiB limit. Each start died about 20 s after starting up. Rolling back to 0.9.2-beta crashed the same way. Raising the limit to 32 GiB let it start: opening the WAL took 88 s, and the pod's working set reached 11.1 GiB. The next start took 14 ms, and the node has used about 0.5 GiB since.

Cause. Every flush truncated the WAL with a range delete over [0, seq). On a quiet node (this pool takes about 1 point/s) the WAL CF's 64 MiB memtable never fills, so the range deletes stay in it: a week of flushes left thousands, all overlapping. RocksDB replays them on open at a cost quadratic in their number. The pool's RocksDB LOG showed two 65 MB logs taking 38 s and 49 s to recover, each into an SST of under 400 KB.

Reproduced with the real RocksDbWal, appending one entry and truncating it N times, then reopening:

N	reopen, before	peak RSS, before	reopen, after	peak RSS, after
1,250	0.6 s	127 MiB
2,500	2.2 s	453 MiB
5,000	9.9 s	1,767 MiB	0.29 s	+1 MiB
30,000	(killed for low memory at 20,000)		0.31 s	+1 MiB

Fix.

• A truncation covers only what is new since the last one. On open, it starts from the oldest entry left.
• Up to 1,024 entries take point deletes, which cost nothing to replay. More take one range delete, and the WAL CF memtable is flushed after 64 of those.
• memtable_max_range_deletions would do the latter inside RocksDB, but the C API doesn't expose it.

Tests.

• No range deletes from small truncations: after 2,000 append-and-truncate cycles, the memtable holds no range deletes (counted from RocksDB's entry and delete counts).
• Range deletes stay bounded: large truncations stay under 64 range deletes per memtable, and the memtable is flushed.
• Exactly the right entries are deleted: truncation removes the entries below the sequence and nothing else, also after a reopen and on an empty WAL.
• The test catches the bug: forcing every truncation to be a range delete fails it.
• CI-equivalent run (libchdb 26.9.0, -D warnings): passes.

Upgrade note. A node that already holds range deletes replays them once on its first start with this, as before. After that they're on disk. The shared pool has been up 29 hours, about 2,900 flushes, a few hundred MiB to replay: safe at its 8 GiB limit. It would reach about 8 GiB after roughly a week of uptime, so it should restart onto this well before then.

Likely related, not verified: the local platform's HyperbyteDB at 10.6 GB after 14.7 days at low traffic, and the e2e nodes at 12.5–12.9 GiB after days idle.

[cursor]

PR Summary

Medium Risk
Changes durable WAL deletion semantics and flush timing on a critical storage path; behavior is covered by new tests but affects recovery and memory on upgrade.

Overview
Fixes RocksDB WAL truncation so quiet nodes no longer accumulate thousands of overlapping [0, seq) range deletes in an unfilled memtable, which caused quadratic replay cost, multi‑GiB RSS, and OOM on startup.

truncate_before now deletes only sequences since the last truncation (truncated_through, initialized from the oldest WAL key on open). Truncations of ≤1024 entries use point deletes; larger spans use a single range delete, and the WAL CF is flushed after 64 range deletes so they do not stack in the active memtable.

Adds tests that assert no range deletes for many small truncations, bounded range deletes with periodic flush, and correct retention across reopen and no-op truncations behind the watermark.

^{Reviewed by Cursor Bugbot for commit 2859102. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 475447f2-1650-4b0a-89bc-697bd55b8fee)