fix: bound registry RSS growth (cap pooled save buffer + return freed pages)#102
Open
TeoSlayer wants to merge 1 commit into
Open
fix: bound registry RSS growth (cap pooled save buffer + return freed pages)#102TeoSlayer wants to merge 1 commit into
TeoSlayer wants to merge 1 commit into
Conversation
…d pages Root-caused from a live heap profile during the recurring registry wedge: live Go heap was ~4GB but RSS ~22.7GB and climbing ~3.3GB/hr with a stable connection count. Two compounding causes, both fixed here without touching the on-disk snapshot format: - flushSave encodes the whole snapshot (~2.7GB at 218k nodes) into a sync.Pool'd bytes.Buffer that grows with node count and is retained forever. putSaveBuf now drops buffers larger than maxPooledSaveBuf (128MB) instead of returning them to the pool, so the giant buffer is GC-eligible after each save. - With GOMEMLIMIT=48GiB the runtime holds freed pages rather than returning them, so RSS ratchets toward the ceiling and eventually GC-thrashes. After a large snapshot, shouldScavenge triggers an async debug.FreeOSMemory at most once per 3 minutes to return the freed pages to the OS. Neither change touches the snapshot bytes, checksum, WAL, or load path — the worst case is an extra allocation or GC, never data corruption. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (live heap profile during the recurring registry wedge)
The registry re-wedges ~daily (the outage this morning). A heap profile of the live process showed live Go heap ~4 GB but RSS ~22.7 GB and climbing ~3.3 GB/hr with a stable ~207K connection count — so it's not a connection leak. Two compounding causes:
flushSaveserializes the whole snapshot (~2.7 GB at 218K nodes) into async.Pool'dbytes.Buffer(bytes.growSlice= 67% of live heap). The pool retains that buffer forever, and it grows with node count.GOMEMLIMIT=48 GiB, the runtime keeps freed pages instead of returning them to the OS, so RSS ratchets toward the ceiling and eventually GC-thrashes → wedge.Fix (no on-disk format change)
putSaveBufdrops buffers larger thanmaxPooledSaveBuf(128 MB) instead of returning them to the pool → the giant snapshot buffer becomes GC-eligible after each save.shouldScavengetriggers an asyncdebug.FreeOSMemory()at most once per 3 minutes → returns freed pages to the OS, keeping RSS near the true working set.Safety: neither change touches the snapshot bytes, checksum, WAL, or load path. Worst case is an extra allocation or GC — never data corruption.
Tests
TestPutSaveBufDropsOversized,TestPutSaveBufKeepsNormal,TestShouldScavengeGate(small-snapshot skip, fires-after-interval, rate-limited-within-window). Existing save/persist/load tests pass; fullgo test -racegreen.Operational note
The live registry (running the #100 perf binary) will hit the 48 GB ceiling and wedge in ~7-8 h without this. A restart resets the clock (~14 h) at the cost of a reconverge dip; deploying this bounds it permanently. Recommend landing with #100 and building a combined binary.
🤖 Generated with Claude Code