Repository navigation
fix(engine): back off and cap the service rollup window after an OOM - #491
Merged
Merged
Conversation
On a memory-limit failure the service rollup narrowed its ingest window toward the one-second floor, but wasted roughly a dozen passes doing it. Two problems compounded: - drainServiceRollup retried an out-of-memory pass back-to-back for the whole tick budget, running the window to the memory limit a dozen times in a row and starving the queries that memory is for. - After rollupChunkWins committed passes the window doubled straight back into the size that had just failed. With no memory of the failing size a dense region failed roughly every fourth pass forever. drainServiceRollup now ends the tick on any error instead of retrying a failing window; the rollup loop's interval (>=1s, 1m by default) spaces the next attempt out, the same cross-tick spacing runBackgroundLoop relies on. The non-OOM path is unchanged: a committed pass with pending work still drains within the budget. rollupChunk gains a ceiling: the smallest window a committed streak grew back into and then found too large. Growth caps strictly below it and holds at the largest known-safe window, so a committed streak never doubles back into a size known to fail. Only a failure the window grew into sets the ceiling -- an initial or shrinking OOM is one episode of pressure, not proof that a grown window is too large -- so a transient spike or a deep backfill still recovers to the full unbounded window once committed passes grow it back. Closes #481
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a memory-limit (OutOfMemory) failure the service rollup pass narrows its ingest window toward the 1s floor, but two things made a dense region thrash instead of settling:
drainServiceRollupretried a failing window back-to-back inside the ~30s tick budget, so going from a 1h window down to the 1s floor ran roughly 12 passes to the memory limit in a row, each one reclaiming the memory the live queries need.rollupChunkWins(3) committed passes the window doubled straight back into the size that had just run out of memory, so a dense region failed roughly every fourth pass forever and the rollup never settled.Closes #481.
Fix (all in
internal/engine/duck.go)Remember the failing size as a ceiling.
rollupChunkgainsceiling(the smallest window a committed streak grew back into and then found too large) andgrew(whether the current window was reached by growing rather than shrinking). Growth now caps strictly below the ceiling and holds at the largest known-safe window instead of doubling back into a size that already OOM'd.Only a failure the window grew into sets the ceiling. An initial or a shrinking out-of-memory pass is one episode of pressure, not evidence that a grown window is too large, so a transient memory spike or a deep backfill still recovers to the full unbounded window once committed passes grow it back — this keeps the existing shrink-cascade recovery behavior intact.
Back off after an OOM.
drainServiceRollupnow ends the tick on any error instead of retrying a failing window back-to-back. Non-OOM errors already returned, so the only behavior change is that an OOM ends the tick; the rollup loop's interval (RollupInterval, 1m default, 1s floor) then spaces the next attempt out. The backoff is deliberately not added to the shared rollup ticker, because that ticker also drives the version rollup, read caches, and edge rollup, and a service-rollup OOM should not slow those healthy rollups. No blocking sleep is added to the drain path. The success path (draining pending committed work within the budget) is unchanged.Tests
Two behavioral tests added to
internal/engine/rollup_chunk_test.go, both confirmed red before the fix:TestServiceRollupChunkDoesNotGrowBackIntoFailingSize— after growing into a window that OOMs, 100 committed passes never reach or exceed that failing size and never snap back to the unbounded window.TestServiceRollupChunkSettlesBelowRememberedCeiling— after an OOM at a grown 30m window, committed passes settle at 15m (the largest known-safe window), proving the ceiling is remembered across the shrink/grow cycle without pinning the window too small.A drain-level OOM test is intentionally omitted: forcing a genuine OutOfMemory through
refreshServiceRollup's real DuckDB SQL would need heavy, flaky memory-pressure fixtures; the contract is covered at the unit level, and the existing engine tests exercise the drain success path for no throughput or correctness regression.Verification
just test "./internal/engine/..."(full suite),just vet, andjust check-static(gofmt, deadcode, behavioral test names) all green.