Skip to content

fix(engine): back off and cap the service rollup window after an OOM - #491

Merged
vishr merged 1 commit into
mainfrom
fix/oom-rollup-backoff
Nov 16, 2026
Merged

vishr merged 1 commit into
mainfrom
fix/oom-rollup-backoff

Conversation

@vishr

@vishr vishr commented Nov 15, 2026

Copy link
Copy Markdown
Member

Problem

On a memory-limit (OutOfMemory) failure the service rollup pass narrows its ingest window toward the 1s floor, but two things made a dense region thrash instead of settling:

  1. No backoff. drainServiceRollup retried a failing window back-to-back inside the ~30s tick budget, so going from a 1h window down to the 1s floor ran roughly 12 passes to the memory limit in a row, each one reclaiming the memory the live queries need.
  2. No memory of the failing size. After rollupChunkWins (3) committed passes the window doubled straight back into the size that had just run out of memory, so a dense region failed roughly every fourth pass forever and the rollup never settled.

Closes #481.

Fix (all in internal/engine/duck.go)

Remember the failing size as a ceiling. rollupChunk gains ceiling (the smallest window a committed streak grew back into and then found too large) and grew (whether the current window was reached by growing rather than shrinking). Growth now caps strictly below the ceiling and holds at the largest known-safe window instead of doubling back into a size that already OOM'd.

Only a failure the window grew into sets the ceiling. An initial or a shrinking out-of-memory pass is one episode of pressure, not evidence that a grown window is too large, so a transient memory spike or a deep backfill still recovers to the full unbounded window once committed passes grow it back — this keeps the existing shrink-cascade recovery behavior intact.

Back off after an OOM. drainServiceRollup now ends the tick on any error instead of retrying a failing window back-to-back. Non-OOM errors already returned, so the only behavior change is that an OOM ends the tick; the rollup loop's interval (RollupInterval, 1m default, 1s floor) then spaces the next attempt out. The backoff is deliberately not added to the shared rollup ticker, because that ticker also drives the version rollup, read caches, and edge rollup, and a service-rollup OOM should not slow those healthy rollups. No blocking sleep is added to the drain path. The success path (draining pending committed work within the budget) is unchanged.

Tests

Two behavioral tests added to internal/engine/rollup_chunk_test.go, both confirmed red before the fix:

  • TestServiceRollupChunkDoesNotGrowBackIntoFailingSize — after growing into a window that OOMs, 100 committed passes never reach or exceed that failing size and never snap back to the unbounded window.
  • TestServiceRollupChunkSettlesBelowRememberedCeiling — after an OOM at a grown 30m window, committed passes settle at 15m (the largest known-safe window), proving the ceiling is remembered across the shrink/grow cycle without pinning the window too small.

A drain-level OOM test is intentionally omitted: forcing a genuine OutOfMemory through refreshServiceRollup's real DuckDB SQL would need heavy, flaky memory-pressure fixtures; the contract is covered at the unit level, and the existing engine tests exercise the drain success path for no throughput or correctness regression.

Verification

just test "./internal/engine/..." (full suite), just vet, and just check-static (gofmt, deadcode, behavioral test names) all green.

On a memory-limit failure the service rollup narrowed its ingest window
toward the one-second floor, but wasted roughly a dozen passes doing it.
Two problems compounded:

- drainServiceRollup retried an out-of-memory pass back-to-back for the
  whole tick budget, running the window to the memory limit a dozen times
  in a row and starving the queries that memory is for.
- After rollupChunkWins committed passes the window doubled straight back
  into the size that had just failed. With no memory of the failing size a
  dense region failed roughly every fourth pass forever.

drainServiceRollup now ends the tick on any error instead of retrying a
failing window; the rollup loop's interval (>=1s, 1m by default) spaces the
next attempt out, the same cross-tick spacing runBackgroundLoop relies on.
The non-OOM path is unchanged: a committed pass with pending work still
drains within the budget.

rollupChunk gains a ceiling: the smallest window a committed streak grew
back into and then found too large. Growth caps strictly below it and holds
at the largest known-safe window, so a committed streak never doubles back
into a size known to fail. Only a failure the window grew into sets the
ceiling -- an initial or shrinking OOM is one episode of pressure, not proof
that a grown window is too large -- so a transient spike or a deep backfill
still recovers to the full unbounded window once committed passes grow it
back.

Closes #481
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 2b73bcd Nov 16, 2026
31 checks passed
@vishr
vishr deleted the fix/oom-rollup-backoff branch November 16, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: OOM rollup fallback has no backoff or memory of the failing size

1 participant