Skip to content

fix(report): one browser per bulk PDF report + capped, time-limited render queue - #90

Merged
buzz-code merged 3 commits into
mainfrom
fix/pdf-render-queue-budget
Nov 4, 2026
Merged

buzz-code merged 3 commits into
mainfrom
fix/pdf-render-queue-budget

Conversation

@buzz-code

Copy link
Copy Markdown
Owner

Fixes buzz-code/ops#44

Cause

GET /student/report with ~59 ids returned 500 in production. Every report row calls convertMarkupToPdf (MarkupToPdfReportGenerator), which launched its own Chromium, and BulkToPdfReportGenerator loops over the rows serially - so one bulk report launched ~59 browsers. Nothing capped how many browsers from different concurrent HTTP requests run at once, and Chromium gets OOM-killed under that load: "Session closed. Most likely the page has been closed." Hanging renders would eventually be killed by the reverse proxy instead of returning a controlled error.

What the fix does

One browser per bulk report. BulkToPdfReportGenerator opens a render scope (node:async_hooks AsyncLocalStorage) around its row loop. convertMarkupToPdf inside the scope renders each row on a new page of ONE shared Chromium and closes nothing per row (pages are closed); the scope closes the browser exactly once when the report ends - also on the error path - and a failing close() never masks the real result or error. The once-per-conversion transient-error retry still works: if the shared browser dies mid-report, the retry relaunches it (dead process closed, no orphans) and the remaining rows continue on the fresh browser. Standalone conversions (SimpleMarkup/EJS/React generators outside a bulk report) keep the per-conversion browser behavior. No public method signatures changed.

Global cap + FIFO queue with a bounded wait. A module-level counter caps concurrent PDF browsers across the whole process. Conversions over the cap wait FIFO, but the wait is bounded: on expiry a new exported typed error PdfRenderQueueTimeoutError is thrown (and puppeteer.launch is never called for that conversion). The release is in a finally on every path - success, thrown error, retry, queue timeout - so a leaked slot cannot deadlock the service.

Time budget shared across attempts. PDF_RENDER_BUDGET_MS (default 45000) is the budget of one convertMarkupToPdf call. A conversion's queue wait is bounded by min(PDF_QUEUE_WAIT_MS, remaining budget), so after a transient-error retry the second attempt only gets the remaining budget for its queue wait, never a fresh full wait.

Env knobs

Env Default Meaning
PDF_MAX_CONCURRENT_BROWSERS 2 Max Chromium processes alive at once in the process
PDF_QUEUE_WAIT_MS 20000 Max time a conversion may wait in the render queue once the cap is full
PDF_RENDER_BUDGET_MS 45000 Total queue-wait budget of one convertMarkupToPdf call, shared across its retry attempts

(PDF_PRINT_TIMEOUT_MS from #89 is untouched.)

The queue timeout surfaces as a distinct typed error (PdfRenderQueueTimeoutError), so callers can map it to a clean 503 instead of the current request hanging until a reverse proxy turns it into an opaque 500.

Tests

  • New utils/report/tests/pdf-render.scope.spec.ts: 3-row bulk report launches once/newPage 3x/close once (also when row 2 fails); shared-browser relaunch at most once with remaining rows continuing; no slot leak after a relaunch (cap of 1 + follow-up conversion must render immediately); standalone conversions keep per-conversion browsers.
  • New utils/report/tests/pdf-render.queue.spec.ts: cap + FIFO wait rejects with PdfRenderQueueTimeoutError after the configured wait without launching; retry's queue wait capped by the remaining RENDER_BUDGET_MS; no slot leak after a failing conversion and after a queue timeout.
  • All 17 existing tests in markup-to-pdf.generator.spec.ts still pass (106 tests across the whole utils/report suite).

Hermes Agent and others added 3 commits November 2, 2026 19:54
…nder queue

Each bulk report row called convertMarkupToPdf, which launched its own
Chromium, and nothing capped how many browsers concurrent HTTP requests
opened at once: a 59-row report spawned 59 browsers and Chromium gets
OOM-killed under that load ("Session closed. Most likely the page has
been closed." - 500 on GET /student/report with many ids).

- BulkToPdfReportGenerator now opens one render scope around its row
  loop (node:async_hooks AsyncLocalStorage): every PDF conversion in the
  scope renders on a new page of ONE shared browser, the browser is
  closed once when the scope ends (also on the error path), and close()
  failures never mask the loop result or error. If the shared browser
  dies mid-report, the once-per-conversion retry relaunches it and the
  remaining rows continue on the fresh browser. Standalone conversions
  keep today's launch-per-conversion behavior.
- Global browser cap with a FIFO wait queue: PDF_MAX_CONCURRENT_BROWSERS
  (default 2), PDF_QUEUE_WAIT_MS (default 20000). Expiring the wait
  throws a typed PdfRenderQueueTimeoutError instead of hanging until the
  reverse proxy kills the request. PDF_RENDER_BUDGET_MS (default 45000)
  is the per-conversion budget shared by all attempts: a retry gets only
  the remaining budget for its queue wait. Slots are released on every
  path (success, error, retry, queue timeout).
- Each row also closes its page, so a many-row report cannot pile up
  open pages inside the shared browser.
@buzz-code
buzz-code merged commit 4c555b9 into main Nov 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant