Repository navigation
fix(report): one browser per bulk PDF report + capped, time-limited render queue - #90
Merged
Merged
Conversation
…nder queue
Each bulk report row called convertMarkupToPdf, which launched its own
Chromium, and nothing capped how many browsers concurrent HTTP requests
opened at once: a 59-row report spawned 59 browsers and Chromium gets
OOM-killed under that load ("Session closed. Most likely the page has
been closed." - 500 on GET /student/report with many ids).
- BulkToPdfReportGenerator now opens one render scope around its row
loop (node:async_hooks AsyncLocalStorage): every PDF conversion in the
scope renders on a new page of ONE shared browser, the browser is
closed once when the scope ends (also on the error path), and close()
failures never mask the loop result or error. If the shared browser
dies mid-report, the once-per-conversion retry relaunches it and the
remaining rows continue on the fresh browser. Standalone conversions
keep today's launch-per-conversion behavior.
- Global browser cap with a FIFO wait queue: PDF_MAX_CONCURRENT_BROWSERS
(default 2), PDF_QUEUE_WAIT_MS (default 20000). Expiring the wait
throws a typed PdfRenderQueueTimeoutError instead of hanging until the
reverse proxy kills the request. PDF_RENDER_BUDGET_MS (default 45000)
is the per-conversion budget shared by all attempts: a retry gets only
the remaining budget for its queue wait. Slots are released on every
path (success, error, retry, queue timeout).
- Each row also closes its page, so a many-row report cannot pile up
open pages inside the shared browser.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes buzz-code/ops#44
Cause
GET /student/report with ~59 ids returned 500 in production. Every report row calls convertMarkupToPdf (MarkupToPdfReportGenerator), which launched its own Chromium, and BulkToPdfReportGenerator loops over the rows serially - so one bulk report launched ~59 browsers. Nothing capped how many browsers from different concurrent HTTP requests run at once, and Chromium gets OOM-killed under that load: "Session closed. Most likely the page has been closed." Hanging renders would eventually be killed by the reverse proxy instead of returning a controlled error.
What the fix does
One browser per bulk report. BulkToPdfReportGenerator opens a render scope (node:async_hooks AsyncLocalStorage) around its row loop. convertMarkupToPdf inside the scope renders each row on a new page of ONE shared Chromium and closes nothing per row (pages are closed); the scope closes the browser exactly once when the report ends - also on the error path - and a failing close() never masks the real result or error. The once-per-conversion transient-error retry still works: if the shared browser dies mid-report, the retry relaunches it (dead process closed, no orphans) and the remaining rows continue on the fresh browser. Standalone conversions (SimpleMarkup/EJS/React generators outside a bulk report) keep the per-conversion browser behavior. No public method signatures changed.
Global cap + FIFO queue with a bounded wait. A module-level counter caps concurrent PDF browsers across the whole process. Conversions over the cap wait FIFO, but the wait is bounded: on expiry a new exported typed error PdfRenderQueueTimeoutError is thrown (and puppeteer.launch is never called for that conversion). The release is in a finally on every path - success, thrown error, retry, queue timeout - so a leaked slot cannot deadlock the service.
Time budget shared across attempts. PDF_RENDER_BUDGET_MS (default 45000) is the budget of one convertMarkupToPdf call. A conversion's queue wait is bounded by min(PDF_QUEUE_WAIT_MS, remaining budget), so after a transient-error retry the second attempt only gets the remaining budget for its queue wait, never a fresh full wait.
Env knobs
(PDF_PRINT_TIMEOUT_MS from #89 is untouched.)
The queue timeout surfaces as a distinct typed error (PdfRenderQueueTimeoutError), so callers can map it to a clean 503 instead of the current request hanging until a reverse proxy turns it into an opaque 500.
Tests