fix(api): bound WS draining + alert webhook so shutdown's HTTP drain is never skipped [BUG-102] - #219
Conversation
…is never skipped [BUG-102]
shutdown()'s overall force-exit timer (SHUTDOWN_TIMEOUT_MS = 10s) fires
process.exit(1) immediately when it elapses — skipping server.close() (the
HTTP drain) entirely if it hasn't been reached yet. But `await wss.close()`
can be blocked far longer than 10s: the `ws` library's own per-socket
close-handshake timeout defaults to 30s, and wss.close() doesn't resolve
until every client has closed. A single slow/unresponsive WS client during
a rolling deploy could make the 10s force-exit fire before server.close()
is ever called, hard-killing in-flight HTTP requests on every such deploy.
Separately, sendInfoAlert (unlike flushSentry, which already caps at
2000ms) has no internal timeout on its webhook call — a hanging webhook
could also consume the same limited budget before WS/HTTP draining even
starts.
Extracted two helpers into a new src/utils/shutdown.ts (testable in
isolation, since index.ts itself has real side effects on import and
isn't unit-tested anywhere in this codebase):
- drainWebSocketClients(clients, timeoutMs): asks every client to close
cleanly, then force-terminates any still open after timeoutMs (3s),
bounding this phase regardless of client behavior.
- withTimeout(promise, ms): races a promise against a timeout,
resolving either way rather than rejecting, for best-effort steps that
must not block shutdown.
Applied both in index.ts's shutdown(): the alert send is now bounded to
2s, and WS draining is now bounded to 3s before wss.close() is called —
leaving the full 10s budget reliably available for server.close() to run.
Added tests for both helpers using fake timers, including the exact
BUG-102 scenario (a client that never acks close still gets force-terminated
within the bound). Verified by temporarily breaking the implementation
(removing the terminate() fallback) and confirming the tests fail, then
restoring the fix.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@Princessdada is attempting to deploy a commit to the Khubair Nasir's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Warning Review limit reached
More reviews will be available in 46 minutes and 58 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Problem
shutdown()'s overall force-exit timer (SHUTDOWN_TIMEOUT_MS = 10s) callsprocess.exit(1)immediately when it elapses — skippingserver.close()(the HTTP drain) entirely if it hasn't been reached yet. Butawait wss.close()(which runs beforeserver.close()) can be blocked far longer than 10s: thewslibrary's own per-socket close-handshake timeout defaults to 30s, andwss.close()doesn't resolve until every client has closed.Impact
A single slow/unresponsive WS client during a rolling deploy can make the 10s force-exit timer fire before
server.close()is ever called, hard-killing in-flight HTTP requests on every such deploy — turning the "graceful" shutdown path into the ungraceful one it was meant to prevent.Separately,
sendInfoAlert(unlikeflushSentry, which already caps at 2000ms) has no internal timeout on its webhookfetch()call (confirmed in the vendored@percolator/sharedsource) — a hanging webhook would also consume the same limited budget before WS/HTTP draining even starts, compounding the race.Fix
Extracted two helpers into a new
src/utils/shutdown.ts(testable in isolation —index.tsitself has real side effects on import and isn't unit-tested anywhere in this codebase):drainWebSocketClients(clients, timeoutMs): asks every client to close cleanly, then force-terminates any still open aftertimeoutMs(3s) — bounding this phase regardless of client behavior, so thewss.close()call right after is never left waiting on a stuck socket.withTimeout(promise, ms): races a promise against a timeout, resolving either way (not rejecting) — used to bound the alert webhook send to 2s.Applied both in
index.ts'sshutdown(). With these bounds, the alert send and WS draining together can take at most ~5s, leaving the full 10s budget reliably available forserver.close()to actually run.Proof of Fix
New tests for both helpers using fake timers, including the exact bug scenario: a client that never acks a close request still gets force-terminated within the 3s bound (not 30s).
Verified by temporarily breaking the implementation (removing the
terminate()fallback) and confirming the relevant tests fail with the expected "terminate not called" assertion errors, then restoring the fix.tsc --noEmitclean (no separate lint script in this repo).Test Output
Full suite: 300/301 passed (294 baseline + 6 new). The 1 failure (
tests/sdk-smoke.test.ts) is pre-existing and unrelated — it asserts on an exact@percolatorct/sdkerror-message string that has drifted from the locally-resolved SDK version in this environment.Related
Found during a broader API audit; no existing open issue/PR covers this.