Skip to content

fix(api): bound WS draining + alert webhook so shutdown's HTTP drain is never skipped [BUG-102] - #219

Open
Morenikeoa wants to merge 1 commit into
dcccrypto:mainfrom
Morenikeoa:fix/shutdown-drain-race
Open

fix(api): bound WS draining + alert webhook so shutdown's HTTP drain is never skipped [BUG-102]#219
Morenikeoa wants to merge 1 commit into
dcccrypto:mainfrom
Morenikeoa:fix/shutdown-drain-race

Conversation

@Morenikeoa

Copy link
Copy Markdown

Problem

shutdown()'s overall force-exit timer (SHUTDOWN_TIMEOUT_MS = 10s) calls process.exit(1) immediately when it elapses — skipping server.close() (the HTTP drain) entirely if it hasn't been reached yet. But await wss.close() (which runs before server.close()) can be blocked far longer than 10s: the ws library's own per-socket close-handshake timeout defaults to 30s, and wss.close() doesn't resolve until every client has closed.

Impact

A single slow/unresponsive WS client during a rolling deploy can make the 10s force-exit timer fire before server.close() is ever called, hard-killing in-flight HTTP requests on every such deploy — turning the "graceful" shutdown path into the ungraceful one it was meant to prevent.

Separately, sendInfoAlert (unlike flushSentry, which already caps at 2000ms) has no internal timeout on its webhook fetch() call (confirmed in the vendored @percolator/shared source) — a hanging webhook would also consume the same limited budget before WS/HTTP draining even starts, compounding the race.

Fix

Extracted two helpers into a new src/utils/shutdown.ts (testable in isolation — index.ts itself has real side effects on import and isn't unit-tested anywhere in this codebase):

  • drainWebSocketClients(clients, timeoutMs): asks every client to close cleanly, then force-terminates any still open after timeoutMs (3s) — bounding this phase regardless of client behavior, so the wss.close() call right after is never left waiting on a stuck socket.
  • withTimeout(promise, ms): races a promise against a timeout, resolving either way (not rejecting) — used to bound the alert webhook send to 2s.

Applied both in index.ts's shutdown(). With these bounds, the alert send and WS draining together can take at most ~5s, leaving the full 10s budget reliably available for server.close() to actually run.

Proof of Fix

New tests for both helpers using fake timers, including the exact bug scenario: a client that never acks a close request still gets force-terminated within the 3s bound (not 30s).

Verified by temporarily breaking the implementation (removing the terminate() fallback) and confirming the relevant tests fail with the expected "terminate not called" assertion errors, then restoring the fix.

  • All existing tests pass — output attached.
  • New tests pass against the fix; fail against the artificially-reverted implementation (verified locally).
  • tsc --noEmit clean (no separate lint script in this repo).

Test Output

✓ tests/utils/shutdown.test.ts (6 tests) 15ms

Full suite: 300/301 passed (294 baseline + 6 new). The 1 failure (tests/sdk-smoke.test.ts) is pre-existing and unrelated — it asserts on an exact @percolatorct/sdk error-message string that has drifted from the locally-resolved SDK version in this environment.

Related

Found during a broader API audit; no existing open issue/PR covers this.

…is never skipped [BUG-102]

shutdown()'s overall force-exit timer (SHUTDOWN_TIMEOUT_MS = 10s) fires
process.exit(1) immediately when it elapses — skipping server.close() (the
HTTP drain) entirely if it hasn't been reached yet. But `await wss.close()`
can be blocked far longer than 10s: the `ws` library's own per-socket
close-handshake timeout defaults to 30s, and wss.close() doesn't resolve
until every client has closed. A single slow/unresponsive WS client during
a rolling deploy could make the 10s force-exit fire before server.close()
is ever called, hard-killing in-flight HTTP requests on every such deploy.

Separately, sendInfoAlert (unlike flushSentry, which already caps at
2000ms) has no internal timeout on its webhook call — a hanging webhook
could also consume the same limited budget before WS/HTTP draining even
starts.

Extracted two helpers into a new src/utils/shutdown.ts (testable in
isolation, since index.ts itself has real side effects on import and
isn't unit-tested anywhere in this codebase):
  - drainWebSocketClients(clients, timeoutMs): asks every client to close
    cleanly, then force-terminates any still open after timeoutMs (3s),
    bounding this phase regardless of client behavior.
  - withTimeout(promise, ms): races a promise against a timeout,
    resolving either way rather than rejecting, for best-effort steps that
    must not block shutdown.

Applied both in index.ts's shutdown(): the alert send is now bounded to
2s, and WS draining is now bounded to 3s before wss.close() is called —
leaving the full 10s budget reliably available for server.close() to run.

Added tests for both helpers using fake timers, including the exact
BUG-102 scenario (a client that never acks close still gets force-terminated
within the bound). Verified by temporarily breaking the implementation
(removing the terminate() fallback) and confirming the tests fail, then
restoring the fix.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
@vercel

vercel Bot commented Jun 26, 2026

Copy link
Copy Markdown

@Princessdada is attempting to deploy a commit to the Khubair Nasir's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Morenikeoa, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 46 minutes and 58 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 57d02bc0-e82b-42b8-8da0-575f290819e5

📥 Commits

Reviewing files that changed from the base of the PR and between b2751f4 and 04fc7c1.

📒 Files selected for processing (3)
  • src/index.ts
  • src/utils/shutdown.ts
  • tests/utils/shutdown.test.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant