Skip to content

fix(websocket): heartbeat disconnects healthy clients; auth and rate-limit bypass - #9

Open
frankstupak wants to merge 1 commit into
SkinnnyJay:mainfrom
frankstupak:lumen-uplift/websocket
Open

fix(websocket): heartbeat disconnects healthy clients; auth and rate-limit bypass#9
frankstupak wants to merge 1 commit into
SkinnnyJay:mainfrom
frankstupak:lumen-uplift/websocket

Conversation

@frankstupak

@frankstupak frankstupak commented Jul 4, 2026

Copy link
Copy Markdown

The headline

The heartbeat reaps every client that follows the protocol. The manager sends {type: "ping"} and reaps anyone whose lastPing goes stale — but there is no PONG handler anywhere in handleBuiltInMessages, no protocol-level pong listener, and the shipped useWebSocket hook never answers pings. Net effect: connect the shipped frontend to the shipped backend and every client is force-disconnected at pingTimeout (60s), forever, on a loop. Answering the heartbeat correctly did not help; sending the existing ping was the only survival strategy, and nothing in the repo does that.

Fixed end-to-end per the canonical ws pattern (isAlive / pong / terminate — see the ws README):

  • Manager handles app-level PONG and listens for protocol-level pong frames (browsers answer those automatically — JS can't see them, so both layers matter)
  • Any inbound message refreshes liveness (traffic is proof of life)
  • The hook answers {type:"ping"} with a pong echoing the timestamp, so the server can compute RTT
  • Dead connections are now actually terminated — previously the reaper deleted the map entry and left the socket open (leak); the client happily kept sending into the void

Also fixed

Rate limiting never covered built-ins. handleBuiltInMessages runs before the onMessage hook, so a rate-limited client could still flood every room via room_message all day. New onBeforeMessage gate hook runs before built-ins; rate limiting moved there (legacy onMessage path preserved, no double-counting — WeakSet-tracked).

Auth bypass. validateToken returned true for anything starting with "Bearer ". Authorization: Bearer lol was a valid credential. Now constant-time compare (crypto.timingSafeEqual) against configurable authTokens.

GET /api/history?offset=0 always returned an empty page. Query params are strings over HTTP; offset ? -offset : undefined treats "0" as truthy → slice(-limit, -0)slice(-limit, 0)[]. Coerced and fixed.

Socket.IO delivery was double-serialized. Socket.IO sockets have a send() method, so if (client.socket.send) routed them down the raw path — the emit branch was dead code and Socket.IO consumers received JSON strings instead of objects. Detection now keys on nsp.

The existing test suite fails as shipped. basic-websocket-server.test.ts room-communication test assigns client1Id/client2Id without declaring them — strict-mode ReferenceError inside the message handler, test burns its full 90s timeout. Baseline: 68 passing / 1 failing, 184s. Now: 88/88 in 6.5s.

Smaller: sendToClient checks readyState before send (no more console spam on CLOSING sockets) · messagesPerSecond reported the partial current window and never decayed — now reports real completed windows and drops to 0 when idle · addClientToRoom/removeClientFromRoom returned true for clients that don't exist — honest now · stop() leaked the cleanup interval on constructed-but-never-started servers · heartbeat traffic no longer eats the hook's 100-slot message history · reconnect uses exponential backoff with 50–100% jitter capped at 30s (fixed-interval retries = synchronized thundering-herd waves on server restart).

Numbers

Broadcast serialized once per recipient. For a 5,000-client broadcast that's 4,999 redundant JSON.stringify calls per message.

clients=5000 rounds=200 payload=970 bytes serialized
legacy per-recipient stringify : 3378.8 ms total, 16.894 ms/broadcast,   295,966 deliveries/s
uplifted single stringify      :   12.0 ms total,  0.060 ms/broadcast, 83,064,992 deliveries/s

281x per broadcast. Bench included at benchmarks/broadcast-bench.ts (npx tsx benchmarks/broadcast-bench.ts).

Verification

  • Subproject: 88/88 tests, 8/8 suites (baseline 68 passing + 1 shipped failure), 184s → 6.5s
  • Root npm run test:all: all 8 websocket suites green; the 6 failing suites are pre-existing missing-dependency errors in other subprojects (fuse.js / @prisma/client / winston not installed), identical before and after this diff
  • tsc --noEmit clean, root eslint 0 problems
  • Public API fully backward compatible — onBeforeMessage is additive, authTokens defaults to the previous "valid-token"

+19 tests covering every fix above. Diff confined to src/api/websocket/.

— Lumen Industries

…g before built-ins, close auth bypass, 281x broadcast

- Heartbeat was broken in both directions: the server never handled PONG
  (app-level or protocol-level) and the frontend hook never answered PING,
  so every correctly-behaving client was force-disconnected at pingTimeout.
  Now: PONG handler + protocol pong listener + any-message activity refresh
  on the server; pong reply (echoing server timestamp for RTT) in the hook.
- Dead connections are now actually terminated (ws terminate / socket.io
  disconnect) instead of just forgotten with the socket left open.
- New onBeforeMessage gate hook runs before built-in handlers; rate limiting
  moved there — previously a rate-limited client could still flood rooms via
  room_message because built-ins ran before the onMessage hook.
- validateToken accepted ANY 'Bearer *' string; now constant-time compare
  against configurable authTokens.
- GET /api/history?offset=0 returned an empty page (string '0' truthy ->
  slice(-limit, -0)); query params now coerced.
- Broadcast serializes once instead of once per recipient: 16.9ms -> 0.060ms
  per 5k-client 1KB broadcast (281x, bench included).
- Socket.IO clients now get structured emit (their send() method made the
  emit branch dead code; consumers received JSON strings).
- Reconnect uses exponential backoff + 50-100% jitter, capped at 30s.
- sendToClient checks readyState; messagesPerSecond reports real completed
  windows and decays; honest addClientToRoom returns; heartbeat noise kept
  out of the hook's message history; stop() no longer leaks the cleanup
  interval on never-started servers; fixed shipped ReferenceError test
  (undeclared client1Id/client2Id) that burned its full 90s timeout.
- +19 tests (88 total, was 68 passing/1 failing); suite 184s -> 6.5s.
@frankstupak frankstupak changed the title websocket: your heartbeat disconnects every healthy client — fixed, plus rate-limit bypass, auth bypass, and a 281x broadcast fix(websocket): heartbeat disconnects healthy clients; auth and rate-limit bypass Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant