Healthy relay connections: heartbeat, per-browser queues, send backpressure, hint for several tabs

[k3mpaxl]

Why

These commits reached fix/websocket-logs after #8 was merged, so they are not on main yet. Same changes in Enricher: k3mpaxl/Enricher#2. Three problems seen on App Service:

• Large boards reconnected every few seconds during sync.
  • The relay forwarded a browser's messages inside that browser's receive loop. While a slow receiver drained, the sender was not read, so its pong stayed unread and uvicorn closed it ("keepalive ping timeout", 1011). Then the sync started over.
  • A browser also queued a whole sync at once, so its own pong waited behind megabytes of upload.
  • The trigger here was the same large board open in several tabs. Every tab receives and processes the whole sync, and tabs of one site often share a renderer thread, so each one reads slowly.
• Some users could not reconnect until the app was restarted, while others stayed connected.
  • The App Service front end answered their WebSocket upgrades with HTTP 429.
  • Connections of browsers that had vanished (closed laptop, changed network) kept their slot in the platform's WebSocket limit, because the front end kept them alive and uvicorn's pings never noticed.
• Nothing told the analyst that a large board was open several times.

What changed

Relay (cece594)

• Per-browser queues. deliver() puts every message for a browser into that browser's own outbox, and one writer task per browser sends at its pace. This covers welcome, actions, frames, sync requests, api-commands and heartbeat replies. A browser more than 64 MiB behind is closed with 1013 and catches up by summary sync when it reconnects.
• Heartbeat. The browser sends {"type":"heartbeat"} every 25 s and the server answers.
  • The server closes a browser that announced a heartbeat in hello and then stays silent for six intervals (2.5 min, code 4000), which frees its slot without a restart.
  • The browser reconnects when it has heard nothing for 2.5 min. Failed attempts back off from 2 s to 30 s.
  • Older browsers without a heartbeat are closed only by pings.
• Send backpressure. drained() makes the browser send the next message or reply part only when bufferedAmount is below 1 MB.
• Pong timeout. The image sets UVICORN_WS_PING_TIMEOUT=120.
• Room cleanup. A browser that leaves between hello and welcome is removed from the room.

Tabs (1fcc0be)

• Hint. A tab shows a dismissible hint when the same board is open in other tabs of this browser. Tabs count each other over the existing identity BroadcastChannel. They count again:
  • when a tab opens or closes the board;
  • when a peer leaves (a killed tab never fires pagehide);
  • once a minute.
• Background tabs. Hidden tabs no longer wait for backpressure. Browsers run their timers once a second or once a minute, so a background tab answering a sync request would have sent about 1 MB per minute.

README. Documents the App Service WebSocket limits per plan (F1: 5, B1: 350) and the cost of several tabs.

Tests

• tests/test_relay_flow.py uses a stalled browser, a fast one and a sender with 40 × 512 KB frames. On the old relay it fails because the sender is closed with 1011.
• test_relay covers the heartbeat: it is answered, silence closes with 4000, and browsers without a heartbeat stay connected.
• query.cjs covers backpressure, including a hidden tab that does not wait.
• The identity e2e test now checks the hint: both tabs show it, it can be dismissed, and it goes away when the other tab closes.
• Backend tests, web unit tests and the Playwright suite (29 passed, 1 skipped) pass. In the browser pane the hint also disappeared within 2 s when the other tab was killed without pagehide.
• import queue and attack impact failed now and then while traces were recorded under load. Without traces they passed 20/20 here and on the base, so the failures look timing-related.

🤖 Generated with Claude Code