Skip to content

Relay: reconnects fail ~22% of the time on hosted communities (500 in auth path, 404 on WS upgrade) #5030

Description

@sunnnsolar

Reporting measurements from a hosted community (*.communities.buzz.xyz), gathered from buzz-acp harness logs on one macOS machine running Buzz Desktop 0.5.5 across ~6 days (2026-08-01 → 08-06). Happy to supply raw logs.

1. Long-lived WebSockets are recycled about every 15–17 minutes

420 relay connection lost — reconnecting… events across the agent sessions on this machine. Histogram of the gap between consecutive drops per session:

gaps in the 14–21 min window: 186 of 420   (median 17.1 min)

15m30s : ##################          (18)
16m00s : ######################################### (41)
16m30s : ###########################  (27)
17m00s : ##########################   (26)
17m30s : ###################          (19)
18m00s : #############################(29)
18m30s : #######                      (7)
19m00s : ##########                   (10)

Note the hard floor at 15m30s — nothing at 13–15 min at all — with a tail upward. That is the shape of a fixed max connection age (~900s) observed on the client's next heartbeat, not random instability. Drops are spread evenly across sessions (87 / 88 / 90) and are mostly not simultaneous — only 3 seconds in the whole window had 3+ sessions drop together — so this looks per-connection rather than a server restart.

If that recycling is intentional (proxy/LB max age), this part is just FYI — the client handles it and most reconnects land in under a second. The next part is the actual problem.

2. About one reconnect in five fails, and the relay is the one erroring

528 reconnect attempts
114 failures (21.6%)
 12 occasions where all 5 attempts were exhausted

Failure reasons:

count error
82 Connection closed
11 Auth failed: error: internal error checking restriction state
9 WebSocket protocol error: Connection reset without closing handshake
8 WebSocket error: HTTP error: 404 Not Found
4 IO error: Connection reset by peer (os error 54)

Two of these are generated by the relay itself and look like genuine server-side bugs:

  • internal error checking restriction state (11×) — a 500 out of the auth path. The client is authenticating with the same key that worked moments earlier.
  • HTTP error: 404 Not Found (8×) — a 404 on the WebSocket upgrade for a relay URL that is otherwise valid and serving.

A representative cascade where every attempt failed against a different error:

17:10:31Z  reconnect attempt 1 failed: Connection closed
17:10:41Z  reconnect attempt 2 failed: Auth failed: error: internal error checking restriction state
17:10:49Z  reconnect attempt 3 failed: WebSocket protocol error: Connection reset without closing handshake
17:10:56Z  reconnect attempt 4 failed: WebSocket error: HTTP error: 404 Not Found
17:11:12Z  reconnect attempt 5 failed: Connection closed

The mix of 500s, 404s and resets inside 40 seconds reads like requests landing on an instance that is not ready to serve — but the auth-path 500 in particular seems worth a look on its own, since it is the relay reporting an internal failure rather than a rejection.

Impact

buzz-acp gives up after 5 attempts, so those 12 exhaustions are windows where an agent session is off the relay until something re-engages it. Combined with #1, every long-lived client is re-rolling this dice roughly 4×/hour.

Related

#5029 addresses the client-side presentation of these blips (the sidebar card currently says "Can't reach the relay / Click to connect" while the backoff loop is mid-retry). It does not address the failures above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions