Skip to content

Desktop: drop a box when its wire closes, so a dead connection can't hang the board - #127

Merged
pallaoro merged 1 commit into
mainfrom
although-i-was-already-using-that
Aug 17, 2026
Merged

Desktop: drop a box when its wire closes, so a dead connection can't hang the board#127
pallaoro merged 1 commit into
mainfrom
although-i-was-already-using-that

Conversation

@pallaoro

Copy link
Copy Markdown
Member

The bug

A box whose transport died silently — Mac asleep, Tailscale flap, the relay's remote end gone — stayed registered as a live engine. Nothing ever removed it: the health probe that tears down a broken box is ws-only, and an RPC has no per-call timeout, so every call routed into that box waited forever. The box stayed green on the board while swallowing requests.

Creating the first task for a repo on such a box left "Setting up <repo> on <box>…" on screen indefinitely. It was parked on connect()'s already-held path — which asks the box for system:hello — so the clone the toast was reporting had not even been requested yet. On the box that clone takes ~3.5s.

Changes

  • A closed wire drops its backendconnect() wires transport.onClose → disconnect(alias), wire-agnostic on purpose: the ws probe only catches a half-open socket, while a real close is all SSH ever gives us.
  • Cache each box's handshake SystemInfo instead of re-asking on every status read. connected() was one system:hello per box on every connect / disconnect / install, so one stale box hung the whole list — and with it connect()'s already-held path. The local engine is still asked live (in-process, and its agent list changes under us); installAgent refreshes the cached info.
  • Merge reads tolerate an engine that can't answerprojects:list / agents:list / loops:list get a per-backend cap and drop that engine's slice instead of the union. A bare Promise.all over a stale backend hung the entire board's project list, and rejected the whole list if one engine errored.
  • Hold SSH connections with ServerAliveInterval=15 / ServerAliveCountMax=3. OpenSSH's own liveness check defaults to off, so a black-holed connection left the ssh child alive with nothing coming back. 15×3 gives a real exit in ~45s — which closes the pipe, and a closed pipe is what drops the box.

Deliberately not a per-call RPC timeout: projects:clone, the installer and tasks:cleanup legitimately run for minutes. Liveness belongs to the channel, not the call.

Verification

  • bun test — 154 pass, including two new regressions: a merge read survives an engine that never answers, and one that errors.
  • Live against the Hetzner box through the edited transport: argv carries the keepalive flags, handshake + a real projects:list succeed, onClose fires on teardown, and projects:clone completes in 3.5s.
  • In the dev app: killing one box's relay drops that box and leaves the sibling connection — and the app — untouched.

Docs: online-ateam.md now states that a box dropping mid-session shows as disconnected, with the keepalive detection window.

…hang the board

A box whose transport died silently (Mac asleep, Tailscale flap, the relay's
remote end gone) stayed registered as a live engine. Nothing ever removed it:
the health probe that tears down a broken box is ws-only, and an RPC has no
per-call timeout, so every call routed into that box waited forever — the box
stayed green on the board while swallowing requests. Creating the first task
for a repo on such a box left "Setting up <repo> on <box>…" on screen
indefinitely: it was parked on connect()'s already-held path, which asks the
box for system:hello, so the clone it was reporting had not even been asked
for. On the box itself that clone takes ~3.5s.

- A closed wire now drops its backend (connect wires transport.onClose ->
  disconnect), wire-agnostic: the ws probe only catches a HALF-open socket,
  while a real close is all SSH ever gives us.
- Cache each box's handshake SystemInfo instead of re-asking on every status
  read. connected() was one system:hello PER BOX on every connect/disconnect/
  install, so one stale box hung the whole list — and with it connect()'s
  already-held path. The local engine is still asked live (in-process, and its
  agent list changes under us); installAgent refreshes the cached info.
- Merge reads (projects/agents/loops list) tolerate an engine that can't
  answer: per-backend cap, and its slice is dropped instead of the union. A
  bare Promise.all over a stale backend hung the entire board's project list,
  and rejected the whole list if one engine errored.
- Hold SSH connections with ServerAliveInterval=15/ServerAliveCountMax=3.
  OpenSSH's own liveness check defaults to OFF, so a black-holed connection
  left the ssh child alive with nothing coming back; 15x3 gives a real exit in
  ~45s, which closes the pipe and drops the box.

Verified live against the Hetzner box through the edited transport (handshake,
a board read, onClose on teardown, and projects:clone in 3.5s), and in the dev
app: killing one box's relay drops that box, leaves the sibling connection and
the app untouched.
@pallaoro
pallaoro enabled auto-merge (squash) August 17, 2026 10:11
@pallaoro
pallaoro merged commit 9756e73 into main Aug 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant