Skip to content

Fix #1681: re-arm terminal reconnect budget on wake - #1682

Merged
amrmelsayed merged 16 commits into
mainfrom
builder/bugfix-1681
Sep 16, 2026
Merged

amrmelsayed merged 16 commits into
mainfrom
builder/bugfix-1681

Conversation

@amrmelsayed

@amrmelsayed amrmelsayed commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

After a laptop sleep, VS Code terminal tabs showed a permanent [Codev: Connection lost. unable to reconnect after 6 attempts. Click here to reconnect] banner while Tower and the detached shellper session were perfectly healthy — only the client view died. This re-arms the reconnect budget on wake and words the give-up banner honestly.

Fixes #1681

Root Cause

apps/vscode/src/terminal-adapter.ts uses BackoffController({ maxAttempts: 6 }). After 6 consecutive transient failures, giveUp() sets gaveUp = true permanently — the retry loop stops until a manual reconnect. The backoff curve [1s,2s,4s,8s,16s,30s] totals ~61s; during sleep the network stack is suspended, so each connect() fails instantly and the whole budget burns before the machine fully wakes. A transient OS event becomes a permanent-looking failure banner on every tab.

The only wired wake signal — onDidChangeWindowState focus rising-edge — called an opt-in, off-by-default SIGWINCH repaint; it never re-armed the reconnect budget or reconnected a gave-up adapter.

Fix

  • terminal-adapter.ts: track the give-up class (transient = terminal-adapter: WebSocket close-handler spams 'Connection lost' in a tight loop with no backoff, no give-up, no actual reconnect #936 budget exhausted vs permanent = 4xx session-gone). New onWake() reconnects a transiently gave-up or still-parked-in-backoff adapter, and is a deliberate no-op for the terminal-adapter: WebSocket close-handler spams 'Connection lost' in a tight loop with no backoff, no give-up, no actual reconnect #936 permanent class and for a healthy OPEN connection — so it never resurrects the retry storm the budget was added to stop, and never drops a live terminal. The exhausted-budget banner is now worded via a one-shot /health probe: "Tower unreachable" vs "reconnect failed (Tower is up)", falling back to the old attempt-count wording when no probe is available. A give-up generation token guards the async banner so a stale probe can't paint over a give-up that changed while /health was in flight (e.g. a permanent 4xx landing on a wake reconnect).
  • terminal-manager.ts: rearmAllOnWake() iterates managed terminals; injects the /health probe via connectionManager.getClient()?.getHealth().
  • extension.ts: calls rearmAllOnWake() unconditionally on the focus rising-edge (before the opt-in repaint).
  • Recovery-render fix (terminal-adapter.ts): the red give-up banner is client-injected text the reconnected agent TUI can't see, so on recovery its repaint overwrote only the left of the banner row and stranded the tail (…k here to reconnect]) on the composer line until the next keystroke. Auto-recovery made give-up→reconnect common, exposing it. The banner previously ended with \r\n and cleared hadReconnectNotice (vscode: terminal-adapter reconnect notices accumulate as orphaned scrollback lines; not cleared on successful reconnect #1001's persistent form, chosen before auto-recovery existed), so a successful reconnect never wiped it. It now owns the current line (no trailing newline) and is tracked as a wipeable notice, so clearReconnectNotice() erases it in place on the next successful open, before the replay paints. While the terminal stays dead nothing overwrites it, so it remains fully visible and clickable.

The #936 4xx fast give-up discrimination (classifyUpgradeError) and the #939 Click here to reconnect affordance are unchanged; RECONNECT_LINK_TEXT is untouched so the link provider still matches the reworded banner.

Test Plan

  • Regression tests added — apps/vscode/src/__tests__/terminal-adapter.test.ts + terminal-manager.test.ts: wake re-arms a transient give-up, the 4xx-permanent give-up is not resurrected on wake, wake skips a parked backoff wait, wake no-ops a healthy OPEN socket / a not-yet-opened pty, the banner wording split (Tower up / unreachable / no-probe fallback / probe-timeout race), the deferred-probe permanent-overwrite guard, the give-up banner wiped on recovery (no composer remnant), and source-level guards for the manager fan-out / probe injection / extension focus-handler wiring.
  • Build passes (pnpm --filter codev-vscode compile/porch build check).
  • All tests pass — full vscode unit suite: 1023 passed; check-types + lint clean.

Simulated vs physical: CI cannot run a real laptop sleep. The tests simulate it by burning the 6-attempt budget then firing a wake signal. The true end-to-end confirmation is a physical sleep → wake → refocus (or stop Tower → let a tab exhaust → restart Tower → refocus); recommend a manual pass before/at merge.

CMAP (impl phase, already run)

gemini APPROVE · codex COMMENT (the stale-probe race) · claude APPROVE. The race was fixed with the generation token (commit deee898). PR-phase CMAP to follow.

Known residuals (out of BUGFIX scope — candidates for follow-up)

  • No throttle on re-arm. Every focus rising-edge fires reconnect() on each non-OPEN adapter (up to codev.maxTerminals). While Tower is genuinely down, rapid window switching restarts any in-flight CONNECTING socket and resets the budget, so on a slow link a connect may never complete. A minimum re-arm interval would bound this while preserving the sleep case.
  • Focus is the only wake signal. The extension host is Node — there is no DOM online/visibilitychange event, so the issue's "window focus, online, visibility" trio collapses to the focus rising-edge. If an OS wake produces no unfocused→focused transition (VS Code frontmost across a display sleep), nothing re-arms and the click-to-reconnect link remains the only path.
  • apps/web sibling. apps/web/Terminal.tsx may have the same sleep/wake gap; left untouched per lane scope. Flagging for a separate item if confirmed.

…banner

Laptop sleep suspends the network stack, so the VS Code terminal adapter's
six-attempt reconnect budget (#936) burns instantly against dead sockets and
every tab lands on a permanent-looking 'unable to reconnect after 6 attempts'
banner while Tower and the detached session are healthy.

- terminal-adapter: track give-up class (transient vs the #936 permanent 4xx
  session-gone) and add onWake(), which reconnects a transiently gave-up or
  still-parked adapter and no-ops a healthy or permanently-gone one. The
  exhausted-budget banner is now worded via a one-shot /health probe:
  'Tower unreachable' vs 'reconnect failed (Tower is up)'.
- terminal-manager: rearmAllOnWake() + an injected /health probe.
- extension: call rearmAllOnWake() unconditionally on the window-focus rising
  edge (the extension host's only wake signal; no DOM online/visibility events).
- tests: signal-driven re-arm, the 4xx-still-gives-up guard, and the banner
  wording split.
CMAP (codex + claude) flagged a race: renderExhaustedGiveUp only rechecked
gaveUp, so if a wake reconnects while /health is pending and the reconnect then
hits a permanent 4xx, the stale transient probe could paint an exhausted-budget
banner over the 'session no longer exists' state. Replace the gaveUp recheck
with a give-up-generation token bumped on every give-up transition (enter,
success, reconnect); the async banner renders only if the token is unchanged.
Adds a deferred-probe regression test.
…ng tests

PR-phase CMAP (codex REQUEST_CHANGES, claude non-blocking):
- Race the /health probe against a 2s cap so a blackholed network can't hold the
  give-up banner for the SDK's full 10s request timeout (claude).
- Guard onWake() until VS Code has open()ed the pty, so a wake landing before
  open() can't leak a socket that open()'s connect() replaces (claude).
- Add source-level wiring guards (this file's established harness pattern) for
  the manager onWake fan-out, the /health-probe injection, and the extension
  focus-handler call, so deleting any wiring point now fails a test (codex).
- Drop a dead 'void pty' in a test (claude nit).
…nant)

Auto-recovery made give-up -> reconnect common, exposing a render glitch: the
red give-up banner is client-injected text the reconnected agent TUI doesn't
know about, so its repaint overwrites only the left of the banner row and
strands the tail ('...k here to reconnect]') on the composer line until the
next keystroke.

The banner ended with a trailing newline and cleared hadReconnectNotice (#1001's
persistent form, chosen before auto-recovery), so a successful reconnect never
wiped it. Make it own the current line (no newline) and track it as a wipeable
notice, so clearReconnectNotice() erases it in place on the next open, before
the replay paints. While the terminal stays dead nothing overwrites it, so it
remains fully visible and clickable. Adds a recovery-render regression test.
…robe timer)

- probeTowerHealth returns null (not false) when there's no client, so the
  banner falls back to neutral attempt-count wording instead of asserting
  'Tower unreachable' for an unconfigured client (claude).
- Clear the 2s probe-race timeout when the probe resolves first, so it doesn't
  outlive the resolved race (claude).
- Widen the injected probe signature to Promise<boolean | null>; add a
  null-probe fallback test and update the source-level sentinel regex.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vscode: terminal reconnect budget exhausts during laptop sleep — permanent 'unable to reconnect' banner while Tower is healthy

1 participant