Session survival: decouple PTYs from the server process (hot-reload without killing terminals) + hardened auto-resume

[web3dev1337]

Problem

Every live terminal is a node-pty child of the orchestrator server process. Any server restart — nodemon picking up a code change, updating to a new version, a crash — kills all running Claude/Codex sessions. Conversations survive on disk (transcripts), but live TUIs are lost and restoration is manual. Separately, after a full reboot the recovery flow works but is not as robust as it should be (stale/duplicated lastConversationId entries were observed in session-recovery data on 2026-07-19).

Direction A — survive server restarts (the "hot reload" ask)

Decouple PTY ownership from the app server:

• tmux-backed sessions (recommended): spawn each session inside a per-session tmux session (tmux new-session -d -s <sessionId>); the server attaches via tmux control mode (tmux -CC) or pipe-pane for output. On server restart, sessions keep running under the tmux server; the orchestrator re-attaches instead of respawning. Bonus: native scrollback survives too.
• Alternative: a minimal long-lived pty-broker daemon (owns node-pty processes, unix-socket IPC to the app server). Same effect, more code to maintain; tmux gives it for free.
• Scope note: nodemon should restart only the app server, never the tmux server; sessionManager's spawn/write/resize/kill paths route through the broker/tmux layer.

Direction B — survive reboots (resume hardening)

Nothing survives power-off except transcripts, so resume is the mechanism — make it excellent:

• One-click / auto "restore all": relaunch each recovered terminal in its cwd with claude --resume <lastConversationId> (fall back to --continue when the id is missing/stale).
• Fix conversation-id capture: several sessions shared identical stale ids (visual-novels work2/7/8, kpop-idol-empire, roblox-game-kit) — the detector is attributing the wrong transcript in some flows.
• Preload scrollback from /api/sessions/:id/log on restore (shipped in PR fix: recompute terminal layout on mobile breakpoint changes #997) so restored terminals don't come back blank.

Explicit non-goal

True process freezing (CRIU checkpoint/restore) — impractical under WSL2 and pointless for agents holding live API streams; those connections can never be thawed. Transcript + resume is the correct persistence model; A+B make it feel seamless.

[web3dev1337]

Design notes — implications of tmux-owned sessions (from review discussion):

CLI compatibility ("can Claude/Codex even run there?") — non-issue. tmux provides a real PTY; from the CLI's perspective a tmux pane is indistinguishable from any terminal (same as node-pty today). The related trap is environmental, not tmux: Claude Code sets CLAUDECODE=1, and tmux sessions inherit env from the tmux server, which inherits from whoever started it first. If the tmux server is ever started from inside an orchestrator/Claude terminal, that leaks into every future session (same class of bug as the ORCHESTRATOR_PORT .env override). Required: dedicated socket (tmux -L agent-workspace), server started by the daemon with scrubbed env (env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT), explicit set-environment hygiene.

Performance — negligible. One tmux server + one pane per session, all C, a few MB total; data path gains one copy per chunk. Prefer pipe-pane (raw bytes, identical stream to today, statusDetector heuristics unchanged) over control mode (-CC wraps output in %output events needing unescaping). Set a sane history-limit (~20k lines) — xterm.js already keeps client-side scrollback; don't double-buffer huge histories per pane.

Logging/visibility — net improvement. capture-pane gives real scrollback that survives server restarts (today the buffer dies with the server), and tmux -L agent-workspace attach -t <id> becomes a debugging escape hatch when the UI is down. Caveat: output produced while the app server is down isn't shipped to server-side logs — backfill from capture-pane on reattach if complete logs matter. UI config per socket: status off, prefix None, tmux mouse mode off, and keep window size synced to xterm (resize-window) to avoid the 80x24 mismatch.

Orphaned processes — the honest flip side. Today a server death kills everything (accidentally clean); with decoupling, sessions outlive the server by design, so lifecycle management becomes explicit: (1) reconcile on startup — enumerate sessions on the socket, re-adopt known ids, surface unknown ones in the UI as "orphaned — adopt or kill"; (2) close-session = tmux kill-session, not detach; (3) escape hatch tmux -L agent-workspace kill-server; (4) session name == orchestrator session id for unambiguous ownership; (5) re-point the existing process-tree SIGTERM/SIGKILL hardening at pane PIDs. Risk to accept knowingly: an agent can keep working (and burning tokens) unattended while the UI is down — reconcile-on-start bounds this; idle agents at a prompt cost nothing.

Platform boundary — tmux exists on Linux/WSL/macOS only. Packaged Windows-native (ConPTY) builds keep the current in-process model or get a broker daemon later; the tmux layer must be optional per platform.

Lifetime boundary — survives app-server restarts and nodemon reloads; does NOT survive wsl --shutdown/reboot. Direction B (resume hardening) remains the reboot answer.

[web3dev1337]

Direction A (survive server restarts) implemented and end-to-end verified in PR #1027. tmux-backed sessions on a per-instance socket; proved on a live 34-session instance that pane pids are identical across a nodemon reload, pre-restart scrollback survives, all sessions re-adopt with zero orphans, and terminals stay interactive afterward. Direction B (reboot resume hardening: one-click restore-all + the stale conversation-id capture bug) is still open and remains the reboot answer.