fix(sidecar): enforce one sidecar per agent per host (CHOO-2653) - #401
Open
abeldantas wants to merge 4 commits into
Open
abeldantas wants to merge 4 commits into
abeldantas wants to merge 4 commits into
Conversation
Console adopting a remote host that already has a sidecar deployed a second one. Both supervised the same sessions, derived the same connection ids, and evicted each other's event streams in a no-backoff loop (~9 reattaches/second). Layer 2 — single-instance guard on the sidecar itself: On startup, read the existing ready file, check its PID, probe its HTTP port. If a healthy sidecar responds, exit cleanly. Layer 1 — broader detection in the launcher: readRunning() now also checks the ready file's PID directly, not just tmux by name. reapStaleSidecarsForAgent() is called before ensureAgentSidecar() so leftover generations are cleaned up before a new one is deployed. Layer 3 — eviction backoff (client + server): Client: SwitchEventStream tracks eviction and applies the same exponential backoff the error path uses, breaking the zero-delay reconnect loop. Server: ConnectionRegistry.open() refuses a reattach within 2s of the previous one, and the per-reattach log is moved from INFO to DEBUG so a storm cannot dominate the service log.
Both branches returned null — an editing leftover from the PID-fallback refactor.
abeldantas
requested review from
amaudruz and
christian-mcdermott
as code owners
September 8, 2026 20:27
Layer 2: existingSidecarIsHealthy — ready file missing, dead PID, live PID with unresponsive port, live PID with healthy response, and self-PID detection. Layer 3 server: ReattachTooSoonError — first reattach succeeds, immediate second is refused, reattach after the interval passes. Extracted existingSidecarIsHealthy to its own module so it can be imported without triggering the sidecar's main().
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new sidecar single-instance health probe appears to authenticate with the wrong header and will likely fail to detect an already-running sidecar, undermining the PR’s primary guardrail.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR hardens the Switch agent sidecar + event-stream stack against duplicate instances by adding server/client reattach throttling, eviction backoff, and more robust “already running” detection/cleanup across Console’s remote sidecar management paths.
Changes:
- Add a server-side minimum reattach interval to prevent rapid reattach/evict loops on the same connection id.
- Add client-side exponential backoff when an SSE stream ends via eviction, avoiding immediate reconnect hammering.
- Add single-instance guarding and improved stale-sidecar reaping/detection before deploying/ensuring sidecars.
File summaries
| File | Description |
|---|---|
| core/switch_core/bridges/agent/protocol/connections.py | Adds reattach-rate limiting and lowers reattach log noise. |
| console/packages/switch-agent-runtime/src/event-stream.ts | Backs off reconnects after eviction frames to prevent livelock. |
| console/apps/switch-console-desktop/src/sidecar/index.ts | Adds a local single-instance health check before sidecar startup. |
| console/apps/switch-console-desktop/src/main/core/sidecar/controller.ts | Reaps stale sidecars before upgrade/restart operations. |
| console/apps/switch-console-desktop/src/main/core/agents/remote-watcher.ts | Reaps stale sidecars before deploying in watcher ensure flow. |
| console/apps/switch-console-desktop/src/main/core/agent-runtime/impl/ssh-agent-runtime.ts | Reaps stale sidecars before coalesced ensure/deploy on SSH runtime startup. |
| console/apps/switch-console-desktop/src/main/core/agent-runtime/impl/remote-sidecar-launcher.ts | Extends “running sidecar” detection to include PID-alive fallback when tmux misses. |
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…D identity - single-instance guard sent Authorization: Bearer but the hook server gates on x-switchdash-token, so the probe always 403'd and the guard never detected a healthy sidecar - readRunning's tmux-missing path now checks the PID's cmdline before trusting it (recycled-PID false positive); a wrong no is safe since relaunch hits the sidecar's own guard - ReattachTooSoonError: param renamed to elapsed_seconds, message and field expose retry_after_seconds - oxfmt on sidecar/index.ts (the failing CI gate)
abeldantas
force-pushed
the
work/one-sidecar-per-agent-host
branch
from
September 8, 2026 23:25
5f1f303 to
ea3705a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When two Switch Console installs share a remote host, or when someone manually starts a sidecar while the Console is also managing one, two sidecar processes end up supervising the same agent sessions. They derive the same connection ids, so each one's event stream evicts the other's — in a loop, at ~9 reattaches per second. The agent looks dead and the server log fills up.
This fixes it by making sure only one sidecar runs per agent per host, and by adding backoff so even if a duplicate somehow starts, it settles instead of storming. No negative externalities.
The fix will only impact edge cases where people have multiple envs/consoles and are bootstrapping agents directly.
But it doesn't negatively impact regular usage.