Voice orb scales from mic input before connection is ready (all providers)

[nithiink]

Summary

The voice orb / blob (--amp volume animation) begins scaling from live mic input before the voice connection is fully established. This affects all providers (OpenAI, Azure, Gemini) — same root cause, different window sizes. The orb should stay at rest until the session is genuinely ready to talk.

Root cause

The orb's volume scaling is gated on connectedRef.current, which mirrors the connected state flag:

• frontend/components/VoiceAgent.tsx:1000 — orbLoop only accumulates audio RMS into --amp when connectedRef.current is true.
• frontend/components/VoiceAgent.tsx:1195 — connected is set to true immediately after await s.start() resolves in connect().

The bug is that start() resolving does not mean the session is ready — in every transport, start() returns before the application-layer handshake that actually makes the session usable completes. Meanwhile the mic is already attached to the orb analyser earlier in start() (via onLocalStream), and the rAF loop is running. So the moment connected flips true, the gate opens and the orb scales from mic input during the still-in-progress handshake.

Gemini (GeminiSession) — largest window, default provider

• start() (frontend/lib/gemini.ts:159-170) awaits initAudio() then openSocket() and returns.
• initAudio() calls onLocalStream (gemini.ts:308), attaching the mic to the orb analyser.
• openSocket() (gemini.ts:174-203) only creates the WebSocket and attaches listeners, then returns. It does not await the socket opening or the setup → setupComplete handshake. setupDone flips true only later, when setupComplete arrives (gemini.ts:439-444).
• So start() resolves → connected = true → orb gate opens, all while setupDone is still false.

OpenAI + Azure (RealtimeSession) — smaller window, same defect

• start() (frontend/lib/realtime.ts:82-162) attaches the mic via onLocalStream (realtime.ts:137), then performs the SDP offer/answer exchange and resolves right after setRemoteDescription (realtime.ts:158), synchronously emitting state: "listening" (realtime.ts:161).
• But the data channel open event → configureSession() (realtime.ts:142) fires asynchronously, after ICE connectivity is established — i.e. after start() has already resolved and connected is true.
• So connected flips true before the data channel is open and configureSession() (session.update with instructions/tools) has run — the session isn't actually ready yet.
• Note: this path also emits its state: "listening" ready-signal prematurely (line 161, before the channel opens), so gating the orb on vstate alone would not fix OpenAI/Azure.

In short: the gate conflates "start() returned" with "session ready". That equivalence holds for no provider — each has a post-start() handshake (setupComplete for Gemini; data-channel open → configureSession for OpenAI/Azure).

Suggested direction (not implemented yet)

Open the orb gate only on a true, provider-reported ready signal — and make each provider emit that signal at its genuine ready point:

• Gemini: real ready = setupComplete (gemini.ts:439-444, already emits state: "listening" there). Either await setupComplete before start() resolves, or don't set connected/open the gate until it fires.
• OpenAI/Azure: move the ready signal out of start() (realtime.ts:160-161) into the data channel open handler (realtime.ts:142), so "ready" reflects the channel actually being open and the session configured.
• Then have VoiceAgent open the orb gate (and ideally flip connected) on that unified ready signal rather than on start() returning — e.g. gate orbLoop on vstate having reached the ready state once every provider emits it at the correct moment, or introduce an explicit onReady callback.

Notes

• Default provider is Gemini, so the most-visible/largest window is the common case, but all three providers are affected.
• Add/adjust a test around the orb gate timing (e.g. assert --amp stays 0 until the ready signal) when fixing.

No PR yet — filing to track.