Repository navigation
feat(codex): add subscription-backed live voice conversations - #36195
romil-bijarnia wants to merge 1 commit into
Conversation
|
Closing: romil-bijarnia already has 3 or more open PRs here, which is the limit for contributors who are not vouched. Once one of them is merged or closed, open this again. See the open PR limit. |
| return yield* (yield* load(threadId)).voice.options; | ||
| }), | ||
| start: Effect.fn("LiveVoice.start")(function* (input, owner) { | ||
| if (entries.has(input.sessionId) || byThread.has(input.threadId)) |
There was a problem hiding this comment.
🟠 High liveVoice/LiveVoice.ts:258
Concurrent start() calls using the same sessionId on different threads can both open provider connections, but the later call overwrites the earlier entry in entries; the first connection then cannot be polled or stopped and is skipped by expiry cleanup. start() reserves threadId before load() yields but does not reserve sessionId; reserve the session ID before yielding and release it if startup fails.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/liveVoice/LiveVoice.ts around line 258:
Concurrent `start()` calls using the same `sessionId` on different threads can both open provider connections, but the later call overwrites the earlier entry in `entries`; the first connection then cannot be polled or stopped and is skipped by expiry cleanup. `start()` reserves `threadId` before `load()` yields but does not reserve `sessionId`; reserve the session ID before yielding and release it if startup fails.
There was a problem hiding this comment.
Fixed on the source branch in 4bbae521e7. Session IDs are reserved across threads before initial validation can suspend, and the reservation survives startup cleanup until it is released. The regression holds the first snapshot read and tries the same ID from a different thread: it was accepted before and now returns busy. The cancellation control also verifies that the ID becomes reusable after failed startup.
This closed PR is still frozen at its old head; the tested fix is visible in the fork review copy. Leaving the thread open until the upstream head can be updated.
There was a problem hiding this comment.
Sorry, I'm unable to act on this request because you do not have permissions within this repository.
| const liveVoice = yield* makeCodexLiveVoice( | ||
| resolveRuntime.pipe( | ||
| Effect.flatMap((effective) => | ||
| runtime.auth.access.pipe( |
There was a problem hiding this comment.
🔴 Critical Drivers/CodexManagedProvider.ts:283
If the ChatGPT profile changes after resolveRuntime completes, liveVoice combines the old effective runtime with the new profile's token, so LiveVoice.start can send the conversation context using credentials for a different account. The access read is outside the runtime's profile binding and does not check effective.revision; acquire the credential under the same auth admission and verify runtime.auth.profileKey(credentials) === effective.revision before opening the connection.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/Drivers/CodexManagedProvider.ts around line 283:
If the ChatGPT profile changes after `resolveRuntime` completes, `liveVoice` combines the old `effective` runtime with the new profile's token, so `LiveVoice.start` can send the conversation context using credentials for a different account. The access read is outside the runtime's profile binding and does not check `effective.revision`; acquire the credential under the same auth admission and verify `runtime.auth.profileKey(credentials) === effective.revision` before opening the connection.
There was a problem hiding this comment.
Fixed on the source branch in 4bbae521e7. Runtime resolution and credential acquisition now share one auth admission and require the credential profile key to match the runtime revision. The admitted child scope remains attached to the acquired voice connection, so replacing credentials closes its resources. Options release their admission when the request ends.
Three controlled regressions failed before and now pass: an external profile change after runtime resolution, credential adoption during the access read, and connection cleanup on replacement. They use dummy credentials and the actual auth controller, not live account switching. This does not establish managed SIWC service availability.
The closed upstream diff is frozen at the old head, so I am leaving this thread open; fork review #1 contains the tested correction.
There was a problem hiding this comment.
Sorry, I'm unable to act on this request because you do not have permissions within this repository.
| if (notify) this.dependencies.onPhase("idle", null); | ||
| } | ||
|
|
||
| mute(muted: boolean) { |
There was a problem hiding this comment.
🟠 High speech/browserLiveVoice.ts:52
Clicking Mute while microphone selection or permission is pending leaves the newly acquired tracks enabled, so the call transmits audio while the UI reports “Microphone muted.” mute() only updates this.stream, and start() does not reapply the requested state when it assigns the stream; persist the mute state and apply it before adding the tracks.
Also found in 1 other location(s)
apps/web/src/components/chat/LiveVoiceButton.tsx:194
Mute is enabled during
connecting, butBrowserLiveVoice.mute()only toggles tracks already inthis.streamand does not remember the requested state. If the user clicks Mute while microphone selection/permission is pending, there are no tracks yet;start()subsequently attaches the microphone without muting it. The dialog continues to display “Microphone muted” while transmitting audio. Disable mute until the stream exists or persist and apply the mute state when acquiring the stream.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/web/src/speech/browserLiveVoice.ts around line 52:
Clicking Mute while microphone selection or permission is pending leaves the newly acquired tracks enabled, so the call transmits audio while the UI reports “Microphone muted.” `mute()` only updates `this.stream`, and `start()` does not reapply the requested state when it assigns the stream; persist the mute state and apply it before adding the tracks.
Also found in 1 other location(s):
- apps/web/src/components/chat/LiveVoiceButton.tsx:194 -- Mute is enabled during `connecting`, but `BrowserLiveVoice.mute()` only toggles tracks already in `this.stream` and does not remember the requested state. If the user clicks Mute while microphone selection/permission is pending, there are no tracks yet; `start()` subsequently attaches the microphone without muting it. The dialog continues to display “Microphone muted” while transmitting audio. Disable mute until the stream exists or persist and apply the mute state when acquiring the stream.
There was a problem hiding this comment.
Fixed on the source branch in 4bbae521e7. The browser controller remembers requested mute while device selection or permission is pending and applies it before adding microphone tracks to the peer. Stop resets the state for the next call.
Both pending phases failed before the fix with an enabled track at addTrack. They now pass with the track disabled before attachment, followed by unmute/remute and cancellation controls. No physical microphone was used.
This closed PR still displays the old head, so the thread remains open pending an upstream update. The fix is present in fork review #1.
There was a problem hiding this comment.
Sorry, I'm unable to act on this request because you do not have permissions within this repository.
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — Adds opt-in subscription-backed voice conversations with coding-task delegation. The new lint suppression requires human review, and unresolved findings identify cross-account context exposure, ineffective muting during startup, and orphaned sessions. Needs human approval because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
Review fixes are pushed to the source branch at 4bbae521e7. This PR is closed by the open-PR limit, so GitHub still displays its earlier head
dd786f08ca. The updated code is visible in fork review #1. The verification below applies to the corrected branch, not the frozen closed diff.The tested base has dictation, but no two-way voice conversation or Aube picker. This adds experimental live voice to saved Codex conversations on web and desktop, using the ChatGPT account already connected to the selected provider. It does not require an API key or a separate voice login.
The server discovers the backend's V3 voice catalogue, owns call authorization and lifecycle, and sends spoken coding requests through normal T3 orchestration. Clients handle microphone selection, playback, mute, stop and captions. Startup reserves session IDs across threads before validation can suspend. Managed connections bind runtime resolution and credentials under one auth admission, reject profile-revision mismatches, and retain that admission for the socket lifetime. Client mute requests persist through microphone selection and permission and are applied before audio tracks are attached. Ending a call does not cancel coding tasks already dispatched. Native mobile dictation is unchanged.
This is an alternative implementation to upstream #29394, not a claim that no related work exists. Unlike that implementation's native-turn interruption/resubmission path, T3 owns the voice delegation before a coding turn starts. A separate repair to the existing implementation is available in jakeleventhal/t3code#7.
Verified on Linux with unmodified Codex 0.167.2 and an existing ChatGPT-backed T3 provider:
gpt-6-lunatask, saved its request andVOICE_HANDOFF_CONFIRMEDreply in the same thread, and spoke the result back. No registry, projection or orchestrator mocks were used. Decoded audio was nonzero, no realtime errors occurred, and stop completed cleanly.vp run check:changed 2d8a843e6be9ca170f3bce98cc92ad76617b00c4 --concurrency 2passed formatting, lint, both Knip passes, GraphQL, 38 affected-package typechecks, 36 tests in seven selected files and three React Compiler checks. This is not a repo-wide suite.Limits: only Sol and Aube received real audio tests; catalogue discovery is not an entitlement guarantee for every profile. Managed SIWC token-sharing voice, native mobile calls, remote/relay/tunnel operation and physical microphone/speaker quality remain unverified. Rejected account routes report errors rather than switching billing, accounts or auth methods.
The before screenshot is tested base
2d8a843e6brendering an isolated copy of the same harmless saved test conversation. All state and account fixtures were separate from the installed nightly and live chats. Screenshots and the credential-free acceptance result are linked below, not committed to source.Updated credential-free end-to-end result, rerun after the review fixes: real Aube audio, exactly one new same-thread Codex run, saved reply and spoken result; no realtime errors.
Current main, same saved test conversation:
Alternative with Aube selected:
Discovered voice profiles: