Skip to content

feat(codex): add subscription-backed live voice conversations - #36195

Closed
romil-bijarnia wants to merge 1 commit into
pingdotgg:mainfrom
romil-bijarnia:feat/live-voice-current-main
Closed

romil-bijarnia wants to merge 1 commit into
pingdotgg:mainfrom
romil-bijarnia:feat/live-voice-current-main

Conversation

@romil-bijarnia

@romil-bijarnia romil-bijarnia commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Review fixes are pushed to the source branch at 4bbae521e7. This PR is closed by the open-PR limit, so GitHub still displays its earlier head dd786f08ca. The updated code is visible in fork review #1. The verification below applies to the corrected branch, not the frozen closed diff.

The tested base has dictation, but no two-way voice conversation or Aube picker. This adds experimental live voice to saved Codex conversations on web and desktop, using the ChatGPT account already connected to the selected provider. It does not require an API key or a separate voice login.

The server discovers the backend's V3 voice catalogue, owns call authorization and lifecycle, and sends spoken coding requests through normal T3 orchestration. Clients handle microphone selection, playback, mute, stop and captions. Startup reserves session IDs across threads before validation can suspend. Managed connections bind runtime resolution and credentials under one auth admission, reject profile-revision mismatches, and retain that admission for the socket lifetime. Client mute requests persist through microphone selection and permission and are applied before audio tracks are attached. Ending a call does not cancel coding tasks already dispatched. Native mobile dictation is unchanged.

This is an alternative implementation to upstream #29394, not a claim that no related work exists. Unlike that implementation's native-turn interruption/resubmission path, T3 owns the voice delegation before a coding turn starts. A separate repair to the existing implementation is available in jakeleventhal/t3code#7.

Verified on Linux with unmodified Codex 0.167.2 and an existing ChatGPT-backed T3 provider:

  • The real client opened the live-voice dialog, loaded 25 discovered profiles and selected Aube. No physical microphone was opened for the screenshot check.
  • An actual HTTP/WebRTC call sent synthetic speech to Aube, dispatched exactly one real gpt-6-luna task, saved its request and VOICE_HANDOFF_CONFIRMED reply in the same thread, and spoke the result back. No registry, projection or orchestrator mocks were used. Decoded audio was nonzero, no realtime errors occurred, and stop completed cleanly.
  • vp run check:changed 2d8a843e6be9ca170f3bce98cc92ad76617b00c4 --concurrency 2 passed formatting, lint, both Knip passes, GraphQL, 38 affected-package typechecks, 36 tests in seven selected files and three React Compiler checks. This is not a repo-wide suite.
  • Cancellation before microphone selection and stopping during thread validation were reproduced and fixed; the latter regression previously dispatched a task after the call ended.
  • Six additional race regressions failed before the review fixes and now pass: three managed-account binding/lifetime cases, both pending-microphone mute phases, and a duplicate session ID across different threads. Account races use dummy profiles and the real auth admission; no actual user accounts were switched.

Limits: only Sol and Aube received real audio tests; catalogue discovery is not an entitlement guarantee for every profile. Managed SIWC token-sharing voice, native mobile calls, remote/relay/tunnel operation and physical microphone/speaker quality remain unverified. Rejected account routes report errors rather than switching billing, accounts or auth methods.

The before screenshot is tested base 2d8a843e6b rendering an isolated copy of the same harmless saved test conversation. All state and account fixtures were separate from the installed nightly and live chats. Screenshots and the credential-free acceptance result are linked below, not committed to source.

Updated credential-free end-to-end result, rerun after the review fixes: real Aube audio, exactly one new same-thread Codex run, saved reply and spoken result; no realtime errors.

Current main, same saved test conversation:

Before: current main has dictation but no live-voice control

Alternative with Aube selected:

After: live-voice dialog with Aube selected

Discovered voice profiles:

Backend-discovered voice profiles including Aube

@github-actions

Copy link
Copy Markdown
Contributor

Closing: romil-bijarnia already has 3 or more open PRs here, which is the limit for contributors who are not vouched.

Once one of them is merged or closed, open this again. See the open PR limit.

@github-actions github-actions Bot closed this Nov 13, 2026
@github-actions github-actions Bot added vouch:unvouched Author is not yet trusted in the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Nov 13, 2026
return yield* (yield* load(threadId)).voice.options;
}),
start: Effect.fn("LiveVoice.start")(function* (input, owner) {
if (entries.has(input.sessionId) || byThread.has(input.threadId))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High liveVoice/LiveVoice.ts:258

Concurrent start() calls using the same sessionId on different threads can both open provider connections, but the later call overwrites the earlier entry in entries; the first connection then cannot be polled or stopped and is skipped by expiry cleanup. start() reserves threadId before load() yields but does not reserve sessionId; reserve the session ID before yielding and release it if startup fails.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/liveVoice/LiveVoice.ts around line 258:

Concurrent `start()` calls using the same `sessionId` on different threads can both open provider connections, but the later call overwrites the earlier entry in `entries`; the first connection then cannot be polled or stopped and is skipped by expiry cleanup. `start()` reserves `threadId` before `load()` yields but does not reserve `sessionId`; reserve the session ID before yielding and release it if startup fails.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed on the source branch in 4bbae521e7. Session IDs are reserved across threads before initial validation can suspend, and the reservation survives startup cleanup until it is released. The regression holds the first snapshot read and tries the same ID from a different thread: it was accepted before and now returns busy. The cancellation control also verifies that the ID becomes reusable after failed startup.

This closed PR is still frozen at its old head; the tested fix is visible in the fork review copy. Leaving the thread open until the upstream head can be updated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

const liveVoice = yield* makeCodexLiveVoice(
resolveRuntime.pipe(
Effect.flatMap((effective) =>
runtime.auth.access.pipe(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical Drivers/CodexManagedProvider.ts:283

If the ChatGPT profile changes after resolveRuntime completes, liveVoice combines the old effective runtime with the new profile's token, so LiveVoice.start can send the conversation context using credentials for a different account. The access read is outside the runtime's profile binding and does not check effective.revision; acquire the credential under the same auth admission and verify runtime.auth.profileKey(credentials) === effective.revision before opening the connection.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/Drivers/CodexManagedProvider.ts around line 283:

If the ChatGPT profile changes after `resolveRuntime` completes, `liveVoice` combines the old `effective` runtime with the new profile's token, so `LiveVoice.start` can send the conversation context using credentials for a different account. The access read is outside the runtime's profile binding and does not check `effective.revision`; acquire the credential under the same auth admission and verify `runtime.auth.profileKey(credentials) === effective.revision` before opening the connection.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed on the source branch in 4bbae521e7. Runtime resolution and credential acquisition now share one auth admission and require the credential profile key to match the runtime revision. The admitted child scope remains attached to the acquired voice connection, so replacing credentials closes its resources. Options release their admission when the request ends.

Three controlled regressions failed before and now pass: an external profile change after runtime resolution, credential adoption during the access read, and connection cleanup on replacement. They use dummy credentials and the actual auth controller, not live account switching. This does not establish managed SIWC service availability.

The closed upstream diff is frozen at the old head, so I am leaving this thread open; fork review #1 contains the tested correction.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

if (notify) this.dependencies.onPhase("idle", null);
}

mute(muted: boolean) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High speech/browserLiveVoice.ts:52

Clicking Mute while microphone selection or permission is pending leaves the newly acquired tracks enabled, so the call transmits audio while the UI reports “Microphone muted.” mute() only updates this.stream, and start() does not reapply the requested state when it assigns the stream; persist the mute state and apply it before adding the tracks.

Also found in 1 other location(s)

apps/web/src/components/chat/LiveVoiceButton.tsx:194

Mute is enabled during connecting, but BrowserLiveVoice.mute() only toggles tracks already in this.stream and does not remember the requested state. If the user clicks Mute while microphone selection/permission is pending, there are no tracks yet; start() subsequently attaches the microphone without muting it. The dialog continues to display “Microphone muted” while transmitting audio. Disable mute until the stream exists or persist and apply the mute state when acquiring the stream.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/web/src/speech/browserLiveVoice.ts around line 52:

Clicking Mute while microphone selection or permission is pending leaves the newly acquired tracks enabled, so the call transmits audio while the UI reports “Microphone muted.” `mute()` only updates `this.stream`, and `start()` does not reapply the requested state when it assigns the stream; persist the mute state and apply it before adding the tracks.

Also found in 1 other location(s):
- apps/web/src/components/chat/LiveVoiceButton.tsx:194 -- Mute is enabled during `connecting`, but `BrowserLiveVoice.mute()` only toggles tracks already in `this.stream` and does not remember the requested state. If the user clicks Mute while microphone selection/permission is pending, there are no tracks yet; `start()` subsequently attaches the microphone without muting it. The dialog continues to display “Microphone muted” while transmitting audio. Disable mute until the stream exists or persist and apply the mute state when acquiring the stream.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed on the source branch in 4bbae521e7. The browser controller remembers requested mute while device selection or permission is pending and applies it before adding microphone tracks to the peer. Stop resets the state for the next call.

Both pending phases failed before the fix with an enabled track at addTrack. They now pass with the track disabled before attachment, followed by unmute/remute and cancellation controls. No physical microphone was used.

This closed PR still displays the old head, so the thread remains open pending an upstream update. The fix is present in fork review #1.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

@macroscopeapp

macroscopeapp Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — Adds opt-in subscription-backed voice conversations with coding-task delegation. The new lint suppression requires human review, and unresolved findings identify cross-account context exposure, ineffective muting during startup, and orphaned sessions.

Needs human approval because:

  • 3 blocking correctness issues found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched Author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant