Real-time voice agent (OpenAI Realtime API) over Loop threads

[devin-ai-integration]

Summary

Adds a real-time voice interface (OpenAI Realtime API) that acts as a conversational router on top of the existing Loop thread/sub-agent system. The voice agent does not reason long-running itself — it creates/continues threads, hands work off to background agents, checks status, and speaks async updates back into the live conversation. Everything runs on-device; there is no new orchestration backend.

The core mapping is intentionally thin — a "thread" is a SimpleConversation, and "sending a message" dispatches a SubAgent (exactly like the primary chat's spawn_sub_agent). So the voice layer reuses all existing persistence, sub-agent execution, and completion-notification plumbing rather than introducing a parallel store.

New module LoopIOS/VoiceAgent/:

• LoopThreadModels — vocabulary: LoopThreadStatus (queued/running/waiting/complete/failed), LoopThreadUpdate (the LoopThreadUpdated event payload with spokenSummary + shouldInterrupt), and .loopThreadUpdated notification.
• LoopThreadService — the local tool layer. createThread → new conversation + dispatch sub-agent; sendMessage → append + dispatch with thread context; status → resolve from the thread's live sub-agent (falling back to the transcript); plus listThreads / summarize. Tracks the latest sub-agent per thread.
• LoopThreadTools — Realtime function-tool schemas + JSON dispatcher for the 5 tools:
  createLoopThread, sendLoopMessage, getLoopThreadStatus, listLoopThreads, summarizeLoopThread.
• LoopThreadEventBridge — translates the existing .subAgentDidPostMessage completion signal into a speakable .loopThreadUpdated event so background work surfaces in a live session.
• RealtimeVoiceSession — owns the WebSocket to wss://api.openai.com/v1/realtime, sends session.update (server VAD, pcm16 in/out, tools), streams mic audio, plays back audio deltas, dispatches function calls, and injects thread updates as conversation items (with response.cancel + playback flush when shouldInterrupt).
• RealtimeAudioEngine — full-duplex 24 kHz mono PCM16 via AVAudioEngine in .voiceChat mode (echo cancellation), with mic→PCM16 conversion and PCM16→float playback.
• VoiceSessionViewController — full-screen session UI reusing AvatarView, with mute/end controls and mic-permission handling.

Wiring:

MessagingVC.setupNav → nav-bar mic button
micButtonTapped:
    scope = currentConversationEntity != nil && !empty ? .thread(id, title) : .global
    present(VoiceSessionViewController(scope:))

• Home (empty conversation) → .global session (can create/route across threads).
• Thread on screen → .thread session; new messages default to continuing that conversation.

Also: NSMicrophoneUsageDescription added to Info.plist; the 7 new files are excluded from the LoopMac/LoopVision targets (they depend on UIKit/AVAudioSession and iOS-only views).

Deferred / notes

• Uses server VAD (hands-free); no manual push-to-talk.
• summarizeLoopThread is transcript-based (no separate summarization model call yet).
• Update injection is delivered as a tagged user turn ([BACKGROUND UPDATE] …) since the Realtime API conversation items don't take a system role.
• Not yet build-verified — this repo requires macOS/Xcode, which isn't available in this environment. Please build in Xcode; the module is self-contained and additive.
• develop did not exist on the fork theashbhat/LoopHarness and the Devin app lacked write access there, so this PR targets getathelas/LoopHarness:develop per your instruction.

Link to Devin session: https://app.devin.ai/sessions/d7346099fc4947efa319fa90ab05ed20

[devin-ai-integration]

Original prompt from API User

Build a real-time voice agent interface on top of the existing Loop Harness iOS app. The repo is https://github.com/theashbhat/LoopHarness.

#``# Overview

Build a real-time voice interface using the OpenAI Realtime API that sits on top of the existing Loop Harness. The voice agent acts as an intelligent conversational router — it does NOT perform long-running reasoning itself. It can create/manage Loop threads, dispatch work to existing agents, check status of running work, summarize progress conversationally, and update artifacts over time.

The user experiences a continuous voice conversation while asynchronous agent work happens in the background. Everything runs locally on the iPhone — no backend orchestration layer.

#``# Architecture

User → OpenAI Realtime API (Voice Session) → Tool Calls (Realtime) → Local Swift Tool Layer → Loop Harness (Existing Thread System) → OpenAI Responses → Local Thread Updates → Realtime Conversation Injection

The iPhone is responsible for: maintaining Loop state, making OpenAI API requests, exposing tools to the realtime session, and injecting asynchronous updates back into the conversation.

#``# Design Philosophy

The primitive is a message, not a task. Instead of "Create Task", the mental model is "Send a message to another agent." Every thread is simply a conversation. The voice agent is the conversational interface into those conversations. It should never expose implementation details unless explicitly asked.

#``# Threading Rules

• Same Thread: Continue when same artifact, same goal, or refinement of previous work.
• New Thread: Create when new artifact, different objective, or unrelated work.
• If uncertain, ask: "Should I continue the existing thread or create a new one?"

#``# Realtime Tool Interface (MVP first)

Implement these tools to expose to the OpenAI Realtime session:

#``#``# createLoopThread
Input: { title: string, initialMessage: string, agent?: string }
Returns: `{ threadId: string, status: "create... (2161 chars truncated...)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
loop-harness	Ready	Preview, Comment	Jul 9, 2026 6:43pm