Add live audio transcription streaming support to Foundry Local JS SDK

[rui-ren]

Here's the updated PR description with the renamed types:

---

Title: Add live audio transcription streaming support to Foundry Local JS SDK

Description:

Adds real-time audio streaming support to the Foundry Local JS SDK, enabling live microphone-to-text transcription via ONNX Runtime GenAI ASR.

The existing AudioClient only supports file-based transcription. This PR introduces LiveAudioTranscriptionClient that accepts continuous PCM audio chunks (e.g., from a microphone) and returns partial/final transcription results as an async iterable.

What's included

New files

• src/openai/liveAudioTranscriptionClient.ts — Streaming client with start(), pushAudioData(), getTranscriptionStream(), stop(), dispose()
• src/openai/liveAudioTranscriptionTypes.ts — LiveAudioTranscriptionResult and CoreErrorResponse interfaces, tryParseCoreError() helper

Modified files

• src/imodel.ts — Added createLiveTranscriptionClient() to interface
• src/model.ts — Delegates to selectedVariant.createLiveTranscriptionClient()
• src/modelVariant.ts — Implementation (creates new LiveAudioTranscriptionClient(modelId, coreInterop))
• src/index.ts — Exports LiveAudioTranscriptionClient, LiveAudioTranscriptionSettings, LiveAudioTranscriptionResult, CoreErrorResponse

API surface

const audioClient = model.createAudioClient();
const session = model.createLiveTranscriptionClient();

session.settings.sampleRate = 16000;
session.settings.channels = 1;
session.settings.language = "en";

await session.start();

// Push audio from microphone callback
await session.pushAudioData(pcmBytes);

// Read results as async iterable
for await (const result of session.getTranscriptionStream()) {
    console.log(result.text);
}

await session.stop();

Design highlights

• Internal async push queue — Bounded AsyncQueue<T> serializes audio pushes from any context (safe for mic callbacks) and provides backpressure. Mirrors C#'s Channel<T> pattern.
• Retry policy — Transient native errors retried with exponential backoff (3 attempts); permanent errors terminate the session
• Settings freeze — Audio format settings are snapshot-copied and Object.freeze()d at start(), immutable during the session
• Buffer copy — pushAudioData() copies the input Uint8Array before queueing, safe when caller reuses buffers
• Drain-on-stop — stop() completes the push queue, waits for the push loop to drain, then calls native stop
• Dispose safety — dispose() wraps stop() in try/catch, never throws

Native core dependency

This PR adds the JS SDK surface. The 3 native commands (audio_stream_start, audio_stream_push, audio_stream_stop) are routed through the existing execute_command / execute_command_with_binary exports. The code compiles with zero TypeScript errors without the native library.

Testing

• ✅ TypeScript compilation — 0 errors across all source files
• ⏳ Integration tests pending native core delivery

Parity with C# SDK

This implementation mirrors the C# LiveAudioTranscriptionSession (branch ruiren/audio-streaming-support-sdk) with identical logic:

• Same session lifecycle: start → push → getStream → stop
• Same push loop with retry and permanent error handling
• Same settings freeze and buffer copy semantics
• Same drain-before-stop ordering
• Same renamed types: LiveAudioTranscription* (matching C# rename)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Error		Mar 26, 2026 11:30pm

[kunal-vaishnavi]

Can you add an E2E example in the samples folder using the JS SDK?

[kunal-vaishnavi]

There are some Copilot-raised issues with this file.

---

Major issues / likely bugs

1) audio_stream_push does not appear to send actual audio bytes (only length)

In pushLoop(), the command payload includes SessionHandle and AudioDataLength, but not the audioData buffer itself:

this.coreInterop.executeCommand("audio_stream_push", {
  Params: {
    SessionHandle: this.sessionHandle!,
    AudioDataLength: audioData.length.toString()
  }
});

Unless CoreInterop.executeCommand implicitly reads from some shared memory or side channel (not shown here), this looks like a functional bug: the core can’t transcribe audio it never receives. At minimum, this deserves a comment explaining the data path; ideally, pass bytes explicitly (e.g., AudioDataBase64, AudioData, or a binary channel supported by CoreInterop).

Recommendation: confirm the expected contract for "audio_stream_push" and update the call to include the buffer (or document how CoreInterop transfers it).

---

2) AsyncQueue backpressure is not safe for “multiple producers”

The comment says: “multiple producers writing” but the implementation has a single backpressureResolve slot:

• If the queue is full, write() stores this.backpressureResolve = resolve.
• If two producers call write() while full, the second overwrites the first resolver and the first producer can hang forever.

Also, tryWrite() ignores maxCapacity entirely and can grow memory unbounded even when a capacity was configured—this defeats the point of pushQueueCapacity.

Recommendation (choose one):

• Make AsyncQueue explicitly single-producer and enforce/document it; or
• Fix it for real multi-producer backpressure by maintaining a FIFO of pending resolvers (e.g., backpressureResolvers: Array<() => void>), and enforce capacity in tryWrite (return false when full).

---

3) Potential “lost wakeup” / fairness issues in backpressure release

The consumer iterator releases backpressure when queue.length < maxCapacity, but only one waiter can be released (backpressureResolve is a single slot). Even with a single producer, this can create awkward behavior under bursty load; with multiple producers it’s incorrect as noted above.

---

Lifecycle / concurrency concerns

Stop ordering is a bit odd (abort happens after draining)

stop() does:

1. this.pushQueue?.complete() (drain then end)
2. await this.pushLoopPromise
3. this.sessionAbortController?.abort()

But abort() can’t influence the loop if you’ve already awaited it. If your intent is “stop now, don’t drain”, you’d abort first. If your intent is “drain then stop”, you don’t need abort at all (except maybe to stop other background tasks).

Recommendation: decide semantics and align:

• Drain semantics: remove abort checks from push loop, or keep them but don’t call abort() (or call it earlier only for external cancellation).
• Immediate stop semantics: abort() first, then pushQueue.complete() (optionally with an error), then await pushLoopPromise.

Push calls during/after stop

pushAudioData() checks this.stopped, so once stop() sets stopped = true, further pushes throw (good). But if someone calls pushAudioData() concurrently right as stop begins, it may pass the check and then block forever if the queue is completed (your write() throws if completed, so it should reject—good), but consider race windows if pushQueue becomes null in the future refactor.

---

Error handling / API ergonomics

• start() catch calls this.outputQueue.complete() but doesn’t pass the error. That means consumers see a normal completion rather than failure if they started reading early.
  • Recommendation: this.outputQueue.complete(err) so getTranscriptionStream() fails loudly.
• stop() always this.outputQueue?.complete() even if there was a fatal push error already set. Because AsyncQueue.complete() no-ops if already completed, this is fine.
• Consider surfacing the native error in a typed way rather than embedding parsed code into a string. You already have tryParseCoreError(); you could add a custom error class with code field.

---

Performance considerations

• AsyncQueue uses Array.shift() which is O(n) per item due to reindexing. For audio streaming, this could become a hotspot.
  • Recommendation: implement the queue as a ring buffer or keep a headIndex and occasionally compact.
• pushAudioData() always copies the buffer. This is safe but can be expensive at high throughput.
  • Recommendation: if callers already supply immutable chunks, consider an opt-in “no copy” mode (documented), or accept ArrayBuffer and slice appropriately.

---

Minor style / maintainability notes

• LiveAudioTranscriptionSettings.snapshot() returns Object.freeze(copy) as LiveAudioTranscriptionSettings; consider returning Readonly<LiveAudioTranscriptionSettings> for better typing.
• AsyncQueue.error getter is unused in this file; either use it or remove it to reduce surface area.
• console.error / console.warn in a library can be noisy. Consider injecting a logger or using an internal debug hook.

---

Suggested concrete changes (priority order)

1. Fix/confirm audio bytes transport in "audio_stream_push" (most critical).
2. Fix AsyncQueue multi-producer backpressure (or document it’s single-producer), and enforce capacity in tryWrite.
3. Replace Array.shift() with a more efficient queue structure.
4. Propagate startup errors to stream consumers by completing outputQueue with the error.
5. Clarify stop semantics (drain vs immediate cancel) and adjust abort usage accordingly.

[kunal-vaishnavi]

Shouldn't this be executeCommandWithBinary instead? You may need to modify interop for JS as you did for C# to add it.