mouth: render chunks ahead of playback (no gap between lines); brain: print tool calls - #4
Open
FryD420 wants to merge 5 commits into
Open
mouth: render chunks ahead of playback (no gap between lines); brain: print tool calls#4FryD420 wants to merge 5 commits into
FryD420 wants to merge 5 commits into
Conversation
The mouth was one thread doing render-then-play-then-render, so every chunk boundary paid synth latency plus the 0.75s prebuffer in series (about a second of dead air per boundary) while the text had long since reached the screen. Now a synth thread renders LOOKAHEAD (2) chunks ahead of the one playing; the next chunk's audio is already finished when the previous one ends and plays with no gap. Both audio laws hold: still one long-lived OutputStream, still the prebuffer (satisfied instantly for pre-rendered chunks). A generation counter ties the two threads together for barge-in: shut_up() bumps it and anything ordered under the old generation is dropped wherever it's found, so nothing stale plays; a pending counter replaces "queue empty" for the speaking flag and wait_done(), since the text queue stopped being the whole story. The brain now logs one line per tool call ([tool] Read: <path>, [tool] Bash: <description>, ...) from the AssistantMessage that lands as the call runs, so the terminal shows what the agent is doing while the voice is quiet instead of a silent thinking loop. tests/test_mouth_lookahead.py drives the mouth with a fake synth and a fake real-time output device: boundary gaps under 80ms (measured ~0-1ms), a sub-prebuffer chunk completes, barge-in plays nothing stale and the mouth speaks fresh text afterwards.
…he session The Agent SDK's stream-json reader defaults to 1 MB per message. Reading a 1080p screenshot (~4 MB PNG, ~5 MB base64 on the wire) exceeded it and crashed the voice session. 16 MB gives ~3x headroom over 1080p and covers 4K. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sh' verb When resume_last_session reattaches, the hidden warmup ping becomes a spoken turn: the agent says what was in flight and asks continue or start fresh. "start fresh" / "new session" / "start a new session" are added as synonyms for the clear verb so the answer is natural. Cold launches are unchanged. Docs and the spoken-console discipline text updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…other signal files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aunch right after a fresh start comes up cold Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two quality-of-life changes from daily use of backtalk as a voice line for my agent.
1. Lookahead synthesis in the mouth
Mouthwas one worker thread doing render, play, render, play. Every chunk boundary paid the synth latency plus the 0.75 s prebuffer in series, which on my machine is about a second of dead air between lines while the text had already streamed to the screen.This splits it into a synth thread and a player thread. The synth renders
LOOKAHEAD(2) chunks ahead of the one playing, so when a chunk ends, the next one's audio is already finished and starts with no gap. Both hard-won audio laws still hold: one long-livedOutputStream, and the prebuffer (which is satisfied instantly for chunks that finished rendering while the previous one played).Barge-in: a generation counter ties the two threads together.
shut_up()bumps it, and anything ordered under the old generation is dropped wherever it turns up (text queue, ready queue, mid-render, or in the player), so nothing stale ever plays. A pending counter replaces "queue empty" for thespeakingflag andwait_done(), since the text queue is no longer the whole story once rendering runs ahead.2.
[tool]lines from the brainask_streamnow logs one line per tool call from theAssistantMessagethat lands as the call runs:So the terminal shows what the agent is doing during a long quiet stretch instead of just the thinking sound.
Test
tests/test_mouth_lookahead.pydrives the mouth with a fake synth (0.3 s to first audio, streamed blocks) and a fake real-time output device, no speakers needed:Run with
.venv/Scripts/python tests/test_mouth_lookahead.py(or the posix equivalent). Also verified live: Kokoro voice, Windows 11, and the[tool]lines against a real session.Happy to adjust naming or fold the lookahead depth into
backtalk.jsonif you would rather have it configurable.🤖 Generated with Claude Code