Skip to content

mouth: render chunks ahead of playback (no gap between lines); brain: print tool calls - #4

Open
FryD420 wants to merge 5 commits into
jaredrhod:mainfrom
FryD420:lookahead-and-tool-lines
Open

mouth: render chunks ahead of playback (no gap between lines); brain: print tool calls#4
FryD420 wants to merge 5 commits into
jaredrhod:mainfrom
FryD420:lookahead-and-tool-lines

Conversation

@FryD420

@FryD420 FryD420 commented Aug 21, 2026

Copy link
Copy Markdown

Two quality-of-life changes from daily use of backtalk as a voice line for my agent.

1. Lookahead synthesis in the mouth

Mouth was one worker thread doing render, play, render, play. Every chunk boundary paid the synth latency plus the 0.75 s prebuffer in series, which on my machine is about a second of dead air between lines while the text had already streamed to the screen.

This splits it into a synth thread and a player thread. The synth renders LOOKAHEAD (2) chunks ahead of the one playing, so when a chunk ends, the next one's audio is already finished and starts with no gap. Both hard-won audio laws still hold: one long-lived OutputStream, and the prebuffer (which is satisfied instantly for chunks that finished rendering while the previous one played).

Barge-in: a generation counter ties the two threads together. shut_up() bumps it, and anything ordered under the old generation is dropped wherever it turns up (text queue, ready queue, mid-render, or in the player), so nothing stale ever plays. A pending counter replaces "queue empty" for the speaking flag and wait_done(), since the text queue is no longer the whole story once rendering runs ahead.

2. [tool] lines from the brain

ask_stream now logs one line per tool call from the AssistantMessage that lands as the call runs:

[tool] Read: E:\my-agent\backtalk\LICENSE
[tool] Bash: Check git remote and status
[tool] Grep: say_chunk in backtalk/

So the terminal shows what the agent is doing during a long quiet stretch instead of just the thinking sound.

Test

tests/test_mouth_lookahead.py drives the mouth with a fake synth (0.3 s to first audio, streamed blocks) and a fake real-time output device, no speakers needed:

  • boundary gaps under 80 ms (measured 0 to 1 ms; the old mouth would have been ~1 s)
  • a chunk shorter than the prebuffer still completes
  • barge-in cuts mid-chunk, plays nothing stale, and the mouth speaks fresh text afterwards

Run with .venv/Scripts/python tests/test_mouth_lookahead.py (or the posix equivalent). Also verified live: Kokoro voice, Windows 11, and the [tool] lines against a real session.

Happy to adjust naming or fold the lookahead depth into backtalk.json if you would rather have it configurable.

🤖 Generated with Claude Code

FryD420 and others added 5 commits August 21, 2026 01:12
The mouth was one thread doing render-then-play-then-render, so every
chunk boundary paid synth latency plus the 0.75s prebuffer in series
(about a second of dead air per boundary) while the text had long since
reached the screen. Now a synth thread renders LOOKAHEAD (2) chunks
ahead of the one playing; the next chunk's audio is already finished
when the previous one ends and plays with no gap. Both audio laws hold:
still one long-lived OutputStream, still the prebuffer (satisfied
instantly for pre-rendered chunks). A generation counter ties the two
threads together for barge-in: shut_up() bumps it and anything ordered
under the old generation is dropped wherever it's found, so nothing
stale plays; a pending counter replaces "queue empty" for the speaking
flag and wait_done(), since the text queue stopped being the whole story.

The brain now logs one line per tool call ([tool] Read: <path>,
[tool] Bash: <description>, ...) from the AssistantMessage that lands as
the call runs, so the terminal shows what the agent is doing while the
voice is quiet instead of a silent thinking loop.

tests/test_mouth_lookahead.py drives the mouth with a fake synth and a
fake real-time output device: boundary gaps under 80ms (measured ~0-1ms),
a sub-prebuffer chunk completes, barge-in plays nothing stale and the
mouth speaks fresh text afterwards.
…he session

The Agent SDK's stream-json reader defaults to 1 MB per message. Reading a
1080p screenshot (~4 MB PNG, ~5 MB base64 on the wire) exceeded it and
crashed the voice session. 16 MB gives ~3x headroom over 1080p and covers 4K.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sh' verb

When resume_last_session reattaches, the hidden warmup ping becomes a
spoken turn: the agent says what was in flight and asks continue or start
fresh. "start fresh" / "new session" / "start a new session" are added
as synonyms for the clear verb so the answer is natural. Cold launches
are unchanged. Docs and the spoken-console discipline text updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…other signal files

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aunch right after a fresh start comes up cold

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant