feat(rt+voice): FEOU→EOU latency compensation - #125
Draft
sam-s10s wants to merge 2 commits into
Draft
Conversation
Inject silence ahead of real time to align each forced EOU with the engine's decode-chunk grid, so the covering chunk completes immediately instead of waiting real-time for the inter-utterance gap. Measured ~50% reduction in FEOU->EndOfUtterance latency with the transcript unchanged. sdk/rt: - force_end_of_utterance(compensate_latency=...) injects one block of PCM silence up to the next decode-chunk boundary + margin, then sends the FEOU at the real speech end (unmoved). Explicit timestamps are treated as real-audio time and shifted onto the server timeline. - Track cumulative injected silence; adjust_timestamp() maps server time back to real audio time. A receive-path hook recursively corrects incoming start_time/end_time on every timestamped message, so all listeners see real audio time. No-op until silence is injected. - Factor the transport write into _send_audio_bytes() so injected silence updates the audio counters without passing through subclass send_audio overrides. sdk/voice: - Add feou_latency_compensation config flag (default True); enable compensation on the forced EOU and match only the forced EndOfUtterance (forced: true). Add tests/voice/test_20_feou_latency.py: a live A/B latency comparison plus offline, deterministic unit tests for the explicit-timestamp shift and round-trip.
sam-s10s
changed the title
Gate whether the resolved timestamp is written to the ForceEndOfUtterance message via include_timestamp (default True). When False, no timestamp is sent and the engine falls back to its own timing; the marker is still computed so latency compensation continues to work.
|
|
||
| await self._send_audio_bytes(payload) | ||
|
|
||
| async def _send_audio_bytes(self, payload: bytes) -> None: |
Contributor
There was a problem hiding this comment.
I am curious if you haven't seen any issues with this, as this should require a lock, as I assume (haven't seen all the implementation) that this gets called from 2 threads? or does it get serialized by the asyncio loop and its fine?
If its synchronous its should be ok
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Reference PR — not intended to merge as-is. This branch is based directly off
mainand isolates only the FEOU→EOU latency compensation so it can be reviewed in isolation. In reality this work was developed on top of several otherpr/*branches (op-model, rt-audio-format, voice-forced-eou rework, voice-8khz, ws-headers, …) and should ideally sit on top of those rather than on baremain. It's opened here primarily for reference/review of the compensation itself. Marked as draft.What this does
When a voice agent decides someone has stopped talking it sends a ForceEndOfUtterance (FEOU) and waits for the engine's EndOfUtterance (EOU). The engine only resolves the FEOU once the decode chunk covering that timestamp is complete, so the client normally waits real-time for the inter-utterance gap to fill it. Instead we inject that silence instantly up to the next chunk boundary (+ margin), so the engine resolves immediately — while leaving the FEOU timestamp at the true speech end.
Measured: ~66% reduction in FEOU→EOU latency, transcript unchanged.
Changes (5 files, FEOU-only)
sdk/rtforce_end_of_utterance(compensate_latency=…)injects one block of PCM silence up to the next decode-chunk boundary + margin, then sends the FEOU at the real speech end (never moved). Explicit timestamps are treated as real-audio time and shifted onto the server timeline.adjust_timestamp()maps server→real time. A receive-path hook recursively correctsstart_time/end_timeon every timestamped message, so all listeners see real audio time. No-op until silence is injected._send_audio_bytes()shared writer so injected silence updates the audio counters without passing through subclasssend_audiooverrides.sdk/voicefeou_latency_compensationconfig flag (defaultTrue); enables compensation on the forced EOU and matches only the forcedEndOfUtterance(forced: true).Tests —
tests/voice/test_20_feou_latency.py: a live A/B latency comparison plus offline, deterministic unit tests (explicit-timestamp shift + round-trip).Deliberately excluded (kept "purest")
The
speechmatics-rtversion pin bump, the Makefileinstall-dev/install-smartchanges, the Smart-Turn audio-buffer fix, and thetest_08/test_14cleanups — all live on the integration branch, not here.CI caveat
Because the
speechmatics-rtpin is not bumped here, CI will resolvespeechmatics-rtfrom PyPI (without these RT methods) unless RT is installed editable. Self-consistent locally; needs the RT side released/pinned (or this rebased onto the RT PR) before CI is green.Verification
Compiles; ruff + mypy pass; all 3 tests pass on the
mainbase (live −53% run + 2 offline unit tests).