Skip to content

Fix early wake-word chime timing - #342

Open
nvddr wants to merge 2 commits into
mainfrom
devdeepr/fix-early-wake-chime
Open

Fix early wake-word chime timing#342
nvddr wants to merge 2 commits into
mainfrom
devdeepr/fix-early-wake-chime

Conversation

@nvddr

@nvddr nvddr commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

What

  • probe wake words on a stable cadence while speech is still in progress
  • add a short silent tail so partial STT can finalize the wake phrase
  • let an in-flight partial probe finish before final transcription
  • suppress the late fallback chime for spoken queries
  • retain fallback acknowledgment for synthetic text and phrase-only turns

Why

Partial probes drifted with STT latency and were cancelled when VAD finalized the utterance. The missed early match then fell back to a chime at final-transcript time, immediately before response speech.

Impact

The acknowledgment chime now follows recognition of Agent or hey agent during the utterance. If early recognition misses a spoken query, the runtime no longer plays a misleading late chime at response start.

Checks

  • tests/test_voice_pipeline.py — 86 passed
  • tests/test_voice_session.py — 5 passed
  • focused wake timing regressions — 6 passed
  • git diff --check

Comment thread tests/test_voice_pipeline.py Fixed
Comment thread agent-sdk/xr-ai-voice/xr_ai_voice/_processors/vad_stt.py Fixed
@nvddr
nvddr requested a review from wenxind-nvidia August 6, 2026 03:53
@wenxind-nvidia

Copy link
Copy Markdown
Collaborator

Blockers

  1. _finish_probe_for_utterance() waits indefinitely for an active partial STT request. The standard STT client timeout is 30 seconds, and VAD awaits on_utterance inline, so a slow probe can block final transcription and subsequent microphone processing for that entire period. Allow only a short grace period before cancelling the probe, with a regression test using a probe that never completes.

  2. GitHub reports mergeable: false and mergeable_state: dirty. The branch conflicts with current main in docs/changelog.md and needs rebasing.

Suggestion

The phrase-only fallback test sends only a synthetic TranscriptionFrame. Add UserStartedSpeakingFrame first so it verifies that spoken phrase-only turns retain their final chime through the new _speech_utterance path.

Nit

VadConfig.stop_probe_after_s still describes one “extra STT pass,” but it now controls a cadence of up to three probes. Update the docstring to reflect that contract.

Validation: 91 focused tests passed, Ruff passed, and git diff --check passed. The PR is still a draft, so only DCO and CodeQL checks have run.

nvddr added 2 commits August 5, 2026 21:45
Signed-off-by: Devdeep Ray <devdeepr@Nvidia.com>
Signed-off-by: Devdeep Ray <devdeepr@Nvidia.com>
@nvddr
nvddr force-pushed the devdeepr/fix-early-wake-chime branch from 8721a9b to b3463c1 Compare August 6, 2026 04:47
@nvddr
nvddr marked this pull request as ready for review August 6, 2026 04:48
@wenxind-nvidia

Copy link
Copy Markdown
Collaborator

Suggestion

VoiceGateProcessor._speech_utterance is cleared only when a TranscriptionFrame arrives or the participant leaves. If a spoken turn ends with empty or failed STT, the marker remains set; a later synthetic wake-prefixed transcript for the same participant is then treated as speech and its fallback listening chime is incorrectly suppressed.

Please add regression coverage for UserStartedSpeakingFrame -> UserStoppedSpeakingFrame with no transcript -> synthetic wake-prefixed TranscriptionFrame, and ensure the synthetic query still emits its chime. Simply clearing the marker on UserStoppedSpeakingFrame is not sufficient because that frame normally arrives before the spoken transcript and would re-enable the late chime this PR is intended to remove.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants