Implemented backchannel filtering for intelligent interruption handling-NIT Durgapur

[Mr-cody-77]

Overview

Implemented intelligent backchannel filtering to improve interruption handling in the voice agent pipeline.

The goal of this implementation was to prevent the agent from unnecessarily stopping when the user says passive acknowledgement words such as "yeah", "ok", "hmm", etc. while the agent is already speaking, while still allowing proper interruption when the user gives an actual interrupting command like "wait", "stop", or asks a new question.

---

Problem Statement

In the default behavior, the Voice Activity Detector (VAD) triggers interruption immediately whenever new user speech is detected.
This causes an issue where even passive listener acknowledgements such as:

• "yeah"
• "ok"
• "hmm"

incorrectly interrupt the agent mid-response.

This creates an unnatural conversation experience because these words are often used by users only to acknowledge listening and not to stop the assistant.

---

Approach / Logic Implemented

To solve this issue, I introduced a delayed transcript-based interruption evaluation mechanism.

Core Logic:

1. When VAD detects user speech while the agent is speaking:

   • Instead of interrupting immediately, the system waits for a short configurable delay.

2. This delay allows STT (Speech-to-Text) enough time to generate a transcript.

3. After transcript generation, the spoken text is passed through a 'BackchannelFilter'.

4. The filter classifies the transcript into one of three categories:

   • ignore → Passive acknowledgement / backchannel detected.
   • interrupt → Genuine interruption / stop command detected.
   • respond → Normal user input when agent is silent.

---

Implementation Details

Added New Components

• Implemented BackchannelConfig for configurable filtering behavior.
• Implemented BackchannelFilter for transcript classification.

Modified Agent Flow

Updated 'AgentActivity' to:

• Delay interruption decision until transcript is available only when Agent is already speaking. If the agent is in ideal state then that pipeline doesnot introduce delay. The response is immediate in ideal state case while in active speaking state a delay is introduced.
• Fetch transcript from audio recognition.
• Use 'BackchannelFilter' before interrupting.
• Ignore passive acknowledgements without pausing current speech.
• Interrupt immediately for real interruption commands.

---

Files Modified

• backchannel.py
• agent_activity.py
• agent_session.py
• test_backchannel_logic.py
• verify_backchannel.py
• test_backchannel_integration.py
• BACKCHANNEL_IMPLEMENTATION (Documentation)

---

Testing / Validation

Since a complete backend/live environment was not provided in the assignment, I created custom standalone test scripts to validate the implemented backchannel logic.

Log Transcripts :-

1. …\agents-assignment > venv\Scripts\python.exe test_backchannel_logic.py

   yeah | speaking=True => ignore
   ok | speaking=True => ignore
   wait | speaking=True => interrupt
   yeah wait | speaking=True => interrupt
   what is python | speaking=True => interrupt
   yeah | speaking=False => respond

2. …\agents-assignment > venv\Scripts\python.exe verify_backchannel.py

============================================================
BACKCHANNEL IMPLEMENTATION VERIFICATION

============================================================
TEST 1: Import Verification

PASS BackchannelConfig import: OK
PASS BackchannelFilter import: OK
PASS tokenize.basic.split_words import: OK

============================================================
TEST 2: Filter Logic Verification

PASS Backchannel during speech
Input: 'yeah', Speaking: True
Expected: ignore, Got: ignore
PASS Backchannel during speech
Input: 'ok', Speaking: True
Expected: ignore, Got: ignore
PASS Backchannel during speech
Input: 'hmm', Speaking: True
Expected: ignore, Got: ignore
PASS Backchannel during speech (hyphenated)
Input: 'uh-huh', Speaking: True
Expected: ignore, Got: ignore
PASS Backchannel during speech (hyphenated)
Input: 'mm-hmm', Speaking: True
Expected: ignore, Got: ignore
PASS Backchannel when silent
Input: 'yeah', Speaking: False
Expected: respond, Got: respond
PASS Command during speech
Input: 'wait', Speaking: True
Expected: interrupt, Got: interrupt
PASS Command during speech
Input: 'stop', Speaking: True
Expected: interrupt, Got: interrupt
PASS Command during speech
Input: 'no', Speaking: True
Expected: interrupt, Got: interrupt
PASS Mixed input (backchannel + command)
Input: 'yeah wait', Speaking: True
Expected: interrupt, Got: interrupt
PASS Question during speech
Input: 'what is that', Speaking: True
Expected: interrupt, Got: interrupt
PASS Empty utterance
Input: '', Speaking: True
Expected: ignore, Got: ignore

Results: 12 passed, 0 failed

============================================================
TEST 3: Required Scenarios Verification

Scenario 1: Long Explanation:
User says 'yeah/ok/uh-huh' while agent talks
PASS 'yeah' (speaking=True) -> ignore
PASS 'ok' (speaking=True) -> ignore
PASS 'uh-huh' (speaking=True) -> ignore
PASS 'mm-hmm' (speaking=True) -> ignore
-> Scenario PASSED

Scenario 2: Passive Affirmation:
User says 'yeah' when agent is silent
PASS 'yeah' (speaking=False) -> respond
PASS 'ok' (speaking=False) -> respond
-> Scenario PASSED

Scenario 3: Correction:
User says 'no stop' to interrupt
PASS 'no stop' (speaking=True) -> interrupt
PASS 'wait' (speaking=True) -> interrupt
-> Scenario PASSED

Scenario 4: Mixed Input:
User says 'yeah but wait' - should interrupt
PASS 'yeah but wait' (speaking=True) -> interrupt
PASS 'ok hold on' (speaking=True) -> interrupt
-> Scenario PASSED

============================================================
TEST 4: Configuration Verification

PASS Default config created

• enabled: True
• backchannel_words: 18 words
• interruption_commands: 20 commands
• stt_settling_delay_ms: 300ms
  PASS Custom config created
• stt_settling_delay_ms: 500ms
  PASS Custom backchannel word 'custom' -> ignore
  PASS Custom command 'custom_command' -> interrupt

============================================================
SUMMARY

PASS: Filter Logic
PASS: Scenarios
PASS: Configuration

============================================================
ALL TESTS PASSED

3. …\agents-assignment > venv\Scripts\python.exe -m pytest tests/test_backchannel_integration.py -v --noconftest 2>&1
   <truncated 32 lines>
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_when_silent_respond[yeah] PASSED [ 30%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_when_silent_respond[ok] PASSED [ 31%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_when_silent_respond[hmm] PASSED [ 32%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_when_silent_respond[sure] PASSED [ 33%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_when_silent_respond[got it] PASSED [ 34%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_response_context[yeah-Great, let's continue.] PASSED [ 36%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_response_context[ok-Okay, moving on.] PASSED [ 37%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_response_context[sure-Perfect.] PASSED [ 38%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[wait] PASSED [ 39%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[stop] PASSED [ 40%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[no] PASSED [ 42%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[hold on] PASSED [ 43%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[hang on] PASSED [ 44%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[pause] PASSED [ 45%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[slow down] PASSED [ 46%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[not yet] PASSED [ 48%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[actually] PASSED [ 49%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[sorry] PASSED [ 50%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[excuse me] PASSED [ 51%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[hey] PASSED [ 53%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_interruption_command_during_speech[hello] PASSED [ 54%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_question_as_interruption[what is that] PASSED [ 55%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_question_as_interruption[tell me more] PASSED [ 56%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_question_as_interruption[who said that] PASSED [ 57%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_question_as_interruption[where are we] PASSED [ 59%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_question_as_interruption[when did that happen] PASSED [ 60%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_question_as_interruption[why is that] PASSED [ 61%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_question_as_interruption[how does it work] PASSED [ 62%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[yeah wait] PASSED [ 63%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[ok but stop] PASSED [ 65%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[hmm no wait] PASSED [ 66%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[yeah okay but wait a second] PASSED [ 67%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[uh-huh but hold on] PASSED [ 68%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[yes but I need to ask something] PASSED [ 69%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[yeah actually] PASSED [ 71%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_mixed_backchannel_and_command_interrupt[ok sorry but] PASSED [ 72%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_command_before_backchannel_interrupt[wait yeah] PASSED [ 73%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_command_before_backchannel_interrupt[stop ok] PASSED [ 74%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_command_before_backchannel_interrupt[no hmm] PASSED [ 75%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_repeated_backchannel_ignore[uh-huh uh-huh] PASSED [ 77%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_repeated_backchannel_ignore[mm-hmm mm-hmm] PASSED [ 78%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_repeated_backchannel_ignore[yeah yeah yeah] PASSED [ 79%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_repeated_backchannel_ignore[ok ok ok] PASSED [ 80%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_repeated_backchannel_ignore[yes yes] PASSED [ 81%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_empty_utterance PASSED [ 83%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_whitespace_only PASSED [ 84%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_unknown_word_during_speech PASSED [ 85%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_backchannel_disabled PASSED [ 86%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_min_words_for_interruption PASSED [ 87%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_custom_backchannel_words PASSED [ 89%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_custom_interruption_commands PASSED [ 90%]
   tests/test_backchannel_integration.py::TestBackchannelFilter::test_stt_settling_delay_config PASSED [ 91%]
   tests/test_backchannel_integration.py::TestBackchannelConfig::test_default_config PASSED [ 92%]
   tests/test_backchannel_integration.py::TestBackchannelConfig::test_all_defaults_populated PASSED [ 93%]
   tests/test_backchannel_integration.py::TestBackchannelIntegration::test_scenario_1_long_explanation PASSED [ 95%]
   tests/test_backchannel_integration.py::TestBackchannelIntegration::test_scenario_2_passive_affirmation PASSED [ 96%]
   tests/test_backchannel_integration.py::TestBackchannelIntegration::test_scenario_3_correction PASSED [ 97%]
   tests/test_backchannel_integration.py::TestBackchannelIntegration::test_scenario_4_mixed_input PASSED [ 98%]
   tests/test_backchannel_integration.py::TestBackchannelIntegration::test_timing_simulation PASSED [100%]

============================== warnings summary ===============================
venv\Lib\site-packages_pytest\config_init_.py:1434
D:\Programs\Companies Task\SalesCode.ai\agents-assignment\venv\Lib\site-packages_pytest\config_init_.py:1434: PytestConfigWarning: Unknown config option: asyncio_default_fixture_loop_scope

self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

venv\Lib\site-packages_pytest\config_init_.py:1434
D:\Programs\Companies Task\SalesCode.ai\agents-assignment\venv\Lib\site-packages_pytest\config_init_.py:1434: PytestConfigWarning: Unknown config option: asyncio_mode

self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================= 83 passed, 2 warnings in 3.74s ========================