VoiceBot - Voice-Enabled Chatbot

A web application that is operated by speech. The browser records a spoken question, the server transcribes it with Whisper, a fine-tuned DistilBERT model classifies the transcript into one of 41 intents, and a response is returned for that intent. The recognised speech and the reply are both displayed, along with the predicted intent and its confidence.

Live application: https://voicebot-slp.vercel.app

Pipeline

microphone  ->  recorded audio  ->  POST /api/voice
                                          |
                    faster-whisper base.en (int8, CTranslate2)
                                          |  recognised text
                    DistilBERT intent classifier (int8 ONNX, 41 classes)
                                          |  intent + confidence
                    response template  ->  transcript, intent and reply shown

Neither model needs PyTorch at inference time, so the deployed service stays small enough for a free hosting tier.

Dataset

CLINC150 (Larson et al., EMNLP 2019), a benchmark built for intent classification that includes an explicit out-of-scope class. This project uses a 40-intent subset spanning eight domains - small talk, utility, travel, auto and commute, banking, home, work, and kitchen and dining - plus the oos class, giving 41 classes in total.

Split	Utterances	Per in-scope intent	Out-of-scope
Train	4,250	100	250
Validation	900	20	100
Test	1,500	30	300

CLINC150 is a classification dataset and ships no replies, so app/responses.json maps each intent to two or three written responses. A few carry placeholders that are filled at request time, so the clock, date and coin-flip answers are live.

Models

Four classifiers were trained on identical splits so the value of the transformer could be measured rather than assumed. Only the last is deployed.

Model	Parameters	Test accuracy	Macro F1
TF-IDF + Logistic Regression	485,071	0.8800	0.9055
Bag-of-words + MLP	375,081	0.8747	0.8981
Embedding + BiLSTM	273,833	0.8860	0.9015
DistilBERT, fine-tuned	66,985,001	0.9460	0.9556
DistilBERT, int8 ONNX (deployed)	-	0.9473	0.9562

Speech recognition uses Whisper base.en with int8 quantisation, through the faster-whisper implementation on the CTranslate2 engine.

Results

Out-of-scope rejection. A prediction whose softmax probability falls below a threshold is answered as out-of-scope rather than guessed at. The threshold was chosen on the validation split and only then measured on test.

Setting	In-scope accuracy	Out-of-scope recall	Macro F1
No threshold	0.9742	0.8333	0.9556
Threshold 0.44	0.9700	0.9033	0.9651

End-to-end through speech. 86 held-out test utterances were synthesised to audio and run through the deployed pipeline.

Metric	Value
Word error rate	0.0394
Transcribed with no errors	80.2%
Intent accuracy from clean text	0.9767
Intent accuracy from speech	0.9651
Mean speech recognition latency	1089 ms
Mean intent inference latency	12 ms
Real-time factor	0.35

Recognition errors cost 1.2 accuracy points, because they tend to fall on words that do not determine the intent.

Figures and tables are regenerated by train/evaluate.py and train/eval_voice.py into results/, and reproduced in report/report.md.

Repository layout

app/              inference pipeline, FastAPI service, model weights
  pipeline.py     Whisper and the ONNX classifier, loaded once and reused
  main.py         API endpoints
  responses.json  intent to response templates
train/            data preparation, training, export and evaluation
web/              React front end
data/             generated CLINC150 splits
results/          metrics, figures, summary tables
report/           written report and viva preparation notes

Running locally

pip install -r requirements-train.txt

python train/prepare_data.py       # build the CLINC150 subset
python train/train_baselines.py    # the three reference models
python train/train_distilbert.py   # the transformer
python train/predict_split.py val  # validation predictions for the threshold
python train/evaluate.py           # figures and results tables
python train/export_onnx.py        # int8 ONNX build that gets deployed
python train/eval_voice.py         # end-to-end speech evaluation

cd app && uvicorn main:app --port 8080   # backend
cd web && npm install && npm run dev      # front end on :5173

All randomness is seeded, so the reported numbers reproduce.

API

Method	Path	Purpose
GET	/health	liveness probe
GET	/api/info	model metadata and the intent list
POST	/api/chat	{"text": "..."} to intent and reply
POST	/api/voice	audio upload to transcript, intent and reply
POST	/api/transcribe	audio upload to transcript only

Deployment

The backend runs as a Docker container on Railway and the front end is a static build on Vercel. DEPLOY.md has the exact commands, including the Google Cloud Run alternative.

References

1. Larson, S., Mahendran, A., Peper, J. J., et al. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. EMNLP 2019.
2. Sanh, V., Debut, L., Chaumond, J., Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS EMC^2 Workshop, 2019.
3. Radford, A., Kim, J. W., Xu, T., et al. Robust Speech Recognition via Large-Scale Weak Supervision. OpenAI, 2022.
4. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.