scrybe_benchmarking

I'm sure there are faster ways to do this, but this allowed me to test locally on my PC and use Flutter in the process. Happy for any PRs or suggestions anyone has.

Setup

• Ensure you have an assets/ directory in the main project directory
• inside assets/ include curated/, derived/, models/, and raw/
• in models/, just put a directory for each model you want to test .
  ├── sherpa-onnx-moonshine-base-en-int8
  │ ├── cached_decode.int8.onnx
  │ ├── encode.int8.onnx
  │ ├── preprocess.onnx
  │ ├── tokens.txt
  │ └── uncached_decode.int8.onnx
  ├── sherpa-onnx-nemo-ctc-en-conformer-large
  │ ├── model.int8.onnx
  │ ├── model.onnx
  │ └── tokens.txt
  
• Inside raw, put Directories grouping wav/srt files together .
  ├── Vital_Signs
  │   ├── 1.1.srt
  │   ├── 1.1.wav
  │   ├── 1.2a.srt
  │   ├── 1.2a.wav
  ├── Mechanical_Ventilation
  │   ├── Vent_v1.srt
  │   ├── Vent_v1.wav
  │   ├── Vent_v10.srt
  │   ├── Vent_v10.wav
  

Conversion

• You must have the folders setup as described above or else this won't work
• When you click the Convert Raw Files button, it will go through each directory in raw/ and create a mirror directory in curated/
• Currently only works on wav and srt files, but shouldn't be that hard to add mp3 support, and possible json basd on this schema
• It will the go through each matching wav/srt file
• It will split them into 20-30 second chunks based on a rough mechanism that can be seen here
• It will save them in a directory with the same name as the file that its chunking .
  ├── Vital_Signs
  │   ├── 1.1
  │   │   ├── 001.srt
  │   │   ├── 001.wav
  │   │   ├── 002.srt
  │   │   ├── 002.wav
  │   ├── 1.2a
  │   │   ├── 001.srt
  │   │   ├── 001.wav
  ├── Mechanical_Ventilation
  │   ├── Vent_v1
  │   │   ├── 001.srt
  │   │   ├── 001.wav
  │   ├── Vent_v10.srt
  │   │   ├── 001.srt
  │   │   ├── 001.wav
  │   │   ├── 002.srt
  │   │   ├── 002.wav
  

Benchmarking

• After you've made sure you have all of the models you need in the ```assets/models/`` directory, add them to models.dart
• I've included a number of different kinds in there to show how they are setup (please let me know if any are incorrect)
• Then just push Run Benchmark
• It will display your progress as it runs through models and files
• It eventually produces resultsl in assets/derived/reports/report.txt that look like the ones below
• As I said, it's not the fastest, but for a quick and dirty basic comparison, its been working ok for me

Run 1

Model	Files	Avg WER	Min WER	Max WER	Word Accuracy	Avg Decode (s)	RTF
sherpa-onnx-moonshine-base-en-int8	365	19.89%	0.00%	100.00%	80.11%	2.33	0.08
sherpa-onnx-streaming-zipformer-en-2023-06-26-mobile	365	36.80%	0.00%	100.00%	63.20%	2.56	0.09
sherpa-onnx-streaming-zipformer-en-2023-06-26-mobile.int8	365	36.81%	0.00%	100.00%	63.19%	1.69	0.06
sherpa-onnx-whisper-small.en	365	20.01%	3.13%	100.00%	79.99%	15.09	0.50
sherpa-onnx-whisper-small.en.int8	365	19.71%	3.13%	100.00%	80.29%	12.67	0.42
sherpa-onnx-whisper-tiny.en	365	23.17%	4.82%	300.00%	76.83%	3.06	0.10
sherpa-onnx-whisper-tiny.en.int8	365	22.98%	4.82%	300.00%	77.02%	2.73	0.09
sherpa-onnx-zipformer-small-en-2023-06-26	365	34.88%	14.68%	100.00%	65.12%	0.68	0.02
sherpa-onnx-zipformer-small-en-2023-06-26.int8	365	34.78%	14.29%	100.00%	65.22%	0.54	0.02

Run 2

• So I'm running on a AMD® Ryzen 7 pro, 64 GB RAM, Budgie Linux
• curated assets ~300MB of .wav files
• Took close to 6 hours (maybe I'll implement parallel processing)

Model	Files	Avg WER	Min WER	Max WER	Word Accuracy	Avg Decode (s)	RTF
sherpa-onnx-moonshine-base-en-int8	380	9.39%	0.00%	100.00%	90.61%	2.24	0.08
sherpa-onnx-nemo-fast-conformer-transducer-en-24500	380	14.28%	0.00%	100.00%	85.72%	0.97	0.04
sherpa-onnx-streaming-zipformer-en-2023-06-26-mobile	380	28.77%	9.09%	100.00%	71.23%	2.38	0.09
sherpa-onnx-streaming-zipformer-en-2023-06-26-mobile.int8	380	28.79%	9.09%	100.00%	71.21%	1.53	0.06
sherpa-onnx-whisper-small.en	380	9.27%	0.00%	100.00%	90.73%	14.45	0.53
sherpa-onnx-whisper-small.en.int8	380	9.32%	0.00%	100.00%	90.68%	12.70	0.46
sherpa-onnx-whisper-tiny.en	380	12.25%	0.00%	300.00%	87.75%	2.89	0.11
sherpa-onnx-whisper-tiny.en.int8	380	12.55%	0.00%	300.00%	87.45%	2.71	0.10
sherpa-onnx-zipformer-small-en-2023-06-26	380	26.84%	7.69%	100.00%	73.16%	0.68	0.02
sherpa-onnx-zipformer-small-en-2023-06-26.int8	380	26.94%	7.69%	100.00%	73.06%	0.55	0.02
sherpa-onnx-nemo-ctc-en-conformer-large	380	22.35%	4.23%	100.00%	77.65%	2.07	0.08
sherpa-onnx-nemo-ctc-en-conformer-small	380	24.87%	6.19%	100.00%	75.13%	0.61	0.02
sherpa-onnx-streaming-zipformer-ctc-small-2024-03-18	380	34.87%	15.38%	100.00%	65.13%	1.39	0.05
sherpa-onnx-streaming-zipformer-en-2023-06-26	380	28.72%	9.09%	100.00%	71.28%	2.61	0.10
sherpa-onnx-streaming-zipformer-en-2023-06-26.int8	380	28.72%	9.09%	100.00%	71.28%	1.78	0.07
sherpa-onnx-zipformer-large-en-2023-06-26	380	25.60%	7.04%	100.00%	74.40%	1.28	0.05
sherpa-onnx-zipformer-large-en-2023-06-26.int8	380	25.65%	7.04%	100.00%	74.35%	0.81	0.03

Run 3

Model	Files	Avg WER	Min WER	Max WER	Word Accuracy	Avg Decode (s)	RTF
sherpa-onnx-moonshine-base-en-int8	1348	12.57%	0.00%	175.00%	87.43%	0.38	0.05