"Beep beep beep." I hear the sound of my wife's insulin pump going off in the middle of the night, and wake her up so she can eat a snack to increase her blood sugar levels. She has type 1 diabetes, and low blood sugars are fatal if left untreated. We are very fortunate that my wife has access to a dexcom sensor machine which can alert her insulin pump to sound an alarm depending on her blood sugar levels (high, low or urgent low), but when she is sleeping or there is a lot of noise it can be hard to hear these alerts. Dexcom has a program which can send phone notifications to 10 contacts with blood sugar alerts (especially helpful for young children learning to manage their diabetes), but from a public health (and personal) perspective I was curious if nearby phones could detect insulin pump alert sounds. A phone app which could detect insulin pump alarms could make automatic calls to emergency services for ugent low blood sugars, even if the person with the application had no idea why a diabetic person was lying unconscious (or even that they were diabetic). For me, as a husband of a type 1 diabetic, I would use this application to have peace-of-mind that my phone could be an extra member on my wife's team, helping her manage an exhausting chronic ilness.
In this project I used deep learning with TensorFlow to predict blood sugar alert sounds from five second audio clips. My results suggest that these alerts can reliably be detected, with an accuracy above human performance, and that a public health phone application could aid in the management of type 1 diabetes.
I formed my own dataset for this project, by overlaying self-recorded clips of insulin pump alarm sounds (high, low and urgent low, included in the 'data/alerts' directory) over the five-second audio clips from the ESC-50 dataset. All the alarm sounds were recorded by my iPhone, at a distance of about two feet to my wife's insulin pump while she played the various alert sounds. I recorded two sets of alarms - one where the the pump was under a blanket (like may be the case at night or during the winter under a heavy jacket, these alerts were called "unclear") and one where the pump was not covered (these alerts were called "clear"). In total there were 2000 audio clips - 1600 made up the training set, 200 made up the development ("dev") set and 200 made up the test set. In each of the three data partitions the examples were split between four classes - negative (no alert), high, low and urgent low. For each of the three alert types half of the examples were overlayed with the clear alert and half with the unclear alert, and no partial alerts were overlayed (i.e. each five-second audio clips either contained no alert or the full duration of an alert). For example, the training set had 400 negative examples, 200 clear high examples, 200 unclear high examples, etc. Each data example was converted into a spectrogram, i.e. 3D tensor, using scipy, and I subsetted the frequency dimension based on the frequencies that existed in the alert sounds both for improved modeling performance and trustworthyness.
Modeling was carried out with TensorFlow version 2.15.0. I tried over 100 variations of models including RNN's, CNN's and mixtures of the two, always with an output softmax layer with 4 output units. The final model was a mixture model, where the RNN and CNN models were pre-trained on the training set and a few dense layers connecting their output layers were then trained on the training set (the mixed model where all weights were trained simultaneously did not perform as well). The performance of this model was about 80% on the training set, dev set and test set. The RNN was a high-dimensional LSTM model (512 hidden units). The CNN model followed a typical architecture, with batch normalization to speed up learning, but with a custom initializer for the convolutional layer. The three convolutions were initialized to three actual training examples of the three alert types, greatly speeding up learning because the alarm signatures were consistent across examples.
The final model had 2.27 million parameters (almost all coming from the RNN, which was necessary for the final model performance), comparable to MobileNet models, meaning that the model could be feasible for lightweight mobile applications.
I compared the accuracy of the models, on the dev set, against human-level performance on the dev set and used this criteria to pick the final model. In order to assess human-level performance I produced the audio clips generated in the dev set, randomly permuted them and their alert labels, and stitched the audio clips together with audio of me making a sound (of length 1 second). This resulted in a 20 minute long audio clip, for which I recorded my predicted labels for each of the 200 sub-clips. I expected to get close to 100% accuracy, but when I compared my predicted labels against the real labels I achieved an accuracy of 40%. There are several reasons for this disparity. Firstly, some of the audio clips had loud constant noise completely masking any audibly-detectable alert. Secondly, the alerts can sound similar for some of their individual tones, meaning that during clips which had inconsistent loud noise I was often confused as to which alert a particular tone I heard came from. Lastly, there was probably some human error due to lapses in focus or perhaps even accidentally recording the wrong alarm. However, these types of errors would also be expected in regular life - it is hard to hear muffled alarms in a loud restaurant, sometimes I can confuse alarms when I only clearly hear a portion of them, and since I'm not always listening for the alarms during everyday life sometimes the best I can offer my wife is "is your pump ringing?"
I'm just a guy with an old laptop who loves his type 1 diabetic wife, but with minimal computational resources I was able to build an alarm detection model with decent, above human-performance, accuracy to aid in diabetes management. Whether models like the one I build will be used in practice for public health initiatives who knows, but I hope that deep learning can help prevent tragedies and improve the quality of life for those struggling with chronic ilnesses.
Would I personally use an application built with this model? Absolutely, because it would work while we slept and in other situations where we might miss hearing an alarm. Would I trust this model as the sole line of defense for diabetes management? No way. Even if the model had 100% test set accuracy, the dataset I used was too limited to fully reflect the variety of situations that occur in our everyday life. Also, the lack of interpretability of the RNN makes that portion of the final model not fully trustworthy. Also from a public health perspective the fact that negatives are often confused as urgent lows means that huge amounts of money may be spent calling emergency services when none are needed. I think deep learning has a ways to go before it can be trusted to fully manage disease management without any human intervention.
On a personal level I learned a huge amount from this project. I learned how modelling, while challenging, is the easiest part of the process - I made many more mistakes trying to build a correctly balanced dataset, preprocessing the data and devising evaluation metrics than I did with modelling. I learned that when I balanced the datasets by alarm type that the number of negative examples overwhelmed the number of unclear alerts of each type and therefore unclear alerts were almost always predicted as negative examples. What worked was having the same number of negative examples as clear highs as unclear highs etc. However, in the final model many negative examples were incorrectly predicted as containing an alarm, which would be a nuissance for a general model user (however as the wife of a diabetic I would not mind - better safe than sorry!). I learned that I shouldn't assume I know what human-level prediction is - I spent a lot of time trying to get 100% model performance because I assumed that would be the only indicator of success, however I achieved significantly lower performance that almost all of the models I trained. I learned that batch normalization is a lifesaver for speeding up learning, that dropout was my most reliable form of regularization, and that smart initialization of parameters (like the convolutions I talked about earlier) sometimes made a 20% difference in performance.
I believe that there are a few significant improvements that could be made to this project. At the data-level there should be both a wider range of alert volumes (rather than just clear and unclear) in the clips and there should be partial alerts overlayed in the clips. At the level of modelling, work should be done to improve accuracy closer to 100%. In terms of model evaluation I could have predicted the dev set labels multiple times and averaged the accuracies for a more robust benchmark. I also could have looked more into model interpretation methods - it was very difficult to do so with the RNN model (you can see my attempts in the 'analysis/interpretation.py' script), but looking at saliency maps of the CNN models helped me decide to make a custom convolution initializer (since randomly-initialized convolutions tended to learn from noise rather than signal).
To run this code you must download the ESC-50 dataset from ESC and place the downloaded folder into the data directory. Running the 'analysis/final_analysis.py' file will generate the final deep learning model and metrics.
If you are having difficulty reading in the audio files with pydub like I was, try following the steps at this link (they helped me a lot!) StackOverflow.
The code in the file 'analysis/modelling.py' can give you an idea of which models I fit and the various iterations of how I built the dataset, but won't run top-to-bottom. The files 'data/data_exploration' and 'analysis/interpretation.py' both show how I interpreted data and model for improving performance.