PULSE — pose-based ultrasound speech evaluation (hands-on edition)

A guided, notebook-by-notebook reproduction of the PULSE research project: using ultrasound video of the tongue (plus audio) to help score children's speech-sound productions. You start from raw ultrasound and end at the paper's headline result — selective fusion of tongue movement and audio beats audio alone at telling a correct production from a disordered one.

This repo is a summer-project lab manual that mirrors how a real ML project actually goes:

• Notebooks 0–2 — get the data. Use the public UltraSuite dataset and the toolkit that ships with it (ustools, vendored in this repo) to turn raw ultrasound into tongue "poses". You drive existing tools here — no reinventing file readers or signal processing.
• Notebooks 3–8 — do the science. Here you write the code: the feature engineering (turning tongue movement into numbers) and the machine learning. Each of these notebooks has exactly one function for you to fill in, with a docstring spec, a self-checking assert, and an immediate plot so you know whether you got it right.

You do not need a GPU. The tongue-tracking model runs on your CPU. You also don't need to process all of UltraSuite (it's ~380 GB) — you fetch a few recordings, and the precomputed features for the full study are committed in data/.

The 9 notebooks

#	Notebook	What you'll do	You code?	Colab
00	setup_and_fetch	Fetch a few recordings; read them with the ustools toolkit	—
01	ultrasound_to_image	Use the toolkit to fan the raw scanlines out into a tongue picture	—
02	find_the_tongue	Run the provided pose model on your CPU → 16 tongue keypoints	—
03	cut_out_phonemes	Slice out /r/, /k/, /g/ using forced-aligned labels; 5 time-bins	✏️
04	articulatory_features	Turn tongue motion into numbers: speed, jerk, shape, curvature	✏️
05	audio_features	The acoustic side: MFCCs, spectral features, formants	✏️
06	selective_fusion	The main result: LOSO scoring, AUC 0.687 → 0.725	✏️
07	seeing_the_difference	LDI & workspace maps: TD ≈ 78° vs SSD ≈ 11° (+ a stats-rigor check)	✏️
08	why_did_it_decide	SHAP + the 68 "rescued" covert-contrast productions	✏️

✏️ = there's one function for you to fill in (look for raise NotImplementedError); a self-checking assert right after tells you when it's correct.

Notebooks are authored as plain .py files with # %% cell markers (open them in VS Code or Jupyter and "Run Cell"); a paired .ipynb copy is committed alongside each one for the Colab badges above. Fully-worked solutions live on a separate solutions branch.

Run in Colab (no local install)

Click any Open in Colab badge in the table above. The first cell of every notebook is a bootstrap that runs only on Colab: it clones this repo, downloads the pose model, installs rsync + the Python deps (pip install -e .), and drops you into the repo — then the notebook runs top-to-bottom. Nothing to install by hand; it's a no-op off Colab.

A few things to know:

• These badges open the student notebooks, the ones with functions for you to fill in. The worked solutions are on the solutions branch, whose README has its own badges pointing there — no URL editing needed.
• Each notebook stands alone. Every badge opens a fresh Colab machine with nothing from the notebook you ran before, so notebooks 01–05 fetch or rebuild whatever they need on the way in. Open them in any order. On your own laptop, where you do work through them in order, that all no-ops.
• Notebooks 00–02 and 05 need the network for the couple of raw recordings they pull from the public UltraSuite rsync mirror. Notebooks 03–04 and 06–08 run entirely from committed data, so they work offline.

Setup (local, with uv)

Prerequisites (install once):

• uv — manages the Python environment
• git-lfs — the 94 MB pose model is stored with Git LFS (sudo apt install git-lfs, or brew install git-lfs)
• rsync — notebook 00 fetches a couple of raw UltraSuite recordings from the public Edinburgh mirror over rsync (sudo apt install rsync, or brew install rsync; preinstalled on macOS/most Linux). On Windows, which doesn't ship rsync, notebook 00 downloads a small portable copy for you the first time (cached under .tools/, checksum-verified) — no install needed. Only notebooks 00–02 and 05 need it — the rest run from committed data offline.

git lfs install            # enable Git LFS
git lfs pull               # download the pose model
uv sync                    # install the Python deps (CPU-only)

Then open notebooks/00_setup_and_fetch.py and start running cells.

A note on coordinates (no millimetres)

Everything here is in pixel coordinates of the ultrasound image. The original research also explored a millimetre (fan-rectified) version; we deliberately skip it — it adds fiddly per-probe geometry and changes none of the ideas. The paper's headline numbers are the pixel-space numbers.

What's in data/

Folder	Contents
features/	The paper's precomputed feature tables for all 1,151 scored phonemes (notebooks 06–08)
results/	Reference result CSVs to check your numbers against
ldi_cache/	Pooled keypoint clouds for the LDI figure (notebook 07)
scores/	Clinician pronunciation scores (1–5)
labels/	Forced-aligned phoneme labels for the sample recordings
weights/	tongue_resnet50.onnx — the CPU pose model (Git LFS)
poses/	The sample recording's tongue poses (247 KB), so notebooks 03–04 have an input even if you haven't run 02 yet. Poses you compute for other recordings land here too, untracked.
raw/, frames/	What you generate: the recordings you fetch (00) and the fanned-out frames (01). Git-ignored.

We precompute only the features for the full 9-child study — running the model + feature extraction on all 1,151 phonemes would take hours and hundreds of GB. Everything for your recording (frames, poses, features) you make yourself in notebooks 01–04. | raw/, frames/ | What you fetch / generate yourself (git-ignored) |

UltraSuite raw data is fetched by you in notebook 00 from the public mirror; we never redistribute it.