feat(model): add Sopro V2 Turbo (sopro_tts) with offline and streaming

[Brioch]

Adds sopro_tts as a community model: samuel-vitorino/sopro-v2-turbo, a 120M Apache-2.0 zero-shot voice-cloning TTS covering English, European Portuguese, French and German at 24 kHz mono. Offline and segment-level streaming.

It ships as its own family; --family sopro, sopro_v2 and sopro_v2_turbo all resolve to it.

Build

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
    -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=sopro_tts
cmake --build build --target audiocpp_cli -j"$(nproc)"

Warm-bench probe (optional, used for the parity work below):

cmake -S . -B build -DENGINE_BUILD_WARMBENCH=ON
cmake --build build --target sopro_probe -j"$(nproc)"

Model

python3 tools/model_manager_v2.py install sopro_v2_turbo_safetensors
# -> models/sopro-v2-turbo/{config.json,tokenizer.model,*.safetensors}

Package ids: sopro_v2_turbo_safetensors (safetensors, real download) and sopro_v2_turbo_f16
(default GGUF). No public audio.cpp GGUF build of this family exists yet, so the GGUF package is
download.kind = "unsupported" and is built locally:

build/bin/audiocpp_gguf \
    --input model=models/sopro-v2-turbo/model.safetensors \
    --input semantic_encoder=models/sopro-v2-turbo/semantic_encoder.safetensors \
    --input speaker_encoder=models/sopro-v2-turbo/speaker_encoder.safetensors \
    --input vocoder=models/sopro-v2-turbo/vocoder.safetensors \
    --family sopro_tts --root models/sopro-v2-turbo \
    --output models/sopro-v2-turbo-GGUF/sopro-v2-turbo-f16.gguf --type f16

Run

Offline:

build/bin/audiocpp_cli \
    --task tts --family sopro_tts \
    --model models/sopro-v2-turbo \
    --backend cpu --threads 8 \
    --text "Sopro is a lightweight text-to-speech model that runs on device." \
    --voice-ref assets/resources/b.wav \
    --language en --seed 1234 --num-inference-steps 8 \
    --out out.wav --metrics

Streaming (one pull event per text segment):

build/bin/audiocpp_cli \
    --task tts --family sopro_tts \
    --model models/sopro-v2-turbo \
    --backend cpu --threads 8 --mode streaming \
    --text "$(cat article.txt)" \
    --voice-ref assets/resources/b.wav \
    --language en --seed 1234 --num-inference-steps 8 --text-chunk-size 120 \
    --out stream.wav --out-dir segments/

Server (mode: "streaming" in the model config), raw PCM is s16le mono at 24 kHz:

curl -N http://127.0.0.1:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"sopro-stream","input":"Stream this sentence as audio.",
       "voice_ref":{"type":"path","path":"assets/resources/b.wav"},
       "response_format":"pcm","stream_format":"audio",
       "options":{"text_chunk_size":120}}' \
  | ffplay -f s16le -ar 24000 -ch_layout mono -nodisp -

Path tests

Two shared long-form cases added to tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json:

• sopro_tts_voice_clone_longform (offline)
• sopro_tts_voice_clone_longform_streaming (streaming, outputs: ["audio", "named_audio"])

python3 tools/audiocpp_cli/run_audiocpp_cli_path_tests.py \
    --cases tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json \
    --family sopro_tts --backend cpu --threads 8
python3 tools/check_loader_catalog_sync.py

sopro_tts_voice_clone_longform (offline) passes: 6026 chars in, 371.6 s of audio out.

sopro_tts_voice_clone_longform_streaming currently fails in the harness, not in the model —
see the runner note below. With that one line patched locally it passes, producing 47
segment_*.wav plus a stream.wav of 371.64 s, exactly matching the offline duration.

check_loader_catalog_sync.py: active_loaders=66 specs=64 packages=187 manager_packages=187, in sync.

Pre-existing: every streaming path-test case fails on --chunk-size

Unrelated to this PR, but it blocks the streaming case above, so flagging rather than fixing here.

run_audiocpp_cli_path_tests.py appends --chunk-size to every streaming case (build_command,
around line 404), but the CLI no longer has that option — it was dropped when streaming moved to
the model's own streaming policy, and --help now says only "--mode streaming uses the selected
model's default streaming policy". require_known_args rejects it:

audiocpp_cli failed: unknown option: --chunk-size

That affects all 14 streaming cases in the catalog, not just sopro's: neutts, omnivoice,
voxcpm1, voxcpm2, dots_tts, qwen3_asr, parakeet_tdt, nemotron_asr, voxtral_realtime,
silero_vad, muscriptor, and both personaplex cases. Deleting the line is enough (a case's
chunk_size key then becomes documentation only); I verified that against sopro but did not run
the other 13, since most of those models are not installed here. Happy to send it as its own PR.

Validation

Backends: CPU (16-core x86-64 Release build, 8 threads) and CUDA (RTX 3090, sm_86,
CUDA build via scripts/build_linux.sh --backend cuda). Every stage was reimplemented
independently in numpy, driven from the checkpoint's own weights, and diffed against the C++.

Stage	Check	Result
Tensor inventory	762 names + shapes vs. the four real files	exact match
Vocoder mel front end	vs. numpy STFT + checkpoint filterbank	max diff 1.9e-3
Vocos backbone + ISTFT head	vs. numpy, all 14 blocks	max diff 1.0e-5
Semantic encoder mel	vs. numpy	max diff 1.9e-5
Semantic encoder transformer	vs. numpy, all 6 layers	max diff 6.0e-5
FSQ token ids	vs. numpy	188/188 identical
Speaker encoder mel / trunk / heads	vs. numpy	max diff 3.7e-5
Acoustic mu, spk, time embedding	vs. numpy	max diff 1.3e-5
Acoustic velocity field, every Euler step	vs. numpy	max diff 5.8e-3
Acoustic self-reconstruction	NMSE vs. the reference's own mel	0.38
Fixed seed reproducibility	byte-identical WAV across runs	pass
Streaming vs. offline, same seed	22.5 s clip, 4 segments	identical sample count; constant 1.15x gain, −47 dB residual
Streaming segment sum	segments vs. --out	exact
matmul_weight_type f16 / bf16 / q8_0	runs clean	pass
Single-file GGUF	end to end	pass
CUDA vs. CPU, same seed, 1.2 s clip	sample count / correlation	identical count (28960), correlation 0.999338, diff −28.8 dB
CUDA long-form	6026 chars end to end	pass, 375.3 s of audio
CUDA streaming vs. CUDA offline	6026 chars, 47 segments	identical duration (375.26 s); segments sum to --out exactly

Timing, RTF, RSS

Run	Backend	Audio	Wall	RTF	Peak RSS	Peak VRAM
Offline, 22.5 s clip	CPU	22.5 s	9.9 s	0.440	—	—
Streaming, same clip, text_chunk_size=120	CPU	22.5 s	10.5 s	0.466	—	—
Offline long-form, 6026 chars	CPU	371.6 s	167.7 s	0.451	1.083 GB	—
Streaming long-form, same text, 48 segments	CPU	371.6 s	—	—	1.086 GB	—
Offline long-form, 6026 chars	CUDA	375.3 s	14.2 s	0.038	887 MiB	1194 MiB
Streaming long-form, same text, 47 segments	CUDA	375.3 s	14.0 s	0.037	—	—

CUDA figures are from the path-test harness with --measure-resources (100 ms sampling, 144
samples), so the RSS and VRAM peaks are observed rather than estimated. CUDA is ~12x the
8-thread CPU run on the same text.

Time-to-first-audio, streaming at text_chunk_size=120: ~3.1 s on CPU (116 ms reference
encode + 2993 ms for a 6.66 s first segment), against ~9.9 s offline for the same text. On CUDA
at text_chunk_size=200 the first segment lands in 393 ms.

Per-segment streaming RTF over the 47 long-form segments on CUDA: min 0.035, median 0.037, max
0.064 — every segment generates faster than it plays, with ~16x headroom in the worst case. On
CPU the worst case is above 1.0 for very short segments (see Known Limitations), so the floor on
useful text_chunk_size is a CPU concern only.

The CUDA streaming run reproduces the CUDA offline duration exactly (375.26 s both ways, with the
47 segments summing to the merged output to the sample). That is the check that matters for the
solver: the velocity graph is replayed once per Euler step and re-uploads its constants each
time, so a backend that mishandled that would show up as drifting segment durations.

Long-lived session, repeated requests

Six requests through one session via --request-sequence, RSS sampled every 2 s:

RSS MB: 918 → 939 → 941 → 941 → 941 → 941 → 941

Flat after the first request; six sopro_tts.reference.prepare_ms entries and six wavs from one
session. No per-request growth. No mem_saver path was needed.

Architecture notes

Five stages, mirroring sopro/ upstream: SentencePiece text (8192 pieces, no phonemiser) →
speaker encoder (~11M, gated depthwise ResNet + attentive-statistics pooling) → semantic encoder
(~82M, Whisper-style front end, FSQ head with levels [7,5,5,5,5] = 4375 codes) → style-prefix
semantic LM (12 blocks, dim 512) → 8-block AdaLN-Zero DiT solving a rectified flow, then a
14-layer Vocos ConvNeXt backbone and one centred ISTFT.

Three implementation details worth review attention:

• LayerScale is folded at load time. The only structural difference from a Qwen3 decoder is a
  LayerScale vector on each residual branch, and those branches end in a bias-free projection, so
  the scale folds exactly into that projection's rows. The shared QwenCausalDecodeRuntime then
  runs the stack unmodified.
• Front-end buffers come from the checkpoint. torchaudio stores its analysis window and mel
  filterbank as persistent buffers; all three front ends load those rather than rebuilding the
  filterbank, which removes the usual mel-parity risk. A checkpoint exported without them fails at
  load with a message naming the missing tensor.
• The velocity graph re-uploads every leaf per Euler step. ggml_gallocr exempts only
  GGML_TENSOR_FLAG_OUTPUT tensors from being freed and reused (ggml_gallocr_free_node in
  ggml-alloc.c); an input leaf's arena space is handed to a later intermediate once its last
  consumer has run. Correct for a one-shot graph, but the solver replays the velocity graph once
  per step, so staging the constants once left steps ≥1 reading whatever overwrote them. This was
  the bug behind the first round of garbage output — worth knowing about for any other iterative
  solver in the tree.

No framework modules were modified.

Known limitations

• Streaming is segment-level, not frame-level. The upstream frame-level path (chunked DiT
  attention plus the causal vocoder, vocoder_streaming.safetensors) is not implemented, and that
  vocoder is not part of the published checkpoint this family loads. Granularity is one text
  segment; text_chunk_size is the latency dial. Every segment re-solves the whole reference mel
  prompt alongside its own frames, so per-segment cost has a floor of roughly ref_seconds worth
  of DiT work — segments shorter than ~1 s take longer to generate than to play, though the stream
  as a whole stays ahead of real time (text_chunk_size=40 measured 0.75 RTF overall).
• Streaming levels causally. Offline level-matches over the finished utterance, which a stream
  cannot see, so the first segment fixes the gain for the rest. Same seed reproduces the offline
  waveform sample-for-sample otherwise.
• CPU and CUDA only. No Vulkan, Metal or HIP coverage in this PR.
• CPU and CUDA outputs diverge on long text. On a 1.2 s clip the two backends agree exactly
  in sample count with correlation 0.999338 (−28.8 dB residual) — float reassociation through
  the DiT and vocoder, compounded across the Euler steps, with the semantic LM drawing the same
  tokens. Over 6026 chars the sampler does eventually drift: CUDA produced 375.3 s against CPU's
  371.6 s, i.e. a different number of generated tokens once a near-boundary logit flips. Expected
  for an autoregressive model across backends, but it means seed reproduces within a backend,
  not across them.
• No direct PyTorch parity. The numpy references are independent reimplementations from the
  same source, which catches implementation bugs but not a shared misreading of the architecture.
• Sampling RNG is not torch-bit-exact. sample_next_token reproduces the reference's masking,
  temperature, top-k and top-p arithmetic exactly, but draws from a seeded std::mt19937_64, so a
  given seed will not reproduce the Python output sample-for-sample. Reproducible within audio.cpp.
• Does not use the framework text chunker. split_text in text_tokenizer.cpp is a port of
  upstream's sopro.text.split_text (sentence → clause → word packing with its own punctuation
  handling) rather than engine::text::split_text_chunks, because swapping it would change output.
  The cost is that this family will not inherit future chunker work and text_chunk_mode is not
  available here. Happy to migrate it if you would rather have the shared path.
• No int8 AR path. The upstream --int8 CPU option has no equivalent; use
  sopro_tts.matmul_weight_type=q8_0.
• Text front end is deliberately minimal upstream: prefer words to symbols (one plus two, not
  1 + 2), avoid mixing languages inside one sentence.

Debugging aids kept in-tree

tests/sopro_tts/sopro_probe.cpp (built with -DENGINE_BUILD_WARMBENCH=ON) exercises each stage
in isolation against a reference clip and reports the crop_on_pause decision, a phase-invariant
mel round trip through the vocoder, the FSQ token histogram, speaker embedding statistics, and an
acoustic self-reconstruction NMSE. SOPRO_DUMP_DIR=<dir> additionally dumps encoder and solver
intermediates as raw f32. Both are what localised the allocator bug above.

WebUI

webui/configs/models_catalog.json declares sopro's ten request options. Without that, the UI
falls back to "every TTS model accepts max_tokens" (+page.svelte:704) and sopro's spec-backed
validator rejects it. Note this only shows up after rebuilding the UI bundle, since catalog.ts
inlines the JSON at build time. The same fallback will hit the next spec-strict family — possibly
worth addressing separately.

[0xShug0]

@Brioch I reached out to the model’s author, and he may review and test the PR.

[samuel-vitorino]

Overall, the integration looks good and the offline pipeline seems okay. There are just two recent upstream changes that should be ported before merging:

Band-limit vocoder synthesis - prevents an identified high-frequency vocoder hiss.
Boost-only, peak-guarded reference normalization - avoids attenuating hot references and keeps normalization consistent with the current package.

[0xShug0]

@Brioch Please resolve the conficts then we are good to go.

[samuel-vitorino]

We have been making a few more changes. I would hold off for a few days until we stabilize the final release. Will ping here when that's done.

[0xShug0]

@Brioch Thank you! PR merged!

@samuel-vitorino Sorry, I want to do some cleanup before Labor Day...Feel free to submit follow-up PRs to port upstream changes.