feat: add native MiraTTS community model

[mirek190]

Summary

Adds MiraTTS as an experimental native community TTS/voice-cloning model.

The port runs the full pipeline in C++:

• Qwen2 autoregressive speech-token generator
• ECAPA-TDNN + Perceiver reference speaker encoder
• finite-scalar speaker quantizer
• conditional acoustic processor
• DAC waveform decoder
• learned FlashSR 16 kHz to 48 kHz upsampler

It also adds a local checkpoint converter, model-spec v1 metadata, documentation, and an upstream-reference parity harness.

Implementation notes

• Mirrors the official MiraTTS defaults: temperature 0.8, top-k 50, top-p 0.95, min-p 0.05, repetition penalty 1.2, and 1024 maximum speech tokens.
• Extends the shared Hugging Face sampler with min_p, covered by unit tests.
• Allows FlashSR to load from an existing tensor source so the converted model remains a single GGUF package.
• Preserves the upstream 8-second reference-audio limit and periodic Hann preprocessing.
• Supports BF16 parity-oriented and Q8 local GGUF conversions.
• Registers tts and clon offline routes through the normal loader/session framework.

Validation

Builds:

cmake --build build\windows-cuda-release --config Release --parallel 12
cmake --build build\windows-vulkan-release --config Release --parallel 12

Focused tests:

ctest --test-dir build\windows-cuda-release --output-on-failure -R "hf_sampler|model_spec_system|flashsr"
python tools\check_loader_catalog_sync.py --self-test
python tools\check_loader_catalog_sync.py

Results:

• CUDA Release build: passed
• Vulkan Release build: passed
• FlashSR, HF sampler, and model-spec tests: 3/3 passed
• Loader/catalog sync: passed
• Loader registration: mira_tts: tts (offline), clon (offline)
• CUDA BF16 end-to-end generation: passed
• CUDA Q8 smoke: passed
• Vulkan Q8 smoke: passed

Representative CUDA BF16 run:

.\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task clon --family mira_tts `
  --model ..\models_v3_test\MiraTTS-GGUF\mira-tts-bf16.gguf `
  --backend cuda `
  --text "Good morning. This is a longer Mira TTS comparison generated with the original implementation and the native audio.cpp port." `
  --voice-ref reference.wav `
  --out mira-native.wav `
  --max-tokens 1024 --temperature 0.8 --top-k 50 --top-p 0.95 `
  --repetition-penalty 1.2 --request-option min_p=0.05 `
  --seed 1234 --metrics

Measured on RTX 3090:

• output: 48 kHz mono
• audio duration: 20.48 s
• wall time: 6.58 s
• RTF: 0.321
• throughput: 3.11x real-time

Reference parity using identical original speech/context tokens:

• waveform correlation: 0.9999646
• SNR: 41.15 dB
• MAE: 0.0004608
• reference preprocessing and all 32 speaker codes match

Conversion

python tools/community_models/convert_mira_tts.py /path/to/MiraTTS /path/to/mira-native
audiocpp_gguf \
  --input language_model=/path/to/mira-native/language_model.safetensors \
  --input speaker_encoder=/path/to/mira-native/speaker_encoder.safetensors \
  --input processor=/path/to/mira-native/processor.safetensors \
  --input decoder=/path/to/mira-native/decoder.safetensors \
  --input upsampler=/path/to/mira-native/upsampler.safetensors \
  --output /path/to/mira-native/mira-tts.gguf --type bf16 \
  --family mira_tts --root /path/to/mira-native

GGUF for testing
https://huggingface.co/mirek190/audio.cpp/blob/main/Text%20to%20audio%20(TTS)/mira-tts-q8.gguf

Current limitations

• Reference audio is required.

[mirek190]

Added native MiraTTS segment-level streaming in afa1b64.

The streaming session now:

• registers both tts and clon as offline|streaming
• splits input at framework text boundaries (text_chunk_size, default 160)
• encodes the reference speaker once and reuses its 32 context codes
• synthesizes and emits each completed 48 kHz segment immediately
• returns the concatenated final result after all events
• supports the normal OpenAI-style SSE speech streaming route

Validation performed:

• CUDA Release build: passed
• Vulkan Release build: passed
• focused FlashSR, HF sampler, model-spec, and CLI request tests: 4/4 passed
• loader/catalog synchronization: passed
• CLI three-sentence stream: five progressive model chunks, 22.66 s final 48 kHz mono audio
• HTTP /v1/audio/speech with stream_format=sse: eight speech.audio.delta transport events, one speech.audio.done, and [DONE]
• single-chunk offline/streaming equivalence: byte-identical WAVs (SHA256 7313AEF29FB91C8E1EFB2BB8D59AE5FEBF5C966560E6F0EBABAF7EE48D9548AF)

This is genuine progressive segment streaming. Each segment still runs the acoustic processor, DAC decoder, and FlashSR as a complete unit; token-level neural-codec streaming remains a possible future optimization.

[mirek190]

Added commit fb2150b with the MiraTTS validation and performance baseline.

Included

• long-lived native CUDA warm benchmark covering cold, repeat, changed-prompt, long-form, and post-long-form repeat requests
• separate streaming benchmark with first-event latency and progressive event checks
• deterministic hash, duration, RTF, WAV readability, and lifecycle validation
• revision-pinned upstream Python benchmark using the same request sequence
• native/upstream waveform and log-mel parity comparison tool
• documented CUDA, streaming, memory, decoder-parity, backend, and performance results

Original MiraTTS versus audio.cpp Q8

Test	Original MiraTTS	audio.cpp Q8	Difference
Cold request, 5.12 s audio	2.227 s	1.707 s	audio.cpp 1.30× faster*
Repeated request, 5.12 s	0.911 s	1.561 s	Original 1.71× faster
Second short prompt, 5.12 s	0.895 s	1.566 s	Original 1.75× faster
Long-form, 15.36 s	2.976 s	4.771 s	Original 1.60× faster
Repeat after long-form, 5.12 s	0.918 s	1.584 s	Original 1.73× faster

* The original cold request paid a one-time ONNX Runtime CUDA fallback cost; warm measurements are the representative steady-state comparison.

The three warm short requests average 0.908 s upstream and 1.570 s in native Q8, making the original implementation about 1.73× faster in this measurement. For the 15.36-second long-form case, the original achieved RTF 0.1938 (5.16× real time), while native Q8 achieved RTF 0.3106 (3.22× real time).

This is not precision-matched: the original uses BF16 while native uses GGUF Q8. The original also pre-encodes and retains the reference context, whereas native currently processes the reference input on each request.

CUDA offline — long-lived session

Request	Wall ms	Audio s	RTF	FNV-1a audio hash
clone_cold	1707.41	5.12	0.3335	426924e4695337c6
clone_repeat	1560.66	5.12	0.3048	426924e4695337c6
short_second_prompt	1566.03	5.12	0.3059	8255f4342afeb014
longform	4771.24	15.36	0.3106	5708fcccf7c25080
clone_repeat_after_longform	1583.74	5.12	0.3093	426924e4695337c6

The three identical requests remain bit-deterministic before and after the changed-prompt and long-form requests.

CUDA streaming

Request	Wall ms	First event ms	Events	Audio s	RTF	FNV-1a audio hash
stream_cold	6639.50	2489.93	4	21.14	0.3141	26eb3542a073a8db
stream_repeat	6498.52	2368.89	4	21.14	0.3074	26eb3542a073a8db

Streaming emits multiple independently consumable audio events, and the merged repeat is bit-deterministic.

Python/native decoder parity

Metric	Python	Native	Result
Sample rate	48000 Hz	48000 Hz	Match
Frame count	122880	122880	Exact match
Waveform cosine	—	—	0.99996735
Log-mel cosine	—	—	0.99943239

This exact-token comparison uses identical upstream speech and context tokens, isolating the native processor/decoder path from autoregressive sampling.

Resident memory — Q8 CUDA

Measurement	Resident usage
Process working set	1429.54 MiB
Process private bytes	5958.28 MiB
Windows GPU Process Memory dedicated usage	2551.55 MiB
nvidia-smi total device memory in use	2552 MiB

These are held-resident samples rather than instrumented peaks across model loading.

Backend coverage

Backend	Build	Runtime result	Status
CUDA, RTX 3090	Passed	Offline, streaming, determinism, and WAV validation passed	Validated
Vulkan, RTX 3090	Passed	Deterministic, but under-generated (3.44 s vs 5.12 s) and diverged on other prompts	Not parity-clean
Vulkan, AMD integrated GPU	Passed	Deterministic, but generated only 0.10 s for the 5.12 s CUDA case	Not parity-clean
CPU	Not run	No runtime measurement recorded	Untested

Validation passed for all saved CUDA offline and streaming cases.

[mirek190]

I am working on performance optimization now and seems a shared Qwen runtime bug: the cached graph’s compact-logit index tensor is overwritten by the allocator after its first use, causing an illegal CUDA access on request two uder
audio.cpp\src\framework\modules\transformers\qwen_causal_decode_runtime.cpp

[mirek190]

MiraTTS Q8 performance update

Commit 5a47c2a substantially improves the native MiraTTS Q8 path on an RTX 3090.

Test	Previous audio.cpp Q8	Optimized audio.cpp Q8	Improvement
Cold request, 5.12 s audio	1707 ms	874 ms	48.8% faster
Repeated request, 5.12 s audio	1561 ms	798 ms	48.9% faster
Second short prompt, 5.12 s audio	1566 ms	798 ms	49.0% faster
Long-form, 15.36 s audio	4771 ms	2530 ms	47.0% faster
Repeat after long-form, 5.12 s audio	1584 ms	835 ms	47.3% faster

Compared with the previously measured warm original MiraTTS implementation, optimized audio.cpp is now approximately 9–15% faster on the repeated and long-form cases. The original cold measurement includes its one-time ONNX fallback/setup cost, so the warm comparisons are the useful ones.

What changed

• Compact LM logits readback: MiraTTS can generate only its 8,192 speech-code tokens plus EOS. The native runtime now keeps the full tied output projection for model semantics, but copies and samples only that 8,193-token alphabet instead of transferring and CPU-sorting all ~166k vocabulary logits at every autoregressive step.
• CUDA DAC fast path enabled: the MiraTTS decoder graph now declares the active backend when it is built. It had silently been treated as a CPU-style graph description and therefore missed audio.cpp's optimized CUDA ConvTranspose1d path. DAC execution for 5.12 seconds of output fell from roughly 375 ms to 32 ms.
• Reference voice reuse: encoded speaker context is cached in a bounded session cache. The default is one reference; --session-option reference_cache_slots=<n> adjusts it and 0 disables it.
• Stable compact prefill handling: compact-logit index tensors are declared as graph inputs, and compact prefill graphs are rebuilt safely to avoid the reused-CUDA-graph illegal-memory-access observed on the second request. Rebuilding costs about 1 ms while preserving the much larger logits-readback saving.
• Stage profiling: prompt, generator, processor, DAC build/upload/compute/readback, FlashSR, reference hashing, and reference encoding timings are now recorded.
• Memory-conscious decoder behavior: a retained two-shape DAC graph cache was tested but rejected. It saved only about 8 ms while increasing private process memory by roughly 2.6 GB after short- and long-form requests, so the final patch does not keep those graph buffers resident.

Validation

Check	Result
Exact output frame counts	Pass, all 5 offline cases
Minimum waveform cosine vs previous native output	0.9999929
Minimum log-mel cosine vs previous native output	0.9999813
Offline cold/repeat determinism	Pass
Streaming repeat determinism	Pass
Streaming output	21.46 s in 3.46 s, RTF 0.161, 4 events
CUDA build/runtime	Pass
Vulkan build/resident-session runtime	Pass

The CUDA fast ConvTranspose path changes floating-point operation ordering, so WAV hashes differ from the slower path; the strict frame-count and high waveform/log-mel similarity checks confirm that the resulting audio remains effectively equivalent.

[0xShug0]

@mirek190 Could you isolate the Qwen3 runtime changes into a separate PR? It would make it easier to track commits that introduce behavioral changes. The only affected model by the change is Music3 and I'm not sure about the perf impact on low end GPUs. Another solution is probably not to disable reuse. It should mark the token-id tensor as input, then re-upload prefill_logits_readback_token_ids_ in run_prefill(), the same way the runtime already re-uploads positions and attention mask before cached graph compute.

BTW, do you know how to stop people from flooding the repo with draft PRs? Pull request limits do not work.

[mirek190]

@mirek190 Could you isolate the Qwen3 runtime changes into a separate PR? It would make it easier to track commits that introduce behavioral changes. The only affected model by the change is Music3 and I'm not sure about the perf impact on low end GPUs. Another solution is probably not to disable reuse. It should mark the token-id tensor as input, then re-upload prefill_logits_readback_token_ids_ in run_prefill(), the same way the runtime already re-uploads positions and attention mask before cached graph compute.

Sure

#426

BTW, do you know how to stop people from flooding the repo with draft PRs? Pull request limits do not work.

If they do not listen ... ban them ;)
I saw it. It looked like a spam.

It is not worth to giving them attention.

[mirek190]

Updated this PR with MiraTTS CPU performance improvements and a Vulkan generation workaround, and merged upstream main at cd98afb.

Commits: 2fc42af (main merge) and d3aeba2 (CPU/Vulkan changes).

Changes

• Added a direct CPU FlashSR implementation that avoids the large intermediate buffers created by the generic convolution graph. The generic path remains available with AUDIOCPP_FLASHSR_DSP=0 for comparison.
• Limited the CPU output projection to the contiguous vocabulary window containing MiraTTS speech tokens and EOS, skipping unused text-vocabulary rows. This uses an optional row offset in the shared Qwen runtime; default behavior for other callers is preserved. AUDIOCPP_MIRA_TTS_SPARSE_HEAD=0 disables the optimization.
• Fixed the CPU speaker encoder's non-contiguous singleton transpose before repeat by reshaping the contiguous tensor.
• Rebuild MiraTTS CPU generation graphs between requests while keeping weights resident, addressing seeded output differences after a longer request.
• For Vulkan, run the MiraTTS Qwen generator on CPU and keep the acoustic stages on Vulkan. This is a correctness workaround for corrupted output observed with the Vulkan generator, not full GPU acceleration; CPU generation remains its performance bottleneck.
• Resolved the shared Qwen runtime merge conflict by retaining upstream's compact-logit input lifetime fix and input re-upload before execution, including reusable prefill graphs.

Validation

CPU, CUDA and Vulkan CLI targets rebuilt successfully against merged main. flashsr_utility_test passed. Post-merge Q8 smoke synthesis used Hello, this is a native Mira TTS test., assets/resources/b.wav, and seed 1234 (CPU: 32 threads; GPU: RTX 3090).

Backend	Request time	Output duration	RTF
CPU Q8	7.246 s	5.980 s	1.212
CUDA Q8	0.894 s	4.640 s	0.193
Vulkan Q8, CPU generator	7.504 s	5.980 s	1.255

CPU and CUDA Q8 WAV hashes match their pre-merge outputs exactly. Vulkan completes with the same duration; its WAV hash differs after the upstream backend update. These checks do not establish cross-backend waveform parity or subjective audio quality.

Before the merge, Q8 and BF16 synthesis completed on all three backends and produced valid mono 48 kHz WAVs. Earlier validation of this optimization work also covered repeated CPU requests, a request after long-form synthesis, and segment streaming. The latest post-merge checks above are Q8 smoke tests.

For the CPU optimization comparison, the same current implementation improved from approximately 21.65 s to 7.18 s before the merge (about 3x). The separate experimental CPU implementation measured 5.31 s; its specialized generation runtime has not been transplanted wholesale, so that performance gap remains.

Build scope: the CLI targets passed; a build of all CPU targets encountered existing Windows setenv compilation errors in unrelated ControlFoley, Higgs STT and Hviske benchmark targets. Those unrelated benchmarks were not modified in this PR update.

[mirek190]

MiraTTS now uses the Vulkan GPU generator, with no CPU-generator fallback. The generator runs on the selected Vulkan device alongside the acoustic stages.

For the short test sentence, generation dropped from approximately 7.5 seconds with the previous CPU-generator fallback to 1.1 seconds on Vulkan GPU (roughly 6-7x faster). The latest Q8 run measured 1.154 seconds for 5.56 seconds of output audio on an RTX 3090. These are local before/after generation measurements, not a controlled benchmark of identical output waveforms, and exclude process startup/model loading.

The fix uses F32 projection precision, grouped flash attention with materialized grouped K/V heads, and F32 output-projection inputs for the Vulkan generator. DirectSetRows KV updates and compact logits remain enabled.

Validation covers Q8 and BF16 offline generation, repeated requests before/after a long request, and progressive streaming. Repeated audio hashes match within each precision. Both full short-sentence samples transcribe correctly with Qwen3-ASR. Other Vulkan devices/vendors and exact CUDA waveform parity are not claimed. Detailed timings are in tests/mira_tts/VULKAN_GPU_VALIDATION.md.

I also merged the latest main (a8fccb4); the merge was clean. The failed macOS CPU check was caused by an unconditional omp.h include and omp_set_num_threads call in FlashSR, despite CI disabling OpenMP. These are now guarded by _OPENMP, allowing serial execution without requiring OpenMP. A local build with ENGINE_ENABLE_OPENMP=OFF and GGML_OPENMP=OFF successfully built the CLI, server, and GGUF converter. Loader/catalog consistency checks also pass. Fresh CI checks will verify the platform builds.

[mirek190]

Current cross-backend sentence benchmark\n\nTest sentence: Hello, this is a native Mira TTS test.\n\nAll six native WAVs were checked with Qwen3-ASR, which recognized the complete requested sentence.\n\n| Implementation | Precision | Generation | Audio length | Compared with original |\n|---|---:|---:|---:|---:|\n| Original MiraTTS, warm | BF16 | 0.851 s | 5.56 s | Baseline |\n| audio.cpp CUDA | Q8 | 0.905 s | 4.64 s | 1.06x slower |\n| audio.cpp CUDA | BF16 | 1.057 s | 5.28 s | 1.24x slower |\n| audio.cpp Vulkan | Q8 | 1.146 s | 5.68 s | 1.35x slower |\n| audio.cpp Vulkan | BF16 | 1.140 s | 5.26 s | 1.34x slower |\n| audio.cpp CPU | Q8 | 11.455 s | 5.40 s | 13.47x slower |\n| audio.cpp CPU | BF16 | 12.760 s | 5.70 s | 15.00x slower |\n\nThe revision-pinned original implementation's cold request took 2.218 seconds because it paid a one-time ONNX/CUDA fallback cost. Its representative warm request took 0.851 seconds.\n\nThis is not a perfectly precision- or workload-matched comparison: upstream uses BF16 and retains the encoded reference context, whereas audio.cpp currently processes the reference input on every request. Native CLI metrics exclude model loading. The generated durations also vary because each backend/precision can sample a different valid token sequence despite using the same text, reference, and seed.\n\nThe Vulkan results use the new GPU generator path with no CPU-generator fallback. On this exact run, Vulkan completed generation in about 1.14 seconds, compared with approximately 7.5 seconds through the former CPU fallback.

[0xShug0]

Thanks @mirek190! I can’t test this right now, but a quick skim found one concern:

if (row_offset != 0 || out.lm_head.weight.shape.dims[0] != row_count) {

This changes shared QwenCausalDecodeRuntime behavior for any caller whose physical lm_head rows differ from decoder.logits_size, even when lm_head_row_offset is not explicitly used.

Since this sparse lm-head window is only needed by MiraTTS, I’d prefer to keep this model-local, so QwenCausalDecodeRuntime does not need to change at all. E.g.,FireRedTTS3 keeps a local lm-head graph.

BTW, we got the official VibeASR integration and audio.cpp is able to support BitNet models!