DiffSinger Refined Phonemizer Implementation

[overdramatic]

DiffSinger Refined Phonemizer Implementation

Overview

This PR introduces the DiffSingerRefinedPhonemizer - an abstract base class based on DiffSingerBasePhonemizer and DiffSingerG2pPhonemizer, that provides comprehensive infrastructure for DiffSinger-based phonemizers in OpenUtau. The implementation delivers advanced phoneme processing capabilities with sophisticated phoneme replacement systems based on replacements lists, word-level, timing-based and duration-dependent modifications

Key Features

• Advanced Phoneme Processing: Supports single, merge (M:1), split (1:M), and many-to-many phoneme replacements
• Multi-language Support: Handles language-specific phoneme mappings and prefixes
• Word-Level Editing: Enables cross-phoneme modifications across entire words
• Comprehensive G2P System: Integrates grapheme-to-phoneme conversion with customizable dictionaries

Benefits

• Production Ready: Comprehensive error handling and logging
• High Performance: Optimized processing with caching
• Flexible: Extensive customization points
• Robust: Graceful fallbacks and validation
• Maintainable: Clean architecture with clear separation of concerns

This implementation provides a solid foundation for sophisticated singing synthesis with DiffSinger, supporting complex linguistic phenomena while maintaining performance and extensibility.

Usage

Derive from DiffSingerRefinedPhonemizer and override:

• EditPhonemesForWord() for word-level modifications
• EditTimedPhonemes() for timing-based transformations
• ApplyDurationBasedReplacements() for duration-dependent changes

For dsdict.yaml replacements:

replacements:
- {from: ng, to: pt/g} #Single Replacement
- {from [k, w] to: pt/kw} #Merge
- {from gw to: [pt/g, pt/w]} #Split
- {from: [k, w, an], to: [pt/s, pt/t, pt/r, pt/a]} #Many-to-Many

It uses the same rules as DiffSingerG2pPhonemizer with from being without the language tag and to with the tag if the model has multi-dict support
The phonemizer folows the replacements in logical order: single -> merge -> split -> many-to-many

[hecko-yes]

Might it make more sense to reverse the order of replacements, or even use the order as listed in the .yaml file for maximum flexibility? For example, this isn't possible if you process single replacements before everything else:

replacements:
- {from: [s, p], to: [s, p_]}
- {from: p, to: p_h}

[overdramatic]

Might it make more sense to reverse the order of replacements, or even use the order as listed in the .yaml file for maximum flexibility? For example, this isn't possible if you process single replacements before everything else:

replacements:
- {from: [s, p], to: [s, p_]}
- {from: p, to: p_h}

I don't get what you are trying to say, but this base class follows the replacement order:
single -> merge -> split -> many-to-many

For more sophisticated or in depth customization, it's better to make inside the phonemizer code

[yqzhishen]

What's the use case of this?

[overdramatic]

What's the use case of this?

@yqzhishen you can see it in this PR: #1841

[yqzhishen]

@overdramatic I don't think these hardcoded linguistic rules are the right philosophy for a machine learning model, which should have context capability.

[overdramatic]

@yqzhishen can you please tell me what hardcoded linguistic rules do you think it should not be included?
On BRAPA (Brazilian Portuguese) cases, those hardcoded linguistics on our phonemizer, like the word boundary rules, is because the G2P on OpenUTAU is very limited. They work in a word level and not in a sentence level, so it doesn't have context capability to change that said boundaries.
The same goes for the duration based rules: Since a G2P doesn't get those phoneme timings, and, on our tests with DiffSinger, trying to merge /a/ /i/ /u/ with their reduction variants /ax/ /i0/ /u0/, could give inconsistencies on inference, it was our best approach to get those reductions right in the phonemizer.
And those options can be turned off/skipped if you add a / on the start of the word or input the phonemes with [ ]

Hope this wikipedia page helps you understand why those hardcoded rules is needed:
Sandhi / Sandhi Portuguese examples

[agentasteriski]

Please reconsider the decision to close this. It may not seem helpful to languages with simple phonetic rules where everything can be matched precisely to the writing, but the rest of us have been begging for this to merge.

[yqzhishen]

The problem wasn't in G2P; the real problem is that you expected G2P to handle what it shouldn't be responsible for.

You proposed 4 categories of linguistic rules:

1. duration-driven reduction variants. DiffSinger is nn-based model, which already takes duration and phoneme IDs as inputs. If the voicebank is made well, it should have already learned to weaken pronunciation on short phones. Adding such manual rules is not only redundant, but also harmful to the semantics of context sequence; not to mention the weird hardcoded threshold which is definitely ungeneralizable.
2. Sandhi rules across words. Again, the model receives full sequence, so it already sees these conjunction patterns. These manual rules are naive statistical trends of connected speech phenomena, and won't be better than what the model should have learned.
3. Dummy or placeholder symbols. These things blurred the lines of responsibility; current protocol is fragile, not commonly recognized, and hard to maintain. From G2P side, it'd better produce a standard form of a specific phoneme; from voicebank side, the interpretion rules are static, thus won't be better than model predictions.
4. Singing technique tags. Out of scope, not G2P-related. cl is Japanese sokuon; vf isn't a phoneme from phonetics perspective.

The PR mainly follows the philosophy of symbolic phonology, proposing explicit knowledge, cascaded inference and universal rules, where correctness is defined by linguists. However, neural acoustics proposes implicit knowledge and end-to-end mapping, where information and context are fundamental concerns, and the behavior is driven by data.

I would like to quote Frederick Jelinek, pioneer of modern speech recognition and natural language processing research, for his famous statement,

Every time I fire a linguist, the performance of the speech recognizer goes up.

I hope my explanation help you find the right way to deal with modern singing voice synthesis.

[nagotown]

@yqzhishen maybe i'm misunderstanding, but based on the logic being mentioned, shouldn't only the BRAPA phonemizer be denied instead of the API? what if i wanted to set replacements for tr -> t, r on an english phonemizer?

[yqzhishen]

@yqzhishen maybe i'm misunderstanding, but based on the logic being mentioned, shouldn't only the BRAPA phonemizer be denied instead of the API? what if i wanted to set replacements for tr -> t, r on an english phonemizer?

Makes sense, sorry. I will review this when I have time.

If there are further discussion around BRAPA, let's move to #1841.

[gnloop]

The problem wasn't in G2P; the real problem is that you expected G2P to handle what it shouldn't be responsible for.

You proposed 4 categories of linguistic rules:

1. duration-driven reduction variants. DiffSinger is nn-based model, which already takes duration and phoneme IDs as inputs. If the voicebank is made well, it should have already learned to weaken pronunciation on short phones. Adding such manual rules is not only redundant, but also harmful to the semantics of context sequence; not to mention the weird hardcoded threshold which is definitely ungeneralizable.
2. Sandhi rules across words. Again, the model receives full sequence, so it already sees these conjunction patterns. These manual rules are naive statistical trends of connected speech phenomena, and won't be better than what the model should have learned.
3. Dummy or placeholder symbols. These things blurred the lines of responsibility; current protocol is fragile, not commonly recognized, and hard to maintain. From G2P side, it'd better produce a standard form of a specific phoneme; from voicebank side, the interpretion rules are static, thus won't be better than model predictions.
4. Singing technique tags. Out of scope, not G2P-related. cl is Japanese sokuon; vf isn't a phoneme from phonetics perspective.

The PR mainly follows the philosophy of symbolic phonology, proposing explicit knowledge, cascaded inference and universal rules, where correctness is defined by linguists. However, neural acoustics proposes implicit knowledge and end-to-end mapping, where information and context are fundamental concerns, and the behavior is driven by data.

I would like to quote Frederick Jelinek, pioneer of modern speech recognition and natural language processing research, for his famous statement,

Every time I fire a linguist, the performance of the speech recognizer goes up.

I hope my explanation help you find the right way to deal with modern singing voice synthesis.

This isn't entirely correct.

I can somewhat agree with point 1.

But something like labeling 's' and 'sh' both as 's' just because "the model will learn the difference based on context" feels very silly.

When a simple edit in the phonemizer can just turn the 's' into 'sh' BEFORE inference why bother? It's an extremely easy fix and it can be used for multiple languages and multiple use cases, not just BRAPA, and it takes away the guess work for the model (especially when using lower amounts of data).

It just gives people more option without having to make an entirely new phonemizer.

For example, the new update to the Italian phonemizer broke some things with the previous system, so I'd have to correct all the labels, or I could just use this replacement system (personally I don't agree with the changes, I would prefer the old labeling style).
It also applies to the DIFFS EN+ phonemizer: it forces the user to use tr and dr phoenmes, but not everyone uses those, so they are stuck with the DIFFS EN phonemizer which has less accuracy for general singing purposes.

Going back to BRAPA specifically, letting the AI guess things is a bad idea when taking into account that there are different rules in different Brazilian regional dialects (or european Portuguese) that often contradict each other, so this is the best idea, since the alternative would be making many different phonemizers.

[overdramatic]

@yqzhishen maybe i'm misunderstanding, but based on the logic being mentioned, shouldn't only the BRAPA phonemizer be denied instead of the API? what if i wanted to set replacements for tr -> t, r on an english phonemizer?

Makes sense, sorry. I will review this when I have time.

If there are further discussion around BRAPA, let's move to #1841.

Thank you!
I've made a reply, hope you can see it soon
#1841 (comment)