Bake multiple lexical pronunciation into baum-welch

[lenzo-ka]

Multi-pronunciation Baum-Welch training

Baum-Welch in SphinxTrain silently uses pronunciation variant [1] for every
multi-pron word in the dictionary, so (2)/(3) entries receive no acoustic
training unless the transcript token already carries an explicit (N) suffix.

This branch fixes that in the utterance-HMM construction layer. The forward,
backward, viterbi, and accumulator code paths are unchanged; they already
operate on a general DAG via state_t.next_state[] and prior_state[]. The
change replaces the implicit linear phone chain produced by mk_phone_list
with an explicit phone_graph_t carrying parallel paths per pronunciation
variant, then resolves triphone contexts and builds the state sequence over
that graph.

Commits

1. Add lexicon base-word variant index — lex_entry_t.next_variant chain
   and a parallel lexicon_t.base_ht keyed by the base word (reading for
   reading(2)). Existing lexicon_lookup behavior unchanged.

2. Add phone_graph_t and graph-aware state-sequence builder — new
   mk_phone_graph, phone_graph_split_contexts, cvt2triphone_graph, and
   state_seq_make_graph. For a single-pron utterance the graph is a strict
   linear chain and the resulting state array is bit-identical to
   state_seq_make's. Renames the two static helpers in state_seq.c and
   shares them via a new private state_seq_internal.h.

3. bw: add -multipron flag selecting the graph utterance HMM — wires a new
   next_utt_states_graph into bw's main loop, gated by -multipron
   (default no for parity). The graph path allocates state_t per call and
   is freed with state_seq_free; the linear path's static-buffer lifetime
   is unchanged.

4. Add CFG_MULTIPRON_TRAINING knob to pass -multipron to bw — new
   CFG_MULTIPRON_TRAINING (default yes) in etc/sphinx_train.cfg and the
   librispeech template, plumbed into every baum_welch.pl under
   01.lda_train, 02.mllt_train, 10.falign_ci_hmm, 20.ci_hmm,
   30.cd_hmm_untied, 50.cd_hmm_tied, 65.mmie_train, and 80.mllr_adapt.

CFG_MULTIPRON_TRAINING is independent of CFG_MULTIPRON (the stage-21
sphinx3_align disambiguation pass). The two compose: tokens already
disambiguated by stage 21 carry an (N) suffix and collapse to a
single-variant expansion in the graph builder; un-disambiguated tokens still
branch on every dictionary variant during Baum-Welch.

Compatibility

• C-side default -multipron no preserves prior behavior. Each commit
  compiles standalone.
• Cfg-side default CFG_MULTIPRON_TRAINING = 'yes' enables the new path in
  this fork's recipes. Set to 'no' to fall back to the linear builder.

Validation

No automated tests added (the repo's existing Perl test/ harness isn't
wired into CMake and lacks a scaffold for this layer). Validated empirically:

• Each commit builds standalone.
• Trained CMU Arctic SLT end-to-end (CI through CD-4g) twice with multipron off and on, then force-aligned the 54 held-out utterances against both
  models with sphinx3_align -outsent.
  • Same model shape (means/variances/mixw/tmat all identical sizes).
  • Different model contents (cmp shows first diff in CI/CD means around
    byte 60).
  • 491 content tokens compared; 452 (92.1%) same variant, 39 (7.9%)
    different. 29 / 54 utterances had at least one disagreement,
    concentrated on high-frequency function words.
• Matches the reference implementation's numbers (93.6% / 6.4% on a
  slightly different cut).

Net change

~1,050 lines new C, ~80 lines modifications, ~60 lines Perl/config.