Skip to content

fix(refinement): stop refining non-English transcripts into English - #1218

Open
Tony25032 wants to merge 1 commit into
jamiepine:mainfrom
Tony25032:fix/refinement-preserves-language
Open

Tony25032 wants to merge 1 commit into
jamiepine:mainfrom
Tony25032:fix/refinement-preserves-language

Conversation

@Tony25032

Copy link
Copy Markdown

Fixes #603

Summary

Non-English dictation gets silently translated: a Chinese (or Korean, Japanese, …) transcript goes into refinement and an English rewrite comes back in transcript_refined. This implements the required fix proposed in #603 — inject the transcript language into the refinement prompt as a hard output constraint.

Root cause

The refinement system prompt (backend/services/refinement.py) is assembled from four English instruction blocks with English-only few-shot examples, and nothing anywhere pins the output language. With temperature=0.2, small refinement models (Qwen3 0.6B/1.7B/4B) pattern-match "the examples are English → the output is English" and treat the task as translate and clean. English dictation hides the bug completely, which is why it went unnoticed.

Reproduction (synthetic Chinese example; see also the Korean repro in #603)

  • Raw STT (language=zh): 「嗯…那个…我们下周开会,啊不对周四周四下午两点开,然后嗯把报告发我一下」
  • Refined before this patch: comes back as an English translation of the same content (observed across Qwen3 0.6B–4B), e.g. "Um… we have a meeting next week, no wait, Thursday at 2 pm, and um, send me the report." — the speaker's wording is silently lost.
  • Refined after this patch: 「我们周四下午两点开会,然后把报告发我一下。」 — stays Chinese, fillers and the mid-utterance self-correction still resolved.

Changes

  • refine_transcript() accepts the Whisper language code and forwards it to build_refinement_prompt().
  • build_refinement_prompt() injects an output-language constraint: a named hard constraint when the language is known ("Your entire output MUST be written in Chinese … Never translate the transcript"), or a same-language rule when the code is unknown. This anchors the model against the English-only examples instead of relying on the model's language-mirroring instincts.
  • refine_capture() forwards the capture row's language.
  • New backend/tests/test_refinement_language.py pins the prompt-level guardrails (pure prompt-builder tests, no model required).

Scope

Deliberately limited to the language-preservation fix requested in #603. The issue's larger asks — localized few-shot examples/filler lists, per-language presets, and a prompt editor UI — are natural follow-ups; happy to work on them if this approach lands.

Testing

  • pytest backend/tests/test_refinement_language.py (new)
  • Existing suites are unaffected: build_refinement_prompt() keeps its previous behaviour when called without language, and refine_transcript()'s new parameter is optional with default None.

Happy to reword the injected constraint if maintainers prefer different phrasing.

The refinement system prompt is written in English with English-only
few-shot examples, so small refinement models read the task as
"translate into the language of the examples" and silently return an
English rewrite of non-English transcripts (jamiepine#603).

- Pass the Whisper language code through refine_transcript ->
  build_refinement_prompt and inject an explicit output-language
  constraint (named language when known, same-language rule when not).
- refine_capture forwards the capture row's language.
- Add prompt-level regression tests.
doublesharp added a commit to doublesharp/voicebox-ds that referenced this pull request Oct 29, 2026
Use the shared QueryClient for offline requests, preserve custom server URLs through startup and migration, retain capture decode diagnostics, preserve refinement language, accept supported STT languages, and explain LuxTTS short-input errors. Adapt upstream PRs jamiepine#1208, jamiepine#1204, jamiepine#1219, jamiepine#1218, jamiepine#1198, jamiepine#1199, jamiepine#1220 and jamiepine#1180; retain remote reconnect refreshes and correctly migrate the legacy bundled port.

Wait for the preview output context to resume before playing. Cancel superseded starts and release preview nodes/object URLs on replacement, completion, failure and unmount. Add CI regressions for the upstream integration and preview lifecycle.

Validation: 292 frontend tests on this candidate; 414 frontend tests with the working UI changes; 53 backend tests; app/web/Tauri frontend typechecks and both frontend builds. Independent review accepted the playback change. Native first-click/autoplay, acoustic output, packaging and model quality remain unverified.

Co-authored-by: vincentfranstyo <66989751+vincentfranstyo@users.noreply.github.com>
Co-authored-by: maxazure <405711+maxazure@users.noreply.github.com>
Co-authored-by: FelixKruger <262215022+FelixKruger@users.noreply.github.com>
Co-authored-by: Tony25032 <141424750+Tony25032@users.noreply.github.com>
Co-authored-by: mvoloch <274143174+mvoloch@users.noreply.github.com>
Co-authored-by: Lvigentini <11659311+Lvigentini@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refinement silently translates non-English transcripts to English; request prompt customization

1 participant