fix(models): undo Moses entity escaping in MADLAD-400 output

[ShaiShmuel]

What is wrong

1. For some target languages, MADLAD-400 returns text in the Moses tokenizer's escaped form: ', " and & come back as ', " and &, with a space on each side.
2. With the target tag <2fr_CA>, d'Android comes back as d ' Android. The same source with <2fr> comes back clean. Arabic, Hebrew and Russian outputs sometimes contain " ... " or a spaced &.
3. The CTranslate2 adapter returns the decoded text unchanged, so the entities and the extra spaces reach the client.

Net effect: in a WMT24++ sample, 34 of 48 fr_CA translations contained ', plus a few Arabic, Hebrew and Russian outputs with " or &.

Change

• We added an unescape_moses_entities load-time option to the CTranslate2 generation adapter. It is off by default, and we turn it on only in the google/madlad400-3b-mt profile.
• When it is on, the adapter unescapes the Moses entity set (& < > ' " [ ] |) in the decoded output. It skips any entity that occurs literally in the input, so a request that translates & keeps it. Unescaping is a single pass, so < becomes <, not <.
• We repair spacing only where the form is unambiguous:
  • French elision, only when the prompt's target tag is French (<2fr>, <2fr_CA>): l ' offre becomes l'offre. This covers c', d', j', l', m', n', s', t', the qu' family and aujourd'hui.
  • Balanced, space-delimited " pairs: " Oui " becomes "Oui".
  • Everything else is unescaped and keeps its spaces. R & D becomes R & D. An odd number of quotes, or a literal " elsewhere in the output, is not respaced.
• Generation parameters, token counts and finish reasons are unchanged.

Sentences that come back joined without a space are a separate issue and are not changed here.

Tests

• test_ctranslate2_moses_entities.py (new): elision, quote pairs, ambiguous quotes, the other entities, single-pass unescaping, entities present in the source, and text without entities. All test strings are synthetic.
• test_ctranslate2_generation.py: the option is off by default, unescapes when on, keeps an entity that the prompt contains, and rejects a non-boolean value.
• We replayed the function offline over the outputs of the WMT24++ sample. Every output with an entity was repaired, no output without one changed, and no entity remained. One output ends with a single unmatched quote, so it keeps its space as intended.

# 62 passed
mise exec -- uv run --frozen --project . --no-sync pytest -q packages/sie_server/tests/adapters/test_ctranslate2_moses_entities.py packages/sie_server/tests/adapters/test_ctranslate2_generation.py
# 2383 passed, 518 skipped (includes the catalog load-time option gate for MADLAD)
mise exec -- uv run --frozen --project . --no-sync pytest -q packages/sie_server/tests/config packages/sie_server/tests/core/test_loader.py packages/sie_server/tests/core/test_model_loader_cache.py packages/sie_server/tests/core/test_registry_memory.py packages/sie_server/tests/test_model_yaml_filenames.py packages/sie_server/tests/adapters/test_ctranslate2_generation.py packages/sie_server/tests/adapters/test_ctranslate2_moses_entities.py
# 30 passed
mise exec -- uv run --frozen --project . --no-sync pytest -q packages/sie_server/tests/test_ipc_server.py -k "madlad or generat"
# all checks passed
mise run lint
mise run typecheck -- packages/sie_server

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added optional Moses entity unescaping for translation output.
  • Improved handling of French apostrophe elisions and unambiguous quoted text spacing.
  • Enabled this behavior by default for the Google MADLAD-400 model configuration.

• Bug Fixes

  • Preserves entities already present in the source text.
  • Keeps ambiguous punctuation and spacing unchanged to avoid unintended output changes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Team
• Run ID: 80907aca-d7d9-4d25-ae17-0cf4a2e32011

📥 Commits

Reviewing files that changed from the base of the PR and between b80a029 and 4488cc0.

📒 Files selected for processing: 5

• packages/sie_server/models/google__madlad400-3b-mt.yaml
• packages/sie_server/src/sie_server/adapters/ctranslate2/_moses_entities.py
• packages/sie_server/src/sie_server/adapters/ctranslate2/generation.py
• packages/sie_server/tests/adapters/test_ctranslate2_generation.py
• packages/sie_server/tests/adapters/test_ctranslate2_moses_entities.py

Included review availability: This review used your included allowance. 9 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 10 reviews per hour.

📝 Walkthrough

CTranslate2 generation can optionally unescape Moses entities using the original prompt to preserve entities already present in the source. The helper also repairs recognized French elisions and unambiguous escaped quote spacing. The MADLAD model configuration enables this option.

Changes: Moses entity unescaping

Layer / File(s)	Summary
Entity decoding and spacing rules packages/sie_server/src/sie_server/adapters/ctranslate2/_moses_entities.py, packages/sie_server/tests/adapters/test_ctranslate2_moses_entities.py	The helper unescapes eligible entities once, preserves entities present in the source, and handles recognized French elisions and unambiguous escaped quote pairs. Tests cover these rules and cases where spacing remains unchanged.
Generation option and output processing packages/sie_server/src/sie_server/adapters/ctranslate2/generation.py, packages/sie_server/models/google__madlad400-3b-mt.yaml, packages/sie_server/tests/adapters/test_ctranslate2_generation.py	The adapter validates and stores the optional setting, retains the prompt, and applies unescaping to decoded output when enabled. The MADLAD model configuration enables the option. Adapter tests cover enabled and disabled behavior, source entity preservation, French target handling, and option validation.

Suggested reviewers: svonava

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to 4488c

This change decodes escaped entities in MADLAD-400 translation output and fixes French elision and quote spacing. Other models are unaffected because the option is off by default. No actionable merge-blocking risk was identified.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 4 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly identifies the main change: enabling Moses entity unescaping for MADLAD-400 output.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 4 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Fix all pre-merge checks with AI

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}