fix(webvtt): decode cue text escapes and allow them in annotations

[kakiuwang-ui]

A WebVTT file whose speaker name contains an escape loses the whole cue:

WEBVTT

00:00.000 --> 00:02.000
<v Ben &amp; Jerry>We charge 5 &lt; 10 &amp; win</v>

WebVTTFile.parse returns no cue blocks for that, and the only trace is a RuntimeWarning. Downstream in docling the line simply is not in the converted document.

The cause is the start tag pattern, which excludes & from the annotation:

r"(?:[ \t](?P<annotation>[^\n\r&>]*))?>"

<v Ben &amp; Jerry> matches no tag, so the payload falls through to a text span, which rejects it for containing <, and parse drops the block. That contradicts is_valid_annotation a few lines up, which explicitly permits &, <, >, ‎, ‏ and   in an annotation — a validator that could never receive the input it was written for.

The second, quieter half: those six escapes were only ever validated, never decoded. Pride & Prejudice kept the escape verbatim into the document, and per WebVTT 1 §6.3 the escape denotes the character.

Implementation

_ENTITY_CHARS maps each escape to the character it denotes, and _VALID_ENTITIES is derived from it so the whitelist and the decode table cannot drift apart. _unescape resolves escapes when a span is built; __str__ writes them back.

Why decode at construction rather than at the call site. str() on the model is the export path, so decoding without a matching escape on the way out would emit invalid WebVTT. Doing both keeps the model holding the characters a reader should see while the file still round-trips. This follows #744, which already established that parse normalises cue text rather than preserving the source byte for byte.

Which characters are written back. Only &, and the one that would terminate the construct — < in cue text, > in an annotation. The other three are optional escapes, so the literal character is kept and a file that never used them round-trips unchanged. A source that wrote > in cue text comes back as a bare >, which is valid and how the spec allows it to be written.

Behaviour change worth a look

WebVTTCueTextSpan(text="Pride & Prejudice").text used to be the escaped string and is now "Pride & Prejudice". One existing assertion pinned the old value:

valid_text = "My favorite book is Pride & Prejudice"
span = WebVTTCueTextSpan(text=valid_text)
assert span.text == valid_text

Reading it in context — it sits right after the &foo; case that must raise — it looks like it was written to say "a valid entity is accepted", with equality as the convenient way to say that, rather than as a statement that escapes must stay raw. I have updated it to assert the decoded text plus str(span) == valid_text, so both directions are pinned. Happy to be told otherwise if keeping text as source-form was deliberate.

Test

• test_webvtt_cue_escapes_in_annotation — the cue above parses to one block, the annotation is Ben & Jerry, the text is We charge 5 < 10 & win, and the block formats back to the original line.
• test_vtt_cue_commponents gains a loop over all six escapes and a check that > comes back as a bare >.

Each fix was checked by breaking it: putting & back in the annotation pattern fails 1, dropping _unescape fails 2, dropping _escape fails 2. tests/test_webvtt.py goes 6 → 7 passed; test_serialization.py + test_docling_doc.py + test_webvtt.py go 186 → 187 with no regressions, which covers the byte-exact round-trip assertion in test_webvtt_file and the .gt.vtt comparisons. No fixture contains an escape today, so none of those baselines move.

Notes for review

• Adjacent gap, deliberately not touched: transforms/serializer/webvtt.py builds cue text from a DoclingDocument without escaping, so a document whose text holds & or < serialises to WebVTT that this parser would then reject. Different path, and it wants its own decision about where escaping belongs. Happy to send it separately.
• The annotation still rejects a bare &, unchanged. Only the pattern was widened, so such input now reaches the validator that was always meant to judge it instead of silently killing the cue.
• Ruby and class annotations are out of scope; the payload is still limited to the spans listed in the WebVTTCueBlock docstring.

[github-actions]

✅ DCO Check Passed

Thanks @kakiuwang-ui, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require two reviewer for test updates	👀 reviews
🟢	Enforce conventional commit	—

🔴 Require two reviewer for test updates

Waiting for

• #approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

---

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[ceberam]

Thanks @kakiuwang-ui for catching this bug and proposing a fix. I have just added a minor suggestion to strengthen the test.

[ceberam]

I would use here _ENTITY_CHARS to cover exactly the escapes that the WebVTT module defines. If _ENTITY_CHARS gains a seventh entry in a future commit, the test automatically covers it; if a mapping is silently changed, the test catches it instead of staying green on stale values.