Skip to content

fix: ROUGE-1 eval returns 0 for non-English languages (ASCII-only tokenizer)#6136

Open
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/non-english-eval-rouge
Open

fix: ROUGE-1 eval returns 0 for non-English languages (ASCII-only tokenizer)#6136
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/non-english-eval-rouge

fix: use proper tokenizer class for Unicode RougeScorer

f5e5c40
Select commit
Loading
Failed to load commit list.
Google CLA / cla/google succeeded Jun 16, 2026 in 7s

✅ All contributors are covered under a CLA with Google

See https://cla.developers.google.com/ for more info about Google's Contributor License Agreement (CLA).

ℹ️ Googlers: Go here to view more details and manage scans for this pull request.

Details

The following contributors were found for this pull request:

f5e5c40 Author: @tcconnally <51974392+tcconnally​@users.noreply.github.com>
891fb52 Author: @rohityan <rya******la​@google.com>

(Only the first commit for a unique contributor is listed.)