Support language variants

[kelson42]

MWoffliner allows now to specify language variants (and this is used to scrape Wikipedia in Simplified Chinese). See openzim/mwoffliner#2450

Which mostly implies that we mostly agreed that we can/could/should have multiple ZIM files with the same content, but different language variants.

But beside of this I believe there is nothing set:

• For the ZIM specification (metadata)
• How the readers should deal with that

[rgaudin]

Where is the documentation regarding those variant handling? I see on https://zh.wikipedia.org/wiki/马里 that there is a drop-down menu that lists several variants and each is a copy of the article apparently in said variant.

It's impossible for me (I can't read any of the Chinese languages) to understand what this is exactly: same content automatically translated ? copies of a source article (which variant) that are manually translated? Articles that are handled independently and thus can/may contain different information?

What lead to them implementing this? If they decided to maintain a single wiki with a specific extension there must be a reason. That should feed our discussion I believe.

[benoit74]

Very interesting blog post on the matter: https://diff.wikimedia.org/2018/03/12/supporting-languages-multiple-writing-systems/

Most interesting documentation is probably:

• https://meta.wikimedia.org/wiki/Automatic_conversion_between_simplified_and_traditional_Chinese
• https://meta.wikimedia.org/wiki/Wikipedias_in_multiple_writing_systems

We should probably generate ZIMs in all variants for all projects using it in a reliable manner, not only Chinese ones.

[kelson42]

One of the thing related to this issue which seems clear Filename and ZIM "Name" metadata should be variant specific and include (lowercased) the variant version of the language.

This means for example:

• wikipedia_zh-hant_all_maxi_2025-10.zim as a filename
• wikipedia_zh-hant_all_maxi as ZIM metadata "Name".

@benoit74 @rgaudin Does that sounds reasonnable? What other points should we consider?

[benoit74]

I mostly agree about filename and ZIM metadata Name.

My concern is that zh-hant is confusing and not officially supported by Wikimedia. Official variants seems to be: zh-cn, zh-hk, zh-mo, zh-my, zh-sg, and zh-tw according to the doc; I have no idea what the --langVariant zh-hant of wikipedia_zh-hant_all](https://farm.openzim.org/recipes/wikipedia_zh-hant_all/config) recipe could mean.

I would recommend to stick to these variants and avoid zh-hant which I'm pretty sure will only cause confusion and further request to split again into two set of ZIMs.

This means that langVariant in wikipedia_zh-hant_all recipe should be fixed accordingly (and recipe name as well).

Do we also agree that it means wikipedia_zh_*_* should all be renamed/replaced to wikipedia_zh-cn_*_* for clarity?

What about ZIM metadata language? Do we agree that both zh-hans and zh-hant versions of zh.wikipedia.org ZIMs will have cmn as Language ZIM metadata?

What about description and title? Do we agree that we need to have description and title in the variant being ZIMed? Should we state the variant used in the ZIM in title or description, preferably in the title? This is not what we do for other languages, so it feel a bit odd after some thought. It could something like 繁體中文維基百科 for traditional Chinese (Traditional Chinese Wikipedia) and 简体中文维基百科 for simplified chinese (Simplified Chinese Wikipedia) ; both propositions sourced from google translate, probably worth to validate with native speakers, but this is more to give a sense of what I'm speaking about. But again not that much convinced myself.

[Markus-Rost]

My concern is that zh-hant is confusing and not officially supported by Wikimedia. Official variants seems to be: zh-cn, zh-hk, zh-mo, zh-my, zh-sg, and zh-tw according to the doc; I have no idea what the --langVariant zh-hant of wikipedia_zh-hant_all](https://farm.openzim.org/recipes/wikipedia_zh-hant_all/config) recipe could mean.

zh-hant just means the Traditional Chinese character set is being used, similar to how zh-hans is just the Simplified Chinese characters. Both are internal language variants in MediaWiki and we should probably avoid using them as langVariant directly.

MediaWiki docs suggest using at least zh-cn, zh-tw and zh-hk language variants for Chinese and we should probably follow that, if we don't want to create ZIM files for all 6 variants enabled on zh.wikipedia.org

https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter

• Chinese (zh):
  • Simplified Chinese (zh-hans): China (zh-cn), Singapore (zh-sg), Malaysia (zh-my)
  • Traditional Chinese (zh-hant): Taiwan (zh-tw), Hong Kong (zh-hk),^[1] Macau (zh-mo)

1. Taiwan and Hong Kong are two major variants written in the same Traditional script with significant differences in phrase usage due to market separation and influence from local zho languages, so you likely want to at least keep CN, TW, and HK in your list of variants. If you insist on flattening the scope of Chinese variants to a script-based Simp/Trad separation, follow what the reporter did in phab:T149278.