You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add files needed for maplibre unicode code geneation script. #103
For context, MapLibre GL JS has a custom text layout implementation because the built-in browser APIs don’t provide low-level access to the text layout engine. The implementation currently relies on downloading these tables directly from the Unicode website. We don’t actually decompose any Arabic characters, but the decomposition mappings indicate if a given presentation form is initial/medial/final/isolate. #102 is helpful, but it turns out that the ArabicShaping table is abridged and there’s a different unabridged table:
The file ArabicShaping.txt is also exceptional, because it omits the listing of many characters whose property value (jt=T) can be derived by rule. Adding an "@missing" line to that file would result in the wrong interpretation of Joining_Type values for omitted characters. The full explicit listing of Joining_Type values and the correct "@missing" line for the default Joining_Type value (jt=U) can be found in the file DerivedJoiningType.txt instead. The values of Joining_Type listed in DerivedJoiningType.txt should be taken as definitive, because of the difficulty of deriving the correct values for all characters based only on the entries in ArabicShaping.txt.
I can take the Decomposition_Mapping part (parse field 5 of UnicodeData.txt, following the parse-bidi-class First/Last + value-alias pattern). One question before I start on DerivedJoiningType: it feeds the same Joining_Type output as the just-merged ArabicShaping parser — would you prefer it to replace that source, or coexist (e.g. as the explicit T-set complement)? Happy to implement either way once you confirm the direction.
Hm, not sure. It kind of looks like MapLibre is downloading and parsing both, but I’m not sure why. The derived table is supposed to be a more comprehensive superset of the base table.
Verified @1ec5's intuition against the 17.0.0 files: DerivedJoiningType (3,004 entries) covers every ArabicShaping Joining_Type value with zero mismatches, plus 2,224 explicit T entries that ArabicShaping leaves implicit. One nuance: 50 code points exist only in ArabicShaping — all Joining_Type=U entries (e.g. U+0600..U+060B), which the derived file omits as implicit default. So for pure Joining_Type consumers the derived file alone suffices; ArabicShaping remains the only source for Joining_Group. Up to @mathiasbynens whether the new parser should complement or migrate — either way my Decomposition_Mapping offer stands as independent work.
Activity
HarelM commentedon Sep 9, 2026
Claude also told me that the following might be something worth changing:
node-unicode-data/scripts/utils.mjs
Line 107 in 144669f
To:
Let me know if this makes sense, again, sorry for pasting AI comments...
1ec5 commentedon Sep 9, 2026
For context, MapLibre GL JS has a custom text layout implementation because the built-in browser APIs don’t provide low-level access to the text layout engine. The implementation currently relies on downloading these tables directly from the Unicode website. We don’t actually decompose any Arabic characters, but the decomposition mappings indicate if a given presentation form is initial/medial/final/isolate. #102 is helpful, but it turns out that the ArabicShaping table is abridged and there’s a different unabridged table:
FekyBaz commentedon Sep 9, 2026
I can take the Decomposition_Mapping part (parse field 5 of UnicodeData.txt, following the parse-bidi-class First/Last + value-alias pattern). One question before I start on DerivedJoiningType: it feeds the same Joining_Type output as the just-merged ArabicShaping parser — would you prefer it to replace that source, or coexist (e.g. as the explicit T-set complement)? Happy to implement either way once you confirm the direction.
1ec5 commentedon Sep 10, 2026
Hm, not sure. It kind of looks like MapLibre is downloading and parsing both, but I’m not sure why. The derived table is supposed to be a more comprehensive superset of the base table.
FekyBaz commentedon Sep 10, 2026
Verified @1ec5's intuition against the 17.0.0 files: DerivedJoiningType (3,004 entries) covers every ArabicShaping Joining_Type value with zero mismatches, plus 2,224 explicit T entries that ArabicShaping leaves implicit. One nuance: 50 code points exist only in ArabicShaping — all Joining_Type=U entries (e.g. U+0600..U+060B), which the derived file omits as implicit default. So for pure Joining_Type consumers the derived file alone suffices; ArabicShaping remains the only source for Joining_Group. Up to @mathiasbynens whether the new parser should complement or migrate — either way my Decomposition_Mapping offer stands as independent work.
HarelM commentedon Sep 15, 2026
@mathiasbynens any chance to add those files? I see you just released a new version, would be nice to have those files in the next one.
mathiasbynens commentedon Sep 16, 2026
Hi @HarelM, I’m supportive of including this data in the packages. I haven’t gotten around to implementing this yet, but am happy to review patches.
HarelM commentedon Sep 16, 2026
Not sure the below is helpful:
But I hope it does.