Substitution fonts download mid-render even when the PDF doesn't use them #2005

Summary

With on-demand core font downloads (#1987), mPDF's substitution list can download fonts in the middle of a render that the PDF never uses. #1990 stops this for the 22.9 MB Sun-ExtA, the 17.6 MB Sun-ExtB and FreeSans (1.5 MB), which it uses only once they're on disk, but DejaVu Sans Condensed (0.7 MB) can still download without being used. The obvious fixes each have a cost, so this needs a decision before the v6 on-demand fonts stack lands.

What happens

When a character is missing from the PDF's font, mPDF walks backupSubsFont in order, loading each font until one has the glyph. Loading a missing core font now downloads it, so every font tried before the one that has the glyph is downloaded even when it isn't used.

From #1990 to #2004, the list is arimo, gfpdf-dejavu-symbols, dejavusanscondensed, freesans (only if on disk), <language fonts>, sun-exta (only if on disk) (Model_Font_Sources::get_substitution_fonts()).

• Japanese text on an English site using Arimo: DejaVu Sans Condensed downloads during the render, doesn't have the glyphs, and the characters render as empty boxes because Sun-ExtA isn't on disk.
• Thai text on the same site: DejaVu Sans Condensed downloads unused and the Thai renders as empty boxes. Correction: an earlier version of this issue said FreeSans draws Thai. It doesn't; neither DejaVu Sans Condensed nor FreeSans has Thai glyphs. Thai only renders when Garuda is used, as the PDF's font or as a language font on a Thai site.

Observed in manual QA of #1987, before #1990's change: a Garuda PDF with Thai and Japanese text, on an empty fonts folder, downloaded DejaVuSansCondensed, FreeSans and Sun-ExtA but embedded only Garuda, Garuda-Bold and Sun-ExtA.

Why the obvious fixes fall short

1. Add DejaVu Sans Condensed and FreeSans to the background sync's required set (Model_Required_Fonts::get_required_font_files()). This costs about 2.2 MB per site. A render before the sync finishes can still download them mid-render. Remove the core fonts a new install wouldn't have on the upgrade to 6.19 #1998's upgrade cleanup would then keep them.
2. Only list a substitute once its file is on disk, as Download a site's core fonts in the background, and substitute from Sun-ExtA and Sun-ExtB only once they're on disk #1990 already does for Sun-ExtA. Nothing downloads mid-render, but on an English site using Arimo nothing ever downloads FreeSans: the sync only fetches the default font, the language fonts and the fonts chosen in PDF settings. Thai, Hebrew and similar text in entries would render as empty boxes for good, where it renders today. Existing sites would regress too, as Remove the core fonts a new install wouldn't have on the upgrade to 6.19 #1998's cleanup removes FreeSans on upgrade.
3. Both: list only the substitutes on disk, and have the sync fetch DejaVu Sans Condensed and FreeSans.

FreeSans (changed in #1990)

mPDF's substitution never handles Arabic or the Indic scripts from Devanagari to Malayalam (SubstituteCharsMB() skips U+0600–06FF and U+0900–0D7F, leaving them to OTL and language fonts), so through substitution FreeSans only draws Syriac, Cherokee, Vai and a few symbols. Checked with a render: with Arimo, DejaVu Sans Condensed and FreeSans all on disk, Devanagari, Bengali and Malayalam stayed in Arimo. #1990 now lists FreeSans only once it's on disk.

A fix in mPDF

GravityPDF/mpdf#827 proposes an optional substitutionFontResolver in the mPDF fork: mPDF would pick the substitute from known per-font coverage (a ~34 KB gzipped map of every core font's Unicode ranges, shipped in Gravity PDF's manifest) instead of loading each backupSubsFont entry to find out. Gravity PDF's resolver would download a small font only when it has the characters, queue large ones (Sun-ExtA) for the background, never download a font that can't draw them, and record the characters nothing could draw so the render can be kept out of the cache. It could also substitute from any core font, e.g. Garuda for Thai on an English site. That would replace the on-disk rules above and answer the first question below.

Related: the render cache

A completed download doesn't invalidate the PDF render cache; only FlushCache::flush() bumps the cache generation. A PDF also isn't marked degraded when a font falls back (only failed remote requests mark it, through Helper_Render_Health). So a PDF rendered with empty boxes, because a font was missing, blocked or skipped, is cached and keeps being served after the font arrives. Confirmed in manual QA of #1987: with downloads blocked, the fallback PDF was cached, and once downloads were allowed again the next view served it without trying a download. Whichever fix is chosen, a font arriving (or a fallback render) probably needs to invalidate or skip the cache.

Questions

• Should substitution ever download mid-render, or only use what's on disk?
• Is about 2.2 MB of DejaVu Sans Condensed and FreeSans an acceptable default for every site?
• Should a fallback render be uncached, or should a completed font download bump the cache generation?