Skip to content

fix: look up standard-font metrics by the declared encoding's glyph name - #461

Merged
PeterStaar-IBM merged 4 commits into
docling-project:mainfrom
usercjr:fix/403-standard-font-widths
Nov 13, 2026
Merged

PeterStaar-IBM merged 4 commits into
docling-project:mainfrom
usercjr:fix/403-standard-font-widths

Conversation

@usercjr

@usercjr usercjr commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Resolves #403.

Without /Widths, the advance and glyph bbox of a non-embedded standard font come from its AFM, which numbers glyphs in the font's built-in (Standard) encoding. get_width(), nonzero_fallback_width() and has_char_bbox()/get_char_bbox() looked the raw character code up there, ignoring the font's declared /Encoding and /Differences. Every code whose glyph differs from Standard's got the metrics of Standard's glyph at that code: WinAnsi © (0xA9) the width of quotesingle, é (0xE9) that of Oslash, the apostrophe (0x27) that of quoteright, a /Differences Aacute at 193 that of grave. Codes with no Standard glyph (í, 0xED) fell back to 500: in the Latin AFMs those are the unencoded (C -1) glyphs, all accented letters, which nothing looked up by name.

#370 already records the width of C -1 glyphs by name and added base_font::has_glyph_name(), but nothing in page_font looks a width up by name, so the cases above still fail on main. This builds on it:

This changes width and bbox metrics only, not glyph-to-Unicode decoding. #459 also changes page_font.h, in other functions; the hunks don't overlap and the branches merge cleanly.

Evidence:

Measured on main (7cf06e6) and this branch, same local build setup (Windows, MSYS2 MinGW g++ 15.2), dataset at the pinned revision.

Text cells, main → this branch:

7413869934284023443-1.pdf p2. Helvetica-Bold with /Differences [193 /Aacute 211 /Oacute] and no base encoding. Á gets grave's 333 units instead of 722, so the heading word is split. Accented letters elsewhere on the page move the words after them. The page goes from 414 to 413 word cells; 54 cells change.

before after
Á (228.60–233.91), REA (240.12–267.71) ÁREA (228.60–267.71)
COMPETICIÓN (292.80–383.03) COMPETICIÓN (292.80–386.63)
repescará, (155.40–201.85), si (204.27–210.83) repescará, (155.40–197.86), si (200.28–206.84)

7413869934284023443-1.pdf p1 has the same heading, so Á and REA become ÁREA and COMPETICIÓN widens the same way (442 → 441 word cells).

annots_02.pdf p1. Helvetica, WinAnsi apostrophe. 4 word cells change:

before after
don't (335.38–357.06) don't (335.38–356.75)
forget (359.84–385.41) forget (359.53–385.10)

annots_01.pdf p1. Helvetica, WinAnsi € and others. 23 word cells change, for example € 174.77–179.27 → 174.77–179.78, then 95.00 181.77–204.29 → 182.28–204.80.

Rendering against pypdfium2, with measure_image_pair and the page selection of test_regression_pypdfium_render.py (954 pages of 457 documents, dataset b09d0574):

page mean_abs_error changed_pixels_ratio panel
7413869934284023443-1.pdf p2 3.918 → 3.248 0.0825 → 0.0789 image
annots_01.pdf p1 2.831 → 2.585 0.0697 → 0.0685 image
7413869934284023443-1.pdf p1 2.862 → 2.829 0.0732 → 0.0729 image
annots_02.pdf p1 1.973 → 1.946 0.0485 → 0.0483 image

9 other pages differ by at most 3e-05 in mean_abs_error, and no page gets worse by more than 1e-05. The other 941 pages are unchanged. Corpus mean: 2.45693 → 2.45591. Pages above the 9.0 tolerance: 13 → 13.

Panels are pypdfium2 | main | this branch:

7413869934284023443-1.pdf p2: pypdfium2, main, this branch

Some m and c glyphs of the body text are missing from both docling-parse panels. That is a separate problem, the same on main, and this PR doesn't change it.

These numbers come from Windows, where non-embedded Helvetica resolves to a different system font than on Linux. I also checked the cited page on Ubuntu 24.04 with the font packages that checks.yml installs: main 3.992 / 0.0833, this branch 3.349 / 0.0798, so the test fails on main and passes here on Linux too. pytest tests/test_unit_*.py there: 695 passed, 10 skipped.

Tests:

  • test_regression_pypdfium_cited_pages.py gains standard-font-differences-width (7413869934284023443-1.pdf p2, limits 3.6 / 0.081). It fails on main and passes here.
  • tests/test_unit_standard_font_widths.py gains cases for WinAnsi ©, é, í and the apostrophe, which fail on main and pass here.
  • pytest tests/test_unit_*.py on Windows: 642 passed, 61 skipped (main: 640 passed). Two tests that feat: working on parallel efficiency #435 added fail on Windows, the same on main and on this branch: test_reference_preflight_handles_native_abort[-6] (signal.Signals(6) doesn't exist there) and test_mid_run_commit_requires_exact_source_equivalence[None].

Ground truth changes:

I haven't regenerated the stored ground truth. Comparing the word cells of all 954 pages between main and this branch, 4 pages change, because the width or bbox of glyphs in non-embedded standard fonts changes:

  • 7413869934284023443-1.pdf p1 and p2: word, char and rendered output (ÁREA becomes one word).
  • annots_01.pdf p1: word, char and rendered output.
  • annots_02.pdf p1: word, char and rendered output.

Checklist:

  • One topic per pull request, rebased on main, no merge commits
  • Superseded pull requests closed and linked (none)
  • Regression dataset documents and pages named
  • Before/after evidence for each cited page
  • Rendering only: pypdfium2 comparison images, corpus-wide metrics table, and an enforced image regression test
  • Blend2D only: link to the upstream report (not applicable)
  • Changes to existing ground truth listed with the reason for each page

This PR was prepared with Claude Code (Claude Opus 5.5). The measurements and tests above were run locally on my machine.

🤖 Generated with Claude Code

Without /Widths, the advance and glyph bbox of a non-embedded standard
font come from its AFM, which numbers glyphs in the font's built-in
(Standard) encoding. The raw character code was looked up directly, so
a font declaring /WinAnsiEncoding got the metrics of whatever Standard
has at that code: `©` (0xA9) the width of `quotesingle`, `é` (0xE9)
that of `Oslash`, the apostrophe (0x27) that of `quoteright`. Codes
with no Standard glyph, like `í` (0xED), fell back to 500.

docling-project#370 already records the width of unencoded (`C -1`) glyphs by name.
Record their bbox by name too, add name-based width and bbox accessors
to base_font, and resolve the code through get_glyph_name() (declared
/Encoding and /Differences) before the code-based lookup in
get_width(), nonzero_fallback_width() and has_char_bbox()/
get_char_bbox(). Fonts without an explicit encoding are unchanged.

Add 7413869934284023443-1.pdf p2 to the cited-page tests: its
Helvetica-Bold maps 193 to Aacute through /Differences, and the heading's
"Á" was drawn with the width of `grave`.

Fixes docling-project#403.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: usercjr <92197541+usercjr@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @usercjr, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@PeterStaar-IBM
PeterStaar-IBM self-requested a review November 13, 2026 06:37
PeterStaar-IBM
PeterStaar-IBM previously approved these changes Nov 13, 2026
@mergify mergify Bot mentioned this pull request Nov 13, 2026
7 tasks
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
@PeterStaar-IBM
PeterStaar-IBM merged commit 3be81ff into docling-project:main Nov 13, 2026
45 of 84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrong advance widths for non-ASCII characters in non-embedded standard-14 fonts with /WinAnsiEncoding (© gets the width of quotesingle)

2 participants