Skip to content

fix: load bundled PDF resources from non-ASCII Windows paths - #468

Open
ainuo-AI wants to merge 2 commits into
docling-project:mainfrom
ainuo-AI:fix/windows-resource-paths
Open

ainuo-AI wants to merge 2 commits into
docling-project:mainfrom
ainuo-AI:fix/windows-resource-paths

Conversation

@ainuo-AI

@ainuo-AI ainuo-AI commented Nov 15, 2026 •

Copy link
Copy Markdown

Resolves #115.

With docling-parse installed under a non-ASCII directory on Windows, DoclingPdfParser() can fail while opening glyphs/standard/additional.dat, even though the file is present. The same package works from an ASCII installation path. This happens during resource initialization, before it gets to the PDF.

The native resource loaders mix UTF-8 strings from Python with narrow filesystem and stream APIs. On Windows, the narrow file opens interpret those bytes using the ANSI codepage. This patch keeps resource-directory strings in UTF-8 and converts them explicitly to native std::filesystem::path objects for filesystem operations.

That covers resource discovery, directory enumeration, and the glyph, encoding, AFM and CMap readers. The serial and threaded parsers, native tools and native test callers all use the same convention when passing the resource directory around. The default build-time resource root is decoded the same way. There is no new setting or public API change.

The regression test copies the installed package into ASCII, Chinese and mixed Latin/Hangul/Cyrillic directories, then parses through the public serial, threaded and rendered APIs. Each case runs in a fresh process so resource caches from the original installation cannot hide a failure. Its small generated PDF checks both Latin text and a predefined GBK-EUC-H CMap; the real-document checks are described below.

Evidence:

The existing regression document regression/dln-v1.pdf, page 1 reproduces this without changing the PDF. The dataset revision is 12531342c926708dcb1387ea25213db4603a7255.

To reproduce, install the package in a Windows virtual environment whose path contains non-ASCII characters, save the linked PDF as dln-v1.pdf, and run:

from docling_parse.pdf_parser import DoclingPdfParser

document = DoclingPdfParser().load("dln-v1.pdf")
assert document.number_of_pages() == 1
document.unload()

On unmodified main, the constructor raises this error (installation prefix shortened):

RuntimeError: filename does not exists: <installation>/docling_parse/pdf_resources/glyphs//standard/additional.dat

The before/after builds used the same source base, 88f1c38, with the same GCC 13.1, CMake 4.2.2, pybind11 3.1.0 and dependency binaries. Python files, resource files and runtime DLLs were held constant; only the native extension was replaced. Both builds were Release builds. The machine was Windows 11 x64 with Python 3.12.10 and ANSI codepage 936.

For dln-v1.pdf, page 1:

Installation Main With this patch
ASCII Serial, threaded and rendered parsing succeed All three succeed
Chinese All three fail opening additional.dat All three succeed

The resource file exists and has the same SHA-256 in every installation. Running Python with -X utf8 does not fix the main build; the system ANSI codepage remains 936.

The failing Chinese installation produces no cells before the fix. After the fix, its cells match the working main ASCII installation exactly, for both serial and threaded parsing:

Page 1 output Main, ASCII installation Patched, ASCII installation Patched, Chinese installation
Character cells 99 99 99
Word cells 28 28 28
Text-line cells 16 16 16
Full cell JSON diff against main ASCII Reference Empty Empty

The page's native RGB output is also unchanged: mean absolute difference 0, changed-pixel ratio 0, compared with the working main ASCII installation. This comparison is against main, not pypdfium2.

As a second real-document check, all 13 pages of the unmodified arXiv paper 1702.05596v1 were parsed. Across both documents, both parsing APIs and both patched installation paths, all 168 per-page character/word/line cell comparisons match main ASCII. DocumentConverter with NativePdfFormatOption also converts both documents from ASCII and Chinese input paths under both installations; the exported Markdown hashes match the working main build.

The native --config CLI was checked separately: main fails with the Chinese resource directory, and the patched build writes the expected page JSON. The config, input PDF and output JSON paths were kept ASCII for that check, so it isolates resource loading.

Local tests and remaining validation

The nine new package-location cases give 6 failures and 3 passes on main: each non-ASCII installation fails in all three parser modes, while the ASCII controls pass. All nine pass with this patch.

The focused suite passed 66 tests, with no skips, covering:

tests/test_unit_unicode_paths.py
tests/test_unit_standard_font_widths.py
tests/test_unit_cmap_encoding.py
tests/test_unit_cjk_fonts.py::test_gbk_euc_codes_decode_to_their_characters
tests/test_unit_threaded_page_range.py
tests/test_unit_load_failure.py
tests/test_unit_load_failure_reason.py
tests/test_unit_cid_to_gid_map.py

Ruff formatting and lint, uv lock --check --offline, and git diff --check passed. Ruff 0.17.1, MyPy 1.20.2 and pytest 9.1.1 match the lock file.

Full-project MyPy still reports 19 errors in this Windows environment. Running it against unmodified main produces the same diagnostics: optional benchmark dependencies are missing, and some benchmark code uses POSIX-only interfaces. There are no new diagnostics; this is not a claim that the full pre-commit run is green.

The native extension, parse, render, run_scaling and structure_concurrency compile. The macOS/Linux memory probe uses getrusage/sysconf and fails to build on Windows on both versions; this patch does not port that probe.

Linux/macOS, MSVC/ARM64, the full test suite and the CI Python matrix have not been run locally. Builds used an ASCII source mirror; the non-ASCII compile-time ROOT_PATH case was reviewed in source, rather than built end to end. The Docling conversion checks used the supported native PDF pipeline, without the model/OCR pipeline.

This changes resource location and I/O, with no rendering-algorithm or font-selection changes. The selected-page RGB check above checks for a side effect; corpus-wide pypdfium2 rendering comparisons have not been run.

Ground truth changes:

None. HF_DATASET_REVISION and the existing expected results are unchanged. The existing dataset document already reproduces the failure, so no dataset PR is needed.

Checklist:

  • One topic per pull request, based on current main, no merge commits
  • Superseded pull requests closed and linked: not applicable, none
  • Regression dataset document and page named
  • Before/after evidence for the cited pages
  • Rendering only: not applicable to this resource I/O fix; no corpus-wide pypdfium2 claim
  • Blend2D only: not applicable, no Blend2D changes
  • Changes to existing ground truth listed: none

Keep resource paths UTF-8 at string boundaries and use native filesystem
paths when locating and opening glyph, encoding, AFM and CMap resources.
Use the same conversions in the serial/threaded parsers and native tools.

Add subprocess regressions for ASCII, Chinese and mixed-script package
locations through the serial, threaded and rendered public APIs.

Fixes docling-project#115.

Signed-off-by: ainuo-AI <2470447645@qq.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @ainuo-AI, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

Comment thread src/parse/utils/files.h Outdated
#ifndef PDF_UTILS_FILES_H
#define PDF_UTILS_FILES_H

#include "utf8_path.h"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you just add it here: src/parse.h

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Peter. I've moved the UTF-8 path helper include into src/parse.h, before resources.h, and removed the three includes you pointed out. I also moved the existing parse.h include ahead of docling_resources.h in both the serial and threaded bindings, so the resource initialization sees the helper definitions first.

I rebuilt the native extension and the parse, render, run_scaling, and structure_concurrency targets on Windows x64 (GCC 13 / Python 3.12). All 66 focused tests passed, including the ASCII, Chinese, and mixed-script package-location tests for serial, threaded, and rendered parsing.

I also reran dln-v1.pdf (page 1) and 1702.05596v1.pdf (all 13 pages) from fresh ASCII and Chinese package locations. DocumentConverter succeeded with both ASCII and Chinese input paths using NativePdfPipeline. The text-cell, Markdown, and page-image hashes match the previous version of this PR. The native CLI also passed with both resource paths. These checks don't cover the model/OCR pipeline.

Ruff and the lock-file check passed. The local MyPy run still reports the same 19 diagnostics as unmodified main, with no new diagnostics.

Updated in a94d62f. The follow-up only changes include placement and adds the required SPDX header to parse.h.

Comment thread src/pybind/docling_resources.h Outdated
#include <Python.h>

#include <resources.h>
#include <parse/utils/utf8_path.h>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add it here

Comment thread src/resources.h Outdated

#define LOGURU_WITH_STREAMS 1
#include <loguru.hpp>
#include <parse/utils/utf8_path.h>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can yoy add it here: src/parse.h

Signed-off-by: ainuo-AI <2470447645@qq.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DoclingPdfParser fails to load resources if user path contains special characters (e.g. á) on Windows

2 participants